Transferring Files to and from Quest
This page provides information on transferring files to and from Quest.
Quest supports several secure file transfer protocols and systems for transferring files to and from Quest. Use an SFTP client if you are unfamiliar with Linux systems and want to transfer files to or from a Windows or Mac computer.
Secure File Transfer Protocol (SFTP) Client
Both Mac and Windows FTP/SFTP clients are available from the Cyberduck site. The Cyberduck Help wiki provides basic usage instructions and feature lists, and should be reviewed before using it for the first time.
To connect to Quest, start Cyberduck and then :
- Click Open Connection in the upper left of the Cyberduck window
- At the top of the Open Connection window that appears, Select SFTP (SSH File Transfer Protocol) from the drop-down menu.
- Enter quest.it.northwestern.edu for server specification
- Enter your NetID in the Username: box and leave the Password: box empty to prevent your NetID password from being saved in a file on your personal computer. Public Key Authentication is not supported.
- Click Connect. You will see a Login failed window.
- Enter your NetID password in the Password: field.
- Click Login.
Northwestern has launched a project to retire Box by August 2022. Therefore, we encourage you to transition from Box to another storage service solution as soon as you are able.
To move files from Box directly onto Quest:
Connect to Quest with X11 forwarding enabled. If you are using FastX to connect to Quest, or used the -X option with ssh, then you have X11 forwarding enabled.
Run Firefox by typing the following commands in the terminal window: module load firefox, followed by firefox to launch the Firefox browser.
In the Firefox browser, type northwestern.box.com and log in.
- Select the file you wish to move onto Quest and click Download in the upper right corner. By default, the file will be placed into the Downloads directory in your home directory.
You can also use FTP to transfer files to and from Box.
- Create an external password in Box by logging in to https://northwestern.box.com, then going to the Account Settings page and clicking "Create Password" in the Authentication section. See screenshot below:
- On Quest, use the lftp program to log in to Box via FTP:
lftp -p 990 -u <your Northwestern email address> ftps://ftp.box.com
Enter your external password (not your NetID password) when prompted.
- Use the lcd ("local cd") command to change directories on Quest and the cd command to change folders in Box. Use the ls command to list files and folders in Box. You can run shell commands on Quest from within lftp by prefixing them with "!". For example, !ls will list local files and directories on Quest.
- To upload files or directories: Use the queue command to add files to your upload queue with put (for a single file) or mput (to add multiple files at once with wildcard expansion). For example:
- To download files or directories: Use the queue command to add files to your download queue with get (for a single file) or mget (to download multiple files at once with wildcard expansion). For example:
To upload directories: Use the mirror -R command, for example:
mirror -R directory_name
To download directories: Use the mirror command, for example:
Secure Copy (SCP)
You can use Secure Copy to transfer individual files to or from Quest using the Linux or Mac OSX Terminal application on the computer you want to transfer files to/from. If you are using a Windows system, you will need PuTTY; Windows users substitute the command pscp for scp in the examples below.
These commands are executed from your computer, NOT from Quest (your Terminal prompt should reference your computer, not Quest).
To transfer files from your computer to Quest:
scp /PathToSourceFile/file NetID@quest.it.northwestern.edu:/PathToTargetDir/file
scp /Users/willie/myfile.txt firstname.lastname@example.org:/home/abc123/myfile.txt
To transfer files from Quest to your computer:
scp NetID@quest.it.northwestern.edu:/PathToSourceDir/file /PathToTargerDir/file
scp email@example.com:/home/abc123/myfile.txt .
This would copy the file /home/abc123/myfile.txt from Quest to the current working directory on your computer with the same name (myfile.txt).
For more examples of Secure Copy, see In Unix, how do I use SCP to securely transfer files between two computers? Note that scp will overwrite existing files without prompting.
If you need to transfer entire directories,
You can download datasets directly to Quest from external ftp and web servers using the wget command (see the Wget manual).
While connected to Quest, you could use commands like:
To copy files into your current working directory on Quest.
With additional options wget can also download whole directories of files recursively and download from password-protected servers.
Globus is a Software-as-a-Service (SaaS) that provides a program interface for file transfer and sharing, as well as identity, profile, and group management. You can use Globus to transfer very large data sets to and from Quest.
A Globus transfer handles all the difficult aspects of data transfer by optimizing parameters to maximize bandwidth usage, managing security configurations, providing automatic fault recovery, and notifying users of completion and problems. Command line and web interfaces are available for use.
Learn more about when to use globus to transfer files to and from Quest. Globus may also be used to transfer files to and from the Research Data Storage Services and FSMRESFILES. When transferring files to/from the Research Data Storage Service or FSMRESFILES, please avoid the use of symbolic links on Quest. This results in duplicate copies of your data on the Research Data Storage Service and FSMRESFILES. Please contact firstname.lastname@example.org if you would like to use this Globus.
File transfer between Quest and the Research Data Storage Service is limited to using the non-audited storage zone, resfiles.northwestern.edu. Quest does not support analysis of regulated data (such as PHI and PII). This includes data placed in FSMRESFILES and other Research Data Storage Service audited storage zones designed for handling such data.
To transfer files, it is recommended to use Globus (see above) or smbclient via the command line. When transferring files to/from the Research Data Storage Service or FSMRESFILES, please avoid the use of symbolic links on Quest. This results in duplicate copies of your data on the Research Data Storage Service and FSMRESFILES.
To check for symbolic links in a directory, use the command:
find [path-to-directory] -type l -ls
To use smbclient to transfer files, after logging into Quest:
1. Uploading files from Quest to Research Data Storage Service:
smbclient '//storage-host-name/share-name' -c 'put file-name-for-upload /path-on-RDSS/file’ -U "ads\netid"
smbclient '//resfiles.northwestern.edu/MY_LAB' -c 'put myQuestFileToCopy /myResfilesDestinationPath/myResfilesDestinationFile' -U "ads\abc123"
2. Downloading files to Quest from Research Data Storage Service:
smbclient '//storage-host-name/share-name' -c 'prompt;recursive;lcd path-on-Quest; cd /path-on-RDSS/file-name; mget *' "ads\netid"
smbclient '//resfiles.northwestern.edu/MY_LAB' -c 'prompt;recursive;lcd /myQuestDestinationpath/myQuestDestinationfile; cd /myResfilesPath/myResfilesFileToCopy; mget *' -U "ads\abc123"
3. Accessing SMB interactive shell at the storage:
smbclient \\\\resfiles.northwestern.edu\\MY_LAB -U "ads\abc123"
NOTE: If individual file size is over 10 GB, add the following socket options to your smbclient transfer.
smbclient YOUR-COMMAND-OPTIONS-FROM-ABOVE --socket-options='TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=131072 SO_SNDBUF=131072'
Following command will list all available commands:
smb: \> help
For more information, on smbclient please see the Samba documentation.