Transferring Files to and from Quest

This page provides information on transferring files to and from Quest.

Quest supports several secure file transfer protocols and systems for transferring files to and from Quest. Use an SFTP client if you are unfamiliar with Linux systems and want to transfer files to or from a Windows or Mac computer.

Secure File Transfer Protocol (SFTP) Client

Both Mac and Windows FTP/SFTP clients are available from the Cyberduck site. The Cyberduck Help wiki provides basic usage instructions and feature lists, and should be reviewed before using it for the first time.

To connect to Quest, start Cyberduck and then :

  1. Click Open Connection in the upper left of the Cyberduck window
  2. At the top of the Open Connection window that appears, Select SFTP (SSH File Transfer Protocol) from the drop-down menu.
  3. Enter quest.it.northwestern.edu for server specification
  4. Enter your NetID in the Username: box and leave the Password: box empty to prevent your NetID password from being saved in a file on your personal computer. Public Key Authentication is not supported.
  5. Click Connect. You will see a Login failed window.
  6. Enter your NetID password in the Password: field.
  7. Click Login.​

Box

To move files from Box directly onto Quest:

  1. Connect to Quest with X11 forwarding enabled. If you are using FastX to connect to Quest, or used the -X option with ssh, then you have X11 forwarding enabled.
  2. Run Firefox by typing the following commands in the terminal window: module load firefox, followed by firefox to launch the Firefox browser.
  3. In the Firefox browser, type northwestern.box.com and log in.
  4. Select the file you wish to move onto Quest and click Download in the upper right corner. By default, the file will be placed into the Downloads directory in your home directory.

You can also use FTP to transfer files to and from Box. To do so, you must create an external password for Box (do NOT use your netid password for this). Find more information on the Box FTP page. From Quest, you can use the command line ftp program to connect to the Box FTP.

Secure Copy (SCP)

You can use Secure Copy to transfer individual files to or from Quest using the Linux or Mac OSX Terminal application on the computer you want to transfer files to/from. If you are using a Windows system, you will need PuTTY; Windows users substitute the command pscp for scp in the examples below.

These commands are executed from your computer, NOT from Quest (your Terminal prompt should reference your computer, not Quest).

To transfer files from your computer to Quest:
scp /PathToSourceFile/file NetID@quest.it.northwestern.edu:/PathToTargetDir/file
Example:
scp /Users/willie/myfile.txt abc123@quest.it.northwestern.edu:/home/abc123/myfile.txt

To transfer files from Quest to your computer:
scp NetID@quest.it.northwestern.edu:/PathToSourceDir/file /PathToTargerDir/file
Example:
scp abc123@quest.it.northwestern.edu:/home/abc123/myfile.txt .
This would copy the file /home/abc123/myfile.txt from Quest to the current working directory on your computer with the same name (myfile.txt).

For more examples of Secure Copy, see In Unix, how do I use SCP to securely transfer files between two computers? Note that scp will overwrite existing files without prompting.

If you need to transfer entire directories,

Wget

You can download datasets directly to Quest from external ftp and web servers using the wget command (see the Wget manual).

While connected to Quest, you could use commands like:
wget http://www.nber.org/nberces/bbg96_87.dta
wget ftp://ftp.bls.gov/pub/doc/ap.txt
To copy files into your current working directory on Quest.

With additional options wget can also download whole directories of files recursively and download from password-protected servers.

Globus

Globus is a Software-as-a-Service (SaaS) that provides a program interface for file transfer and sharing, as well as identity, profile, and group management. You can use Globus to transfer very large data sets to and from Quest.

A Globus transfer handles all the difficult aspects of data transfer by optimizing parameters to maximize bandwidth usage, managing security configurations, providing automatic fault recovery, and notifying users of completion and problems. Command line and web interfaces are available for use.

Learn more about when to use globus to transfer files to and from Quest. Globus may also be used to transfer files to and from the Research Data Storage Services and FSMRESFILES. When transferring files to/from the Research Data Storage Service or FSMRESFILES, please avoid the use of symbolic links on Quest. This results in duplicate copies of your data on the Research Data Storage Service and FSMRESFILES. Please contact quest-help@northwestern.edu if you would like to use this Globus.

SMBCLIENT

File transfer between Quest and the Research Data Storage Service is limited to using the non-audited storage zone, resfiles.northwestern.edu. Quest does not support analysis of regulated data (such as PHI and PII). This includes data placed in FSMRESFILES and other Research Data Storage Service audited storage zones designed for handling such data.

To transfer files, it is recommended to use Globus (see above) or smbclient via the command line. When transferring files to/from the Research Data Storage Service or FSMRESFILES, please avoid the use of symbolic links on Quest. This results in duplicate copies of your data on the Research Data Storage Service and FSMRESFILES.

To check for symbolic links in a directory, use the command:
find [path-to-directory] -type l -ls

To use smbclient to transfer files, after logging into Quest:

1. Uploading files from Quest to Research Data Storage Service:

smbclient '//storage-host-name/share-name' -c 'put file-name-for-upload

/path-on-RDSS/file’ -U "ads\netid"

Example:

smbclient '//resfiles.northwestern.edu/MY_LAB' -c 'put myQuestFileToCopy

/myResfilesDestinationPath/myResfilesDestinationFile' -U "ads\abc123"

2. Downloading files to Quest from Research Data Storage Service:

smbclient '//storage-host-name/share-name' -c 'prompt;recursive;lcd

path-on-Quest; cd /path-on-RDSS/file-name; mget *' -U

"ads\netid"

Example:

smbclient '//resfiles.northwestern.edu/MY_LAB' -c 'prompt;recursive;lcd  /myQuestDestinationpath/myQuestDestinationfile;  cd /myResfilesPath/myResfilesFileToCopy;  mget *' -U "ads\abc123"

3. Accessing SMB interactive shell at the storage:

smbclient \\\\resfiles.northwestern.edu\\MY_LAB -U "ads\abc123"

NOTE: If individual file size is over 10 GB, add the following socket options to your smbclient transfer.

smbclient YOUR-COMMAND-OPTIONS-FROM-ABOVE --socket-options='TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=131072 SO_SNDBUF=131072'

Following command will list all available commands:

smb: \> help

For more information, on smbclient please see the Samba documentation.

See Also:




Keywords:Quest, file, file transfer, ftp, sftp, globus, smb, scp, secure copy, rsync, cyberduck, Box, FSMRESFILES, rdss, filetransfer   Doc ID:70521
Owner:Research Computing .Group:Northwestern
Created:2017-02-07 22:29 CSTUpdated:2017-10-06 07:21 CST
Sites:Northwestern
CleanURL:https://kb.northwestern.edu/quest-filetransfer
Feedback:  1   0