Using Globus Online with Quest

Walkthrough on how and when to use Globus to transfer files to and from Quest.

Overview

Globus is a Software-as-a-Service (SaaS) that provides a program interface for file transfer and sharing, as well as identity, profile, and group management. It provides high performance, secure, third-party data movement and synchronization between endpoints.

A Globus transfer handles all the difficult aspects of data transfer allowing by optimizing parameters to maximize bandwidth usage, managing security configurations, providing automatic fault recovery, and notifying users of completion and problems. Command line and web interfaces are available for use.

When to Use Globus

  1. Transferring very large data sets (larger than a few GBs). Globus uses GridFTP, a high performance transfer protocol that uses parallel TCP streams for optimal bandwidth. For large transfers (TBs and above), it is significantly faster (8X) than using scp, rsync or sftp.
  2. Collaboration and data sharing. Many national labs, universities, and data centers around the world use it for data management.
  3. “Fire and forget” transfers so that users can start transferring files and work on something else, while it automatically optimizes transfer settings, retries any failed attempts, and emails you when the file transfer is done. In case of any dropped network connection, it picks up right where it left off and performs a checksum to check for file corruption and confirms the data integrity.

Usage: Transfer to/from Quest

Globus transfers files between two endpoints.  Quest is one endpoint. 

If you are transferring files to or from your personal computer, you need to first set up a personal endpoint (see next section below). If you are transferring files to or from another system, you need the name of the endpoint for that system.

Once you know the endpoint from which you'll be transferring files to Quest, or to which you'll be transferring files from Quest:

  1. Sign in to Globus. The primary option is to "Use your existing organizational login." Northwestern is available as an organization. When you select Northwestern and click continue, you will be redirected to NU Online Passport to log in through a websso authentication (or signed in automatically if your browser already has your credentials).
  2. You will then be on the File Manager screen.
  3. Use the "Collection" search box to find Globus endpoints.
    • If using a personal endpoint, your personal endpoint will be listed under the tab "Your Collections".  Make sure that you have Globus Personal Connect running on your computer first so that Globus can connect to your system.
    • If using an endpoint on another system, type the name into the Collection field at the top of the window. That will bring up a list of search results from which you can choose the correct endpoint.
    • Browse the collection and click on the file(s) to be transferred, then click "Transfer or Sync to..."
  4. In the right panel, click in the Collection box and type Northwestern Quest. This will bring up several search options.  Choose the one that only says "Northwestern Quest."
    Globus Northwestern Quest Collection
  5. In the two file windows, navigate to the files you want to transfer and the directory to which you want to transfer the files.  Choose files to transfer, and request the transfer with one of the directional arrows. A green notification panel will appear to confirm that the transfer request has been submitted. Once the transfer is completed, you will see an activity notice at the bottom of the Transfer Files page and receive an email from Globus.

Setup a Personal Endpoint

If you want to transfer from a laptop or workstation, install Globus Connect Personal to set up a personal endpoint that will allow you to share files to and from this machine.

  1. Follow the link above, and then select the link for your operating system.
  2. The first step in the installation instructions (before the numbered steps) is to create a Globus Connect Personal endpoint.  Follow that link.  Northwestern University is listed as an Identity Provider for you to choose.  Choosing Northwestern will establish a Globus account associated with your NetId so that you do not need to establish a different account; you will be redirected to NU Online Passport to log in through a websso authentication.
  3. Choose a display name for your personal laptop/workstation when prompted.  Then click the button to generate a setup key.  Copy the key to a file on your personal computer. 
  4. Choose the button appropriate to your operating system to download and install the Globus Personal Connect software.
  5. After installing Globus Personal Connect, run the application and you will be asked for a setup key. This is the key you generated earlier and copied to a file on your computer.  After you enter the key, no application windows will be visible for Globus Personal Connect, but it will be running in the background (look for it in your task bar or menu).

To Transfer Files:

  1. Log into Globus Online, select Manage Data and then Dashboard and click on Globus Connect Personal and enter an endpoint name to uniquely identify your machine.
  2. Click Generate Setup Key, copy and paste the setup key into the Globus Connect window to complete the setup process.
  3. Your endpoint is added to the list of endpoints and should be identified under the format <Globus-username>#<Globus-connect-endpoint-name>.

Other Globus Endpoints

If you want to transfer from another storage system or HPC cluster, the system must be setup as a Server endpoint. This is typically done by system administrators.

National cyberinfrastructure resources have established Globus endpoints. For example, the XSEDE project is under the format xsede#<resource-name>. The Stampede system at Texas Advanced Computing Center is named xsede#stampede.

You can also use Globus to transfer data to and from Amazon S3

Save

See Also:




Keywords:quest, globus, file, transfer, sharing, data transfer, management   Doc ID:71271
Owner:Research Computing .Group:Northwestern
Created:2017-03-02 16:29 CDTUpdated:2019-04-15 13:20 CDT
Sites:Northwestern
Feedback:  0   0