Using Globus Online with Quest

Walkthrough on how and when to use Globus to transfer files to and from Quest.

Overview

Globus is a Software-as-a-Service (SaaS) that provides a program interface for file transfer and sharing, as well as identity, profile, and group management. It provides high performance, secure, third-party data movement and synchronization between endpoints.

A Globus transfer handles all the difficult aspects of data transfer allowing by optimizing parameters to maximize bandwidth usage, managing security configurations, providing automatic fault recovery, and notifying users of completion and problems. Command line and web interfaces are available for use.

When to Use Globus

  1. Transferring very large data sets (larger than a few GBs). Globus uses GridFTP, a high performance transfer protocol that uses parallel TCP streams for optimal bandwidth. For large transfers (TBs and above), it is significantly faster (8X) than using scp, rsync or sftp.
  2. Collaboration and data sharing. Many national labs, universities, and data centers around the world use it for data management.
  3. “Fire and forget” transfers so that users can start transferring files and work on something else, while it automatically optimizes transfer settings, retries any failed attempts, and emails you when the file transfer is done. In case of any dropped network connection, it picks up right where it left off and performs a checksum to check for file corruption and confirms the data integrity.

Usage: Transfer to/from Quest

Globus transfers files between two endpoints.  Quest is one endpoint. 

If you are transferring files to or from your personal computer, you need to first set up a personal endpoint (see next section below). If you are transferring files to or from another system, you need the name of the endpoint for that system.

Once you know the endpoint from which you'll be transferring files to Quest, or to which you'll be transferring files from Quest:

  1. Sign in to Globus. The primary option is to "Use your existing organizational login." Northwestern is available as an organization. When you select Northwestern and click continue, you will be redirected to NU Online Passport to log in through a websso authentication (or signed in automatically if your browser already has your credentials).
  2. You will then be on the File Manager screen.
  3. Use the "Collection" search box to find Globus endpoints.
    • If using a personal endpoint, your personal endpoint will be listed under the tab "Your Collections".  Make sure that you have Globus Personal Connect running on your computer first so that Globus can connect to your system.
    • If using an endpoint on another system, type the name into the Collection field at the top of the window. That will bring up a list of search results from which you can choose the correct endpoint.
    • Browse the collection and click on the file(s) to be transferred, then click "Transfer or Sync to..."
  4. In the right panel, click in the Collection box and type Northwestern Quest. This will bring up several search options.  Choose the one that only says "Northwestern Quest."
    Globus Northwestern Quest Collection
  5. Activate the endpoint when prompted by logging in with your NetID and password. The Northwestern Quest endpoint can be activated for a maximum of seven days. After this activation expires, you will need to re-activate to transfer files to or from this endpoint, unless you create a Shared Endpoint (see below).
  6. In the two file windows, navigate to the files you want to transfer and the directory to which you want to transfer the files.  Choose files to transfer, and request the transfer with one of the directional arrows. A green notification panel will appear to confirm that the transfer request has been submitted. Once the transfer is completed, you will see an activity notice at the bottom of the Transfer Files page and receive an email from Globus.

Setup a Shared Endpoint

Managed endpoints can be activated for a maximum of seven days before manual reactivation in the Globus web UI is required. However, you can create "shared endpoints" within a managed endpoint that do not require re-activation and use them like any other endpoint. What's more, you can grant read-only or read-write access to a shared endpoint to any other Globus user, even if they are at a different institution. This allows you to grant access to a collaborator at another institution, allowing them to upload or download files to a Quest project folder or RDSS share.
  1. In the Globus file manager, search for and click on the managed endpoint you wish to create a shared endpoint within (e.g. "Northwestern Quest" or "Northwestern Quest RDSS"). Activate if necessary.
  2. Click the "Share" button in the right menu of the file manager
    Share endpoint button
  3. Click the "Add a Shared Endpoint" button
  4. Next to the Path input, click the "Browse" button and find the folder you wish to share. For RDSS shares, this will look like "/rdss/<netid>/resfiles/<foldername>", and for Quest project folders this will look like "/projects/<projectfolder>". Click the "Select" button when you have highlighted the folder you wish to share.
  5. For Share Display Name, give the shared endpoint a friendly name that you and your collaborators will recognize, e.g. "<Project name> share on resfiles".
  6. Click the "Create Share" button.
Your share is now created, and is accessible only by you. To find it in the Globus file manager, search for it using the display name you gave it, or click the "Endpoints" navigation then "Shareable by You".

To grant access to other users, follow these steps:
  1. Click the "Add Permissions - Share With" button on the "Sharing" tab of your shared endpoint.
  2. The Path input on this page is relative to the root of the shared endpoint, not the managed endpoint (Quest or RDSS). To grant access to the entire shared endpoint, leave the Path as "/". To grant access to only a subfolder, enter it as "/<subfolder>".
  3. Search for users by name or Globus identity. The default Globus identity for Northwestern users is <netid>@northwestern.edu.
  4. The default permissions are read-only. To grant write access to the shared endpoint, click the "write" checkbox.
  5. Click the "Add Permission" button to grant access.
Note that shared endpoints do not require reactivation. Thus if you wish to use the Globus CLI to script transfers, for example regular backups from cron, it is highly recommended to use a shared endpoint.

Setup a Personal Endpoint

If you want to transfer from a laptop or workstation, install Globus Connect Personal to set up a personal endpoint that will allow you to share files to and from this machine.

  1. Follow the link above, and then select the link for your operating system.
  2. The first step in the installation instructions (before the numbered steps) is to create a Globus Connect Personal endpoint.  Follow that link.  Northwestern University is listed as an Identity Provider for you to choose.  Choosing Northwestern will establish a Globus account associated with your NetId so that you do not need to establish a different account; you will be redirected to NU Online Passport to log in through a websso authentication.
  3. Choose a display name for your personal laptop/workstation when prompted.  Then click the button to generate a setup key.  Copy the key to a file on your personal computer. 
  4. Choose the button appropriate to your operating system to download and install the Globus Personal Connect software.
  5. After installing Globus Personal Connect, run the application and you will be asked for a setup key. This is the key you generated earlier and copied to a file on your computer.  After you enter the key, no application windows will be visible for Globus Personal Connect, but it will be running in the background (look for it in your task bar or menu).

To Transfer Files:

  1. Log into Globus Online, select Manage Data and then Dashboard and click on Globus Connect Personal and enter an endpoint name to uniquely identify your machine.
  2. Click Generate Setup Key, copy and paste the setup key into the Globus Connect window to complete the setup process.
  3. Your endpoint is added to the list of endpoints and should be identified under the format <Globus-username>#<Globus-connect-endpoint-name>.

Other Globus Endpoints

If you want to transfer from another storage system or HPC cluster, the system must be setup as a Server endpoint. This is typically done by system administrators.

National cyberinfrastructure resources have established Globus endpoints. For example, the XSEDE project is under the format xsede#<resource-name>. The Stampede system at Texas Advanced Computing Center is named xsede#stampede.

You can also use Globus to transfer data to and from Amazon S3

Save

See Also:




Keywords:quest, globus, file, transfer, sharing, data transfer, management   Doc ID:71271
Owner:Research Computing .Group:Northwestern
Created:2017-03-02 16:29 CDTUpdated:2019-12-19 16:01 CDT
Sites:Northwestern
CleanURL:https://kb.northwestern.edu/using-globus-online-with-quest
Feedback:  0   0