Getting Started on the Genomics Compute Cluster (b1042) on Quest

All genomics researchers at Northwestern can apply for access to the genomics compute cluster.

About the Genomics Compute Cluster on Quest

The Feinberg School of Medicine, in an initiative spearheaded by the Feinberg School of Medicine Office of the Dean, the Center for Genetic Medicine, and the Department of Biochemistry and Molecular Genetics provides over 4,000 compute cores on Northwestern’s High Performance Computing Cluster, Quest, to be used for genomics research. This University resource has been made available to the greater genomics community in an effort to foster genomics research and empower the computational genomics community at Northwestern University.

How do I start using the Genomics Compute Cluster?

You will first need to log in to Quest.

Moving files onto Quest

If you have sequencing done by NUSeq Core, they will deliver your FASTQ files via Northwestern Box. To get files from Box onto Quest:

  1. Log in to Quest. Northwestern IT recommends you use FastX to connect, but experienced users may opt for an SSH connection with X11 forwarding enabled.
  2. Run Firefox by typing the following commands in the terminal window: module load firefox, followed by firefox to launch the Firefox browser. A Firefox browser window will open.
  3. In the Firefox browser that opened, go to https://northwestern.box.com and log in with your NetID credentials.
  4. Select the file you wish to move onto Quest and click Download in the upper right corner. By default, the file will be placed into the Downloads directory in your home directory.

Note: If you want the file to go directly into /projects/b1042, before downloading click on the menu icon in the upper right of the Box browser, and click Preferences. From there you can select which directory to download into on Quest, including the /projects directories.

For information about transferring files from your local machine, RDSS (via smbclient), or Globus to Quest please visit the Transferring Files on Quest web page.

Using bioinformatics software available on Quest

To see a complete list of software installed on Quest by the Quest administrators, type module avail. Modules are scripts that make it possible to run software in your shell; if you would like to run a particular software package, use the module load command, and then run the software.

If you would like to use additional software packages not currently installed on Quest, please see Installing Software on Quest.

Running jobs on the Genomics Compute Cluster: Submission Scripts

To submit, monitor, modify, and delete jobs on Quest you must use Slurm commands as outlined in the Quest Slurm Quick Start Guide.

Genomics Compute Cluster Partitions

Partitions are defined for different types of jobs and have different resource limits. Note that Partitions are also called queues.

Feinberg Partitions:

  • genomics: For jobs that require less than 48 hours to run. Researchers may submit multiple jobs to the genomics partition at one time and utilize many cores simultaneously. Jobs can request up to 180GB of RAM, and 52 cores.
    Script example: #SBATCH -p genomics
  • genomicslong: For jobs that require between 48 - 240 hours to run. Priority is lower on this partition than the genomics partition to reserve more resources for shorter jobs. Jobs may request up to 180GB of RAM, and 52 cores.
    Script example: #SBATCH -p genomicslong
  • genomics-gpu: Submits jobs to the NVIDIA A100 gpu cards. See Using Genomics Compute Cluster GPUs on the GPUs on Quest page for instructions on how to submit jobs to these gpus.
    Script example: #SBATCH -p genomics-gpu
  • genomics-himem: Submits jobs to the 1TB high-memory nodes. Jobs may request up to 1TB of RAM, and 28 cores.
    Script example: #SBATCH -p genomics-himem
  • genomics-burst: For special projects, large jobs requiring more than 5 nodes or that run for more than 48 hours may use the genomics-burst partition by contacting quest-help@northwestern.edu. Before using genomics-burst partition, users must meet with a Research Computing Services consultant to confirm that code has been reviewed for efficiency. These appointments will be set up after reservations are requested and are intended to ensure best practices for this shared resource. Because of the resources involved, jobs may need to wait for availability when requesting the genomics-burst partition. It is advised to schedule at least three (3) weeks in advance.

Non-Feinberg Partitions:

  • genomicsguest: For jobs that require less than 48 hours to run, a lower-priority version of the genomics partition. Jobs can request up to 180GB of RAM, and 52 cores.
    Script example: #SBATCH -p genomicsguest
  • genomicsguestex: For jobs that require between 48 - 240 hours to run, a lower-priority version of the genomicslong partition. Jobs can request up to 180GB of RAM, and 52 cores.
    Script example: #SBATCH -p genomicsguestex
  • genomicsguest-gpu: Submits jobs to the NVIDIA A100 gpu cards. See Using Genomics Compute Cluster GPUs on the GPUs on Quest page for instructions on how to submit jobs to these gpus.
    Script example: #SBATCH -p genomics-guest-gpu
  • genomics-himem: Submits jobs to the 1TB high-memory nodes. Jobs may request up to 1TB of RAM, and 28 cores.
    Script example: #SBATCH -p genomics-himem

For example job scripts see the GitHub repository of example jobs.

See Also:




Keywords:genomics, quest, research computing, b1042, bioinformatics, genetics, bwa, rna-seq, gatk   Doc ID:78602
Owner:Research Computing .Group:Northwestern
Created:2017-12-07 13:38 CSTUpdated:2021-12-07 11:49 CST
Sites:Northwestern
CleanURL:https://kb.northwestern.edu/genomics-compute-cluster
Feedback:  37   35