Running Jupyter Notebook on Quest

Jupyter Notebook can be launched and run on a Quest compute node through an interactive job on Quest.

To schedule the interactive job from the command line on Quest, ssh into a login node and type:

srun -A <allocation_name> -p <queue_name> -N 1 --tasks-per-node=1 --mem-per-cpu=4G --time=04:00:00 --pty bash -l

This example requests a single core for a 4 hour job. Substitute an active allocation name and queue name, for example if using allocation p12345 this might be:

srun -A p12345 -p short -N 1 --tasks-per-node=1 --mem-per-cpu=4G --time=04:00:00 --pty bash -l

Note that the more cores requested, the longer the wait for the interactive session to start. Do not request more than 1 node for Jupyter notebook sessions.

Once the session begins, get the name of the compute node the session has landed on:

hostname

Load the version of python that you require for the session. For example, to load Python 2.7:

module load python/anaconda

To use Python 3 instead, use module load python/anaconda3 or module load python/anaconda3.6. Once the correct Python module has loaded, type:

jupyter notebook --port=8899 --no-browser

This example uses port 8899, but a different port can be specified if necessary (for example, if another user is already using that port).

Once jupyter notebook is running on the compute node, open a new terminal window on your local computer, and type:

ssh -L 8899:localhost:8899 <your_netID>@quest.northwestern.edu ssh -N -L 8899:localhost:8899 qnode<number>

If using a different port number, substitute it for the number "8899" in the command. Be sure and use your netID, and at the end replace qnode<number> with the name of the compute node. You will be prompted for your Quest password, which will not return a prompt.

On your local computer, open up your browser and connect to http://localhost:8899/. If not using port 8899, change the address to use the port specified when you launched jupyter notebook. Your browser is now connected to the jupyter notebook session running on Quest.

Note that your jupyter notebook session will quit abruptly when the walltime of the interactive job comes to an end. Save often and be aware of walltime to avoid losing your work.

Accessing Conda Environments in Jupyter

By default, jupyter will use the "root" environment of the Anaconda module you have loaded. If you are using a conda environment, for example to be able to install specific versions of software or because you are sharing an environment with other people in your group, you will need to manually install the environment into your jupyter configuration. This can only be done from the command line, not within the jupyter notebook. First, load whichever Anaconda module you are using and activate your environment, for example:

module load python/anaconda3.6
source activate <environment_name>

Then install the ipykernel package if it is not already installed:

conda install ipykernel

After the package is finished installing, use it to install the environment as an available kernel within jupyter, substituting the appropriate name and display name for your environment:

python -m ipykernel install --user --name <environment_name> --display-name "My Environment"

Then, when you launch jupyter using the commands given above, you will have the option of launching a notebook using your newly installed environment as the kernel:

jupyter new notebook menu

See Also:




Keywords:research computing, jupyter, python, quest, anaconda, notebook, ipython   Doc ID:88448
Owner:Research Computing .Group:Northwestern
Created:2018-12-13 14:51 CDTUpdated:2019-08-27 16:47 CDT
Sites:Northwestern
CleanURL:https://kb.northwestern.edu/running-jupyter-notebook-on-quest
Feedback:  0   0