Using Python on Quest
Tips for using Python on Quest.
- Using Python on Login Nodes
- Using Python with Scheduled Jobs
- Python Package Management
- Jupyter Notebooks on Compute Nodes via Interactive Jobs
Python scripts can be run on the login nodes with some limits. Users cannot use more than 4 cores or 4 GB of memory in total for all the applications they are running on a login node. These limits are implemented to ensure a smooth operation for all users connecting to Quest through login nodes.
All Quest nodes natively include Python (version 2.7.5). However, this Python version has no package support and users are strongly encouraged to use Python distributions provided by the module system. You can see the available versions of Python with the command
The command output will include several python/<distribution> modules. Currently, only Anaconda distributions are actively supported on Quest. You can make a particular version of Python available to use by loading the corresponding module. For instance, if you would like to work with Anaconda distribution of Python 3.6, you should type the command
The default module, which is identified in the output of module avail python command, will be loaded if you do not specify the <distribution> part when you load the Python module.
To start the interpreter, type
python on the command line. You can also run your Python script directly by typing the following line on the command line
Python can be used interactively on login without scheduling jobs as discussed above. Note that, however, the login node use is for development and non-computationally intensive tests runs. Compute nodes can be accessed via scheduling jobs on Quest. There are two classes of jobs: interactive and batch.
Interactive Use on the Compute Nodes
To use Python interactively, please first submit an interactive job to obtain the cpu and memory resources you need from a compute node. Once your job starts, your command line will move to a compute node where you can proceed as if you are on the login node. If you are planning to do computationally intensive work while maintaining interactivity with the script or the interpreter, this type of use recommended. This is also the way to request resources to use an IDE such as Spyder (which is available as part of the Anaconda distributions of Python).
In submission scripts for batch jobs, you must include the command to load the version of Python that you want to run, then add the line where you can call Python:
which causes the output and errors to be written to the files for stdout and stderr, respectively, which are determined by your job submission parameters (see Submitting a Job on Quest).See Example of Python Batch Jobs for Quest for template files you can adapt for your work.
module load python will load the current default version of Python, to ensure the future compatibility of your scripts with the Quest system, we highly recommend that you always load a specific version of Python rather than relying on the default version.
It is important to make sure that the required packages for a Python script are already installed in the Python module you have loaded. If the packages are not available your job will fail. The most reliable way to test the availability is to import the package within the Python interpreter. If you don't receive any errors after the import, that means the package is available. You can also inquire the version of the package with the Python command
<package-name>.__version__ as most packages include this information. For instance if you would like to test Numpy's existence and query its version you could do the following:
$ module load python/anaconda3.6 $ python >>> import numpy as np >>> np.__version__
pip freeze will list packages in your current environment, but the list may be incomplete. It is best to test package availability by importing the package, as above.
Local Package Installation
If you'd like to run a package not included in the system wide version of Python, you can install the package locally for yourself in your home directory and still use the system wide version of Python to run it. Pip package manager is one of the most common ways of installing Python packages. Use:
pip install --user <package_name>
to install packages locally. Note that it's the --user flag which causes the installation files to go into your local directories .local/lib/python and ./local/bin. Once you've done the local install, you must add the local package executables in .local/bin to your PATH:
To add .local/bin to your path every time you login, add the above line to your ~/.bashrc file (learn more about .bashrc). Once .local/bin is in your path, the system-wide version of Python will be able to locate your locally installed package.
If the package is widely used in your community, you can request it to be installed for the Python module system wide. This way everyone who loads the Python module will have access to the package. Email email@example.com to make this request.
You can use conda environments with the Anaconda Python modules on Quest, or virtual environments with any Python module. To share an environment with other members of your allocation, you can create the environment in a specific directory.
With conda, use the --prefix option to create the environment in a specific location, such as in your /projects/<allocationID> directory. To use the environment (either interactively from the command line or as part of your job submission script), you then need to point to this directory when activating it.
For example, to create a conda environment called env1 in a subdirectory of a project directory (here p10000 as an example), with the package sqlalchemy included:
module load python/anaconda3.6
conda create --prefix /projects/p10000/pythonenvs/env1 sqlalchemy
To use this environment, change to the directory where you installed it:
module load python/anaconda3.6
source activate env1
or, specify the full path to the environment when you activate it:
module load python/anaconda3.6
source activate /projects/p10000/pythonenvs/env1
"conda activate" vs. "source activate":
Older conda versions used the source activate <environment name> syntax to activate an environment on Linux, which differed from how environments were activated on Windows. In order for conda to work consistently across platforms, its behavior changed slightly in version 4.6, and now conda activate <environment name> works on all platforms. To use the new conda, run this command one time to set up your environment:
echo ". /software/anaconda3/2018.12/etc/profile.d/conda.sh" >> ~/.bashrc
Then log out of Quest and log back in. Once you have done this, you can use conda activate instead of source activate, although the latter will still work, so existing batch scripts for example can continue to use source activate.