Submitting HPC Jobs Tutorial

Submitting and Running HPC Jobs

Submitting and running jobs on the cluster is a straight forward procedure with 3 basic steps:

  1. Setup and launch.
  2. Monitor progress.
  3. Retrieve and analyse the results.

Key to launching jobs is what we refer to as the PBS Script. It contains a series of instructions that you could provide on the command line (to the command qsub). You will almost certainly find using a script a lot easier! We talk about this script a bit lower down.

Setup and launching of HPC jobs

Users with more than one project can set the default project to which their quota is captured by typing the following command (followed by the project ID):

set_default_project

Alternatively, this can be done by logging into the HPC User Community page on the VPAC website front page and selecting Cluster Account at the top of the page. Their users will be able to choose their default project.

When uploading files in preparation for launching a HPC job it is good practice to keep each job in separate folders, labeled in an intuitive way, such as:

Namd_albumin_run_01

Writing all files to the top level home directory will very quickly become difficult to follow and easy to make mistakes!

Launching jobs on the cluster is controlled by PBS software (Portable Batch System), which allocates the compute nodes and time requested by the user which the user specifies in the pbs_batch_script.

The user can edit the pbs_batch_script to change the number of preferences including the number of cpus to use, the length of time to run the job and the name of the program executable. It is important to realize alll lines that start with #PBS pass a PBS command, while adding a white space does not.

For example, compare the lines:

#PBS -l walltime=24:0:0
# PBS -l walltime=24:0:0


The former will implement a PBS walltime, whereas the latter will not.

Note that your PBS script does two jobs, firstly it tells the scheduler about your job, how many CPUs you need, what to do if something goes wrong etc. This instruction lines always start with "#PBS". Lots of PBS users use a trick of disabling an instruction by putting a space in that string, typically between the "#" and the "PBS". Secondly, the scheduler will 'run' your script when it comes time to launch your job. Now, any line starting with a "#" will be ignored, but any real command will be observed.

Here are some parameters you will see in a PBS Script, the ones you almost always need (usually a variation of the example) are in bold :

#!/bin/bash
# At the start of all scripts.
# pbs script for a NAMD job:
# A comment line.
#PBS -N Job_name
# The name of the job.
#PBS -l nodes=8
# Asking for 8 cores.
#PBS -m ae
# Mail user if job aborts (a) or ends (e) (currently commented out).
#PBS -l walltime=24:10:5
# Set the job to run 24 hours 10 minutes and 5 seconds.
#PBS -o output_file_name.txt
# Set output file name.
module load namd
# Load the NAMD module
cd $PBS_O_WORKDIR
# Change to the Work director.
# Launching the job!
mpiexec namd2 config_file
# Launching the executable “namd2” with the file “config_file”.


More notes on pbs scripts

Nodes:

Requesting more nodes for a particular job does not necessarily mean better performance or faster completion. This depends on how well parallelized the program is. Also, requesting large numbers of cpus (> 64) may result in the job waiting in the queue for days or weeks while the scheduler makes resources available. For example, a 32 cpu job running in 5 days, may take 64 cpus 7 days (3 days to run plus and extra 4 days while waiting in the queue!).

Note that there are some well-documented scripts on /common/examples/PBS

Memory:

By specifying how much memory you need, you can let the scheduler either sneak your job in somewhere quickly or make sure that there is enough memory for your job to run happily. Without this information we sometimes see a number of jobs all running on one node and all needing more than the default memory. And this can result in jobs running badly or even failing.

Walltime:

Walltime is the length of time specified in the pbs script for which the job will run. Make sure that you have specified enough walltime for a particular job to complete! Your job will end after the allocated walltime whether it is finished or not, sometimes resulting in lost data if the program does not checkpoint restart files. If you are not sure how long a job will run, set a generous walltime, and then check the job status before the walltime expires.

If your job needs more time simply email help@vpac.org and ask the system administration to change the walltime for the job. If we see the message in time we'll fix it for you.

Modules:

Modules are simply a way to set the environment variables appropriate for the job you want to run. In your pbs_batch_script make sure you have module load [module_name] before the executable line. For example, for a NAMD job, we include:

module load namd

for a FLUENT job we would include:

module load fluent

You can type module avail at the command line to see what modules are installed.

Submitting the job

Once you are satisfied with your pbs_batch_script and have uploaded all the input files for your job simply type at the command line:

qsub pbs_batch_script

The important thing is if you are submitting to the Enrico you should modify the pbs script to submit to the fermi queue which has the Enrico compute node assigned to. i.e.,

#PBS -q fermi

Monitoring progress

Once a job is launched it is possible to monitor how it is progressing. One way to do this is type at the command line:

showq

This will show all current jobs on the cluster so to pick out the lines relevant to you use the useful grep function:

showq | grep [username]

If all is well you should see a line like:

> showq |grep joe
83238 joe Running 16 23:26:49 Wed Feb 13 14:38:33

Reading across the line, the first number is the job number, (83238), owned by joe and the status of the job is “Running”. The job is running on 16 cpus and has 23 hours 26 minutes and 49 seconds remaining. It was submitted on Wed Feb 13 at 14:38:33.

Another way to see this information is with the -u option:

showq -u [username]

If the cluster is busy, you job may have to wait in the queue, in which case the status of the job would be 'Idle'.

If your job is listed as 'Idle', one can check when it is likely to start by the command:

showstart [job_number]

If you don't see any line then either your job has finished or died!

If you know the job number you can view the output while it is still running by typing at the command line:

jobstatus o [job_number]

Likewise, to see error messages, (if any), type

jobstatus e [job_number]

Stopping a job

If you see some output you don't like and wish to stop your job prematurely, you may do so with the command:

qdel [job_number]

Retrieving and Analyzing Results

Often it is most convenient, depending on the type of simulation, to download the output of job to your local computer for analysis. If you have run your job in a folder then it is simply a matter of downloading that entire folder to your desktop.

Most post-processing the data can usually be done happily on most desktops and don't require further HPC resources at that point.

Top of Page