Useful Resources for New Raijin Users

Access the Raijin command line:

ssh username@raijin.nci.org.au

• How to access Raijin
• Help Resources
• New users and projects
• Rajin User Guide
• Raijin and queue status
• Raijin software list – Click on a software package to see a sample PBS script.
• Raijin Software request form

Software marked by an orange dot has license restrictions. Projects need to be added to that user group via request. Please contact us.

For project and user management NCI uses Mancini which allows you to add new users to your project, request access to license restricted software, request new projects etc.

Useful commands on Raijin:

• nci_account -P project-code: Shows allocated and used resources. Replaces quotasu on Orange.
• lquota: shows disk usage and quota. Replaces quota on Orange.
• nf_limits: shows limits of walltime and memory for specified project

Use the man pages for more options of the commands above.

Queues on Raijin

Raijin has several batch queues

• The normal queue consists of nodes identical to Orange: 2×8 cores with Intel Sandy Bridge. 1 hour compute time costs 1 SU per core.
• The express queue has the same nodes as the normal queue but jobs get a priority for which the cost is 3x as high compared to the normal queue.
• The normalbw and expressbw are similar to the ‘normal’ and ‘express’ queues but the nodes are 2×14 cores based on Intel Broadwell. Depending on your code these nodes are faster (mainly if your code uses AVX-2). Code must be recompiled for these nodes to take advantage of the new CPU architecture. See Raijin handbook. The relative costs for the two queues are 1.25 and 3.75 respectively.
• Raijin also has a 30 GPU nodes with 4xK80 each, 2 GPU nodes with 4xP100 each and 32 KNL nodes (64 cores per node). Please see the NCI handbook for details and how to compile the code for these nodes. Since there are only a few of these nodes check also the batch queue how many jobs are waiting.
• In the hugemem queue jobs with up to 1TB can be run. Soon this will be upgraded to 3 nodes with 3TB each.
• Use the copy queue to copy large data sets from /short to the MDSS archive using 1 core. See the handbook for details.
• The walltime is usually 48h. Extensions can be negotiated. Make sure you explored the option of checkpointing your program before asking for an extension.

Storage on Raijin

Please note the following differences to Orange:

• /home on Raijin has a quota of 2GB (60GB on Orange)
• /short/project-code replaces /projects/project-code. Quota depends on project
• /g/data{1,2,3} replaces /projects/project-code. Quota depends on project.

Scheduler on Raijin

Raijin uses a customised version of PBS Pro. PBS scripts have to be adapted from Orange to Raijin in order to work. Please see the software list and included sample scripts.

HPC Training Courses

Our HPC training courses will be fully updated to include everything you need to know to work productively on Raijin, including revised PBS scripts to get you started. Please visit Training to find a training course near you.

More information and help

Support continues for projects under the Intersect partner share via hpc@intersect.org.au.

Please reach out for any help, questions or concerns at any time.