You are here

hpc.Time Supercomputing

High-Performance Computing (HPC) is the use of large-scale, off-site computers and parallel processing techniques for solving complex computational problems. HPC technology focuses on using parallel processing algorithms and systems. HPC is typically used for solving advanced problems and performing research activities through computer modelling, simulation and analysis. The terms High-Performance Computing and Supercomputing are used interchangeably.


Uses of HPC

HPC systems are most often useful for researchers:

  • with parallelisable tasks (parallelism can be achieved on a fine or coarse grain level — on a fine level you have a program which is intrinsically parallel while on a coarse grain level you might run many instances of the same program for parameter studies)
  • to take advantage of fine/coarse grain parallelism, embarrassingly parallel and parameter studies techniques
  • with computing jobs needing more memory (RAM) than available on their local systems (if you run out of memory on your desktop system or can no longer process your data quickly enough)
  • needing access to software packages offered by the HPC facilities.
     

HPC jobs run in a non-interactive batch mode. These jobs  are submitted to a queue, and when the time comes for them to be executed they can run without any user intervention. This means that HPC systems rarely sit idle but utilise the hardware fully. It has the benefit that you can queue your jobs and get on with other things while you wait for them to execute. If you are accustomed to using your program interactively, you will have to learn how to use its batch mode in order to use it for HPC.


HPC Training

Intersect offers a series of HPC courses to meet the needs of all researchers, whether you are at a Beginner, Intermediate or Advanced HPC level. Check our Training page for more details. 

  • Intensive Beginner to Intermediate HPC - Introduction to UNIX for HPC
  • Intermediate HPC - From Unix to HPC
  • Advanced HPC - Parallel Programming 
     

Supercomputing Facilities for Intersect researchers

As a researcher you have access to a number of facilities:

  • The National Facility at NCI (National Computing Infrastructure), raijin.NCI.org.au, which serves researchers nationally
  • Intersect's Orange facility, orange.Intersect.org.auwhich serves researchers regionally
  • The NeCTAR and OwnTime Research Clouds, over 4,500 local and 32,000 distributed computing cores 
  • Facilities local to your institution or faculty

 

For details about the last option, please contact your institution.

A few factors come into play when selecting between the National Facility and Intersect's Orange facility:

  • The availability of your software prerequisites
  • The extent of your processing needs
  • The number of nodes required for a single job
  • How much memory is required memory
  • The availability of "specialised facilities" tailored to your needs

 


Choosing your HPC Facility

To make the right choice, take the following steps:

  1. Investigate the software available at the NCI Raijin and on Intersect's Orange. The following links go to the same page - you can navigate between machines using the site menu at the top of this NCI page.
  2. Don't be dissuaded if your required software is not listed - researchers can request to have new software packages installed. However, requests cannot always be fulfilled.
  3. Compare the hardware specifications at the National Facility, NCI, and on Orange:
  1. Consider the memory requirements of your research problem. Orange is a large-scale shared memory system while NCI’s Raijin facility is a distributed memory system. This means that Orange is suited to certain classes of research applications where access to large amounts of memory is a key performance requirement.
  2. Also have a look NCI's Specialised Facilities. NCI offers specialised Bioinformatics and Imaging and Visualisation facilities. If your research is in either of these areas, then NCI could well be the right choice for you.

 

 The NeCTAR Research Cloud, rc.NeCTAR.org.au is powered by OpenStack.org, a cloud orchestration system that controls large pools of compute, storage, and networking resources throughout multiple Australian data centres, all managed through a dashboard that gives administrators control while empowering consumers with web self-serve resource provisioning. Pre-baked virtual machine images are catalogued at: https://support.rc.nectar.org.au/docs/images

 

Example decision paths

  • Alice has a large amount of Gaussian jobs. After checking the software listings, she chooses the National Facility because it supports Gaussian.
  • Bob has an extremely memory intensive application needing more than 128GB memory per node. The National Facility's cluster and Orange have the same architecture, but Orange has some nodes with up to 256GB memory which are not available on Raijin, so Bob chooses Orange.
  • Carlos wants to run S3D on 2,000 cores in a single a job. He cannot run such a large job on Orange (it has only 1,660 cores), he opts for the National Facility's cluster because it has a larger number of cores.
  • Dorothy wants to conduct computationally intensive gene sequencing. She chooses both the National Facility so she can take advantage of the specialised bioinformatics facility and Intersect Orange so that she gets additional computing resources.

Start Up and Ad-hoc Projects

So you've chosen your facility - how do you gain access to it? There are two ways to quickly access resources on Orange: Ad-hoc and start-ups.

The easiest way is to get an ad-hoc project on the system you want to try. It is usually possible to get a small amount of time to try the system out, without going through the full allocation process.

Intersect has smaller (<20K SUs/quarter) HPC resource to allocate throughout year — get in contact if you have a small project and we will try accomodate you as a start-up project.

All HPC projects are limited in terms of the amount of computing resources (e.g. processing time) to which you have access. This is referred to as your quota, and when it runs out you can still run in the bonus queue. Ad-hoc projects have a fairly limited quota, but it is enough for you to assess if HPC is right for you, i.e. to install your application, submit a job, and see if it all runs smoothly.

A typical ad-hoc project will give you 1,000 service units of computing resources. One SU usually corresponds to one CPU core-hour, so if you run a parallelised program that uses four processes and runs for one hour before finishing, you will have used 4 SUs of your quota.

For an ad-hoc project simply:

1      Contact your eResearch Analyst or your institution’s HPC contact to request an ad-hoc project. This lets them know that your application is coming so they can approve it.

2      Go to the following link and register using the Application Form for new projects at NCI. As can be seen, the form is quite straightforward:

3. Once you have submitted this form, you will be issued with an "identity number". Send this identity number to your contact and they will process your application.

4. When the application has been processed you will receive and email with your account username and password. This indicates that you are ready to use the HPC system. At this point it's best to let your contact know and they can assist you.

 

Once you've successfully completed your evaluation using an ad-hoc project, you can apply for a much larger quota through the merit-based resource allocation scheme. This application will be subject to appraisal along with other applications, and time will be granted on the basis of merit.

A start-up requires a full application, but can be made outside the regular annual Resource Allocation Round, (i.e. October application for January start). Applications can be very brief. Please be aware start-up allocations are modest, up to 20k SUs/quarter (if still available for that quarter) and are eligible for top-ups each mid-quarter.

NCI also run some start-up projects on Raijin. The Default limit is 1k SUs/quarter but this can be discussed with NCI. 


How to use HPC

Once your start-up project request has been processed, you should receive the details of the username and password required to login to your account. You can then connect to the machine and submit your first compute job.

When getting started on NCI, see the User Guide at https://nf.nci.org.au/facilities/userguide/ for instructions on how to log in and use the job-submission system. 

Connecting to the machines

There are two key ways to access the HPC machines. The first is via an interactive terminal that allows you to issue commands, such as adding your compute job to the queue. The second mode of access is for transferring files to and from your computer and the HPC machine, for example the data you wish to process and, once the job is complete, the results files.

To connect interactively to Orange you'll need a Secure Shell (SSH) client. If you're on Linux or Mac OS X, chances are that you can just open a terminal and issue one of the following command:

ssh orange.intersect.org.au

When prompted, enter your username and password. If you're running Windows, you'll need to download an SSH client. A good free option is Putty.

To transfer files to and from your machine, you'll need a File Transfer Protocol (FTP) client that supports Secure-FTP (SFTP). You can do this from the command line on Linux, Mac OS X, and Windows:

sftp orange.intersect.org.au

But you might find it easier to use a graphical FTP program, such as FileZilla.

Running jobs

There are slight differences. Please see the machine handbooks for Orange and NCI.
 


HPC Software Registries

Complete lists of all software installed are available at the NCI software list. On the NCI page there is a drop down menu from which you can select almost all HPC machines in Australia, including Intersect's Orange.

Please note that software packages marked with a yellow dot carry license restrictions. Please contact us to establish whether you can use this software. 

Setting up software environments

To set up the environment for a software package you need to use the module system. This is identical on both machines but the module names might differ. To see the exact names of the modules visit the links given above or to get a list use the command:

module avail

To load, for example, the latest Intel compilers on Orange, you would use the command:

module load intel/11.1.069

This sets up your environment (variables and path). The reason for using this module system is to allow for different versions of the same software package.

Other useful module commands include:

A list of already loaded modules: module list

To show you what a module does: module show package-name 

To unload a package module unload <package> ()

Software installation and requests

If you would like to have software installed on Orange, please email to time@intersect.org.au and specify the download site or download the software yourself and let us know where to find it in your home directory. You are also welcome to install software yourself in your home directory.

For software requests at NCI please use the following form: http://nf.nci.org.au/accounts/forms/software.php


Where to go next

You've selected the right facility, you've got your start-up project, you've run a test compute job and everything looks fine. How do you get more computing resources (ie. service units) so that you can run a full-scale compute job and get some real results?

Irrespective of whether you're using Intersect's Orange or NCI's cluster, computing resources are granted in merit allocation rounds held periodically. You can apply in the following rounds:

  • Orange resource allocation rounds occur once per year where applcations open in October for the following calendar year. Further details at hpc.Time Merit Allocation.
  • Intersect has a partner share arrangement with NCI. This provides a second means of accessing the NCI facilities. Consider applying directly through NCI in the first instance and making an application through Intersect if your request to NCI is not granted. Further details at hpc.Time Merit Allocation.
  • NCI merit allocation round. This occurs annually in October for allocations starting the following January. 
  • Through Intersect you can gain resources on Raijin and Orange. For Intersect resources on Raijin there is a mid-term adjustment after 6 weeks from the start of each quarter. Adjustments are made on a usage basis. Projects which have used less than 30% of their resources at that time will be scaled back. Prior to this an email is sent to the CI of that project. 

 


Checklist

It is recommended that you work through this checklist to help assess if HPC is of benefit to you:

  1. Estimate your computing needs. Work up a rough estimate of the number of CPU hours your compute job will require. As a guide a dual core desktop machine running flat out will provide you with about 1500 core hours (service units) per month. If your current computing infrastructure can handle your computing needs in the timeframe of your research project then skilling up in HPC might cost you time overall.
  2. Check your memory and disk requirements. Disk space is limited, especially on Orange.
  3. Leave plenty of time. If you think HPC could be of use to you, then start investigating it early into your research project. It can take time to assess if your software will run on HPC and to become comfortable in using the command line, module, and queueing systems if you've never used similar systems before. Finally, because HPC is a shared resource, your job will most likely spend some time in the queue awaiting execution.
  4. Check for available software. Verify that your chosen facility supports the software your job depends on. This is particularly important in the case of commercial software which may not be available due to licensing costs.
  5. Consider the nature of your research problem. Can it inherently take advantage of the parallelisation of HPC? If not, can you run multiple instances at the same time with different parameters or data to process? If not, then your compute job may not be able to take advantage of HPC's capabilities.
  6. Check that your software can be used non-interactively. If your processing involves interacting with the program, for instance, clicking buttons or selecting input files using a file browser, then you'll need to find a way to do those things from the command line using a batch mode, if your program supports it.

 

Please email time@intersect.org.au for questions, advice and support.

Software Registries

Complete lists of all software installed are available at the NCI software list. On the NCI page there is a drop down menu from which you can select almost all HPC machines in Australia, including Intersect's Orange.

Please note that software packages marked with a yellow dot carry license restrictions. Please contact us to establish whether you can use this software. 

Setting up software environments

To set up the environment for a software package you need to use the module system. This is identical on both machines but the module names might differ. To see the exact names of the modules visit the links given above or to get a list use the command:

module avail

To load, for example, the latest Intel compilers on Orange, you would use the command:

module load intel/11.1.069

This sets up your environment (variables and path). The reason for using this module system is to allow for different versions of the same software package.

Other useful module commands include:

A list of already loaded modules: module list

To show you what a module does: module show package-name 

To unload a package module unload <package> ()

Software installation and requests

If you would like to have software installed on Orange, please email to time@intersect.org.au and specify the download site or download the software yourself and let us know where to find it in your home directory. You are also welcome to install software yourself in your home directory.

For software requests at NCI please use the following form: http://nf.nci.org.au/accounts/forms/software.php