1 July 2016 Changes to Improve Sustainability
The NSW node of the NeCTAR Research Computing Cloud has adopted a consumption funding model to sustain its future in the national eResearch landscape. Learn more here.
What is CloudTime?
CloudTime is the NeCTAR Research Cloud is a national collaboration to provide high-powered, reliable cloud computing infrastructure to Australian researchers. NeCTAR allows researchers to deploy virtual machines (VMs). Using the VMs, they can then build and use software tools to run experiments, process and analyse data, or host complex online research applications – without having to build or maintain the necessary hardware.
The NeCTAR Research Cloud consists of 8 cloud nodes across Australia.
Intersect owns and operates one of the national nodes, offering around 5000 of the Australian capability of approximately 40,000 x86 computing CPU cores. Production cores are used for private cloud computing for Intersect customers, offered as OwnTime.intersect.org.au. Supporting assets include large scale memory, high speed networking, high I/O disk and virtualised infrastructure based on OpenStack, all connected via the AARNet ScienceDMZ high speed optical network and colocated with Space.intersect.org.au.
NeCTAR supports several operating systems and provides a large number of pre-built images that researchers can use as the basis for their own VMs, without having to configure the entire system. Researchers can also take snapshots of their own configured VM and make them available for others to use.
Typical uses of the NeCTAR Research Cloud include:
- Deploying compute resources for running simulation and data analysis software
- Hosting of websites, databases, web applications and other online resources
- Establishing customised access to online data sets and data analysis tools
- Hosting of online research tools and domain-specific virtual laboratories.
Who can access the CloudTime?
The NeCTAR Research Cloud can be accessed by any researcher from any organisation that participates in the Australian Access Federation (AAF). All Australian universities are members of AAF. Project allocations on the Intersect node are authorised by participating member organisations.
How do I access the CloudTime?
To access the research cloud, you will need to log in via AAF using your university login and password. Once logged in, you will see a Dashboard with your ‘personal trial project’ (which is identified with a ‘pt-’ code). If you have additional projects, such as project allocations, these will be listed in the Dashboard as well.
Personal Trial projects
Every user account in the NeCTAR research cloud is allocated a small amount of computing resource – a maximum of two virtual server instances, totalling no more than two cores – with which they can build a basic VM that will have around the same power as a desktop computer. Personal trial instances will expire after 4000 CPU hours, which is about three months of continuous use. They are intended for limited testing purposes and are not recommended for production servers. In order to obtain a server to be used for production, it’s best to apply for a project allocation.
Project allocations can be much larger than your personal trial and can last for much longer . Within a project allocation, you may build as many instances as your allocated resources allow and these instances will persist for the life of the project. Instances built within project allocations can also be shared among other users. They are designed for collaborative research.
To obtain a project allocation, you need to submit a request through the NeCTAR Dashboard. Allocation requests are reviewed by an allocation committee for merit, suitability to cloud use, and available capacity of the cloud and, for Intersect Australia, authorised by a particpating member organisation. When submitting the online form to request a project allocation, please provide as much information as possible regarding your project to expedite the process.
Building a Virtual Machine
Building an instance, as part of either your personal trial or a project allocation, is done through the Dashboard. A wizard will guide you through the process, which consists of selecting the ‘flavour’ (size) of the machine, the image to boot from, the ‘availability zone’, which is the node it will be hosted on, and some additional details. Once built, a VM can be imaged, terminated, shut off or rebooted as necessary, and more instances can be deployed, as long as your total usage does not exceed the resources available in your project.
What else do I need to know?
There are a number of important pieces of information relating to NeCTAR VMs, including the size of the instance, storage options, access and security, and reliability.
1. VM size options
There are several ‘flavours’ of instance available, measured in cores. One core consists of 1 CPU, 4GB memory and 30GB of local disk storage.
The flavours are as follows:
- Small: 1 CPU, 4GB memory, 30GB ephemeral disk, 10GB system disk
- Medium: 2 CPUs, 8GB memory, 60GB ephemeral disk, 10GB system disk
- Large: 4 CPUs, 16GB memory, 120GB ephemeral disk, 10GB system disk
- XLarge: 8 CPUs, 32GB memory, 240GB ephemeral disk, 10GB system disk
- XXLarge: 16 CPUs, 64GB memory, 480GB ephemeral disk, 10GB system disk
A personal trial is equivalent to a ‘medium’ offering, meaning that you can build one medium VM, or two small VMs, immediately upon logging in to the Dashboard.
When applying for a project allocation, you will be asked to select the flavour that you will require. An application for a small or a medium-sized allocation is more likely to be approved than an application for an XXL sized allocation. For this reason, explain why you need the size of server you are requesting, with as much information as possible.
2. Storage options
NeCTAR offers three types of storage, intended for different uses. Ephemeral storage is the default, and each VM will be configured with an ephemeral disk attached, the size of which depends upon the size of the instance. Ephemeral disks exist only as long as the VM they are attached to remains active. They are deleted when an instance is terminated.
Users can also apply for persistent block storage (volumes) for their allocations. Volumes work like network attached storage devices; they can be mounted and unmounted from VMs within your project allocation (as long as they are within the same availability zone), the data remains persistent if not attached to a VM, and they can be backed up with snapshots.
NeCTAR also offers object storage for higher reliability, although configuring object storage requires more advanced skills. The benefits of object storage are that the data can be distributed across many availability zones, and can be accessed via http tools even when not attached to any instance.
3. Access to VMs
In order to connect to and work on your VM, you will need to configure security groups (firewall rules), which enable traffic through certain ports for different kinds of access. The default security groups for project trials are:
- SSH opens tcp port 22 to traffic from all sources (for logging in via ssh)
- HTTP opens tcp ports 80 and 443 to traffic from all sources (for web servers)
- ICMP opens all ICMP traffic from all sources (to allow pinging your VMs IP address).
For project allocations, all security groups will need to be configured. See the Security Groups technical guide for assistance. Security groups can be configured and created while a VM is running. Changes will take effect immediately.
To access your VM over SSH, you will also need to create an SSH keypair. A keypair works like a lock and key and means that you do not need a password to log in, as long as you have your private half of the key pair (the key), and the server has the public half of the key pair (the lock). SSH keypairs are very secure, as you never transmit a password over the web to log in to the server.
Keypairs can be generated using the NeCTAR dashboard in the ‘access and security’ tab. When you have generated a keypair, the public key will be written to the VM, and the private key will be available for you to download and place in your .ssh directory. You will then need to configure your machine to use it to connect to the VM. If you already have a keypair or if you create one on your own machine, you may upload your public key to the dashboard. Once you have a keypair in your NeCTAR account, you will be able to use the public key for any instance you build, and the private key on any machine you wish to connect from.
You must create/upload your keys prior to launching a VM, otherwise NeCTAR will be unable to write the public key into the .ssh directory of the VM. You will also be unable to connect from any client device. For assistance in creating and using SSH keypairs, see the SSH keypairs technical guide.
4. Reaching your quota
When a project’s resource quota has been reached, an email is sent to the user, warning them that their VMs will be terminated in one month. At this point, you should ensure that you have a backup of your data and make a snapshot of the VM. If you still need the machines to run after the end-date, you should submit an allocation request, or amend the existing request with a new date.
5. Reliability of NeCTAR services
The hardware in cloud computing can be susceptible to power outages, scheduled downtime and other events. These may affect your project allocations and trial allocations. NeCTAR does allow you to select a particular node to host your VM (referred to as an ‘availability zone’), but each node is susceptible to temporary failures.
If your project requires a server with uninterrupted uptime, you should take measures to ensure that a power outage does not affect it, such as a parallel instance running the same processes hosted at a different node.
If your project does not require constant uptime, it may be sufficient to simply boot the instance again when the node is restored. NeCTAR routinely communicates to users via email service information, including outages and scheduled maintenance.
For advice in selecting the most appropriate NeCTAR VM for your project, or assistance in requesting or configuring an allocation, please contact the NeCTAR Research Cloud support page or email firstname.lastname@example.org for assistance.