Enhancing HPC Usage Efficiency by Building User Capability: A Collaboration between NSW DCCEEW and Intersect 3AI Platform

By Jiaxin Fan | Aug 22nd 2024

Background

During the 2023-2024 financial year, an initiative was established between Intersect’s Advanced Analytics and AI (3AI) Platform and the Science High-Performance Computing (HPC) team at the New South Wales Department of Climate Change, Energy, the Environment and Water (NSW DCCEEW, formerly known as NSW Department of Planning and Environment) led by Dr Qurat Tariq. This project aims to provide technical user support and training on the Science, Data, and Compute (SDC) platform – alinux based HPC infrastructure crucial for scientific research across the NSW DCCEEW. This project was bolstered by Intersect’s decade-long experience in delivering top-notch digital skills training in conjunction with the expertise of 3AI’s Research Data Scientist, Jiaxin Fan, in providing HPC support.

NSW DCCEEW Science High Performance Computing Team, June 2024

Challenge

One of the main challenges faced by scientists was the knowledge gap they had for Linux and fundamentals of the HPC platform. Many users lacked sufficient IT and programming skills , resulting in both underutilisation of the HPC resources and reduced capability to perform scientific activities effectively. Dr Tariq explains:

“The science teams using the system were expanding and finding it difficult to get the new team members up to speed with skills required to use the system. Most novice users were relying on senior colleagues for guidance and SDC support .“

Solution

A comprehensive and multi-faceted program was implemented by the 3AI team to bridge the gap between the SDC capabilities and the users’ proficiency. The four major aspects included developing new user documentation, offering one-on-one support, running fortnightly Hacky Hour sessions, and facilitating digital skills training.
1. Facilitating Digital Skills Training: Regular training sessions were delivered to enhance the scientists’ HPC literacy. These sessions covered a broad range of topics, from basic command-line skills to advanced data analysis techniques, ensuring users could maximise the use of the HPC platform.
2. Developing New User Documentation: Clear, concise, and user-friendly documentation was created to guide users through the basic and advanced functionalities of the SDC. These documents included infrastructure overviews, step-by-step instructions, troubleshooting tips, and examples tailored to common scientific tasks.
3. Offering One-on-One Support: Personalised support was provided to users through one-on-one sessions. These sessions catered for individual projects, ensuring that scientists received relevant assistance specific to the challenges they need addressing.
4. Running Fortnightly Hacky Hour Sessions: To foster a community and facilitate knowledge sharing, fortnightly “Hacky Hour” sessions were introduced. These informal gatherings allowed users to ask questions, share experiences, and collaboratively solve problems related to the HPC platform.

Jiaxin presented the outcome of the SDC user support program at the NSW DCCEEW Remote Sensing and Landscape Science Branch Day, June 2024

Outcome

The initiative was well received by both the SDC management team and the SDC user group. Scientists reported greater confidence in using the HPC platform and a significant reduction in the time required to complete computational tasks. The community-building efforts through Hacky Hour sessions fostered a collaborative environment, encouraging peer-to-peer support and continuous learning. Dr Tariq emphasises,

“I think we were able to tackle the problem head-on with this muti-pronged approach. Offering courses was not going to be enough without the relevant context and follow up support. The combined effort has helped improve user experience and confidence.“

Recognising the value of the support and training program, the Science HPC team have decided to extend the initiative into the 2024-2025 financial year, ensuring ongoing support for its user community and sustained enhancement of scientific capabilities.