SDC Kubernetes Cluster Re-launch and Support

Background

The Intersect engineers are engaged with NSW Department of Climate Change, Energy, the Environment and Water (the department) Climate and Atmospheric Science and Science Data Compute Staff and the NSW Department of Planning, Housing and Infrastructure (NSW DPHI) Digital Information Office (DIO) staff to continuously support the release of iteratively produced NSW and Australian Regional Climate Modelling (NARCliM) project climate projections generations (NARCliM2.0) on NSW Climate Data Portal (the Portal) while this web application progressed from beta stage to production stage.

While supporting this transition process, all the backend services of the Portal (internal services), THREDDS Data server (TDS) and SDC API (data processing and transferring services) needed to be reviewed, adjusted and migrated to the new clusters in alignment with the current security and application requirements from all stakeholders. 

Where we were

  • NSW Climate Data Portal web application was publicly released in September 2025.
  • The production version of Portal needed to be deployed within a DIO managed AWS environment meeting their governance mandate.
  • SDC infrastructures for Kubernetes have evolved over the years in parallel as new NARCliM generations were prepared.

How we supported the Re-launch

  • Make sure the architecture design is implemented correctly and communicate important changes and bugs to stakeholders before adjusting to accommodate any technical changes during the transition.
  • Uphold a stable SDC API specification and consistent code implementation, so the dataset subsetting and uploading workflows maintain efficient and consistent performance servicing all the NARCliM data requests.
  • Tested and refined the Kubernetes templates for deployment on SDC infrastructures against newly established Kubernetes clusters.

What we offer

We Intersect engineers offer a wide spectrum of technical support on platforms such as AWS and Kubernetes. Particularly, we like to highlight the following expertise for this project:

  • Data API design and implementation
  • Technical gap analysis and management
  • AWS services integration
  • Rancher Kubernetes design and implementation
  • Networking & Application security 
  • Continuous improvement for an integrated system

Outcome

  • Successful go-live of NSW Data Climate Portal production site: https://climatedata.environment.nsw.gov.au
  • Successfully verification of the end-to-end workflows for all deployments of the latest NSW Data Climate Portal backend services on SDC established Kubernetes clusters to DIO hosted AWS instances

Summary

Intersect served as the technical engine behind the successful transition of the NSW Climate Data Portal from a beta pilot to a robust, high-performance production platform. By modernising complex backend services and harmonising data workflows with enterprise-grade security standards, we ensured the seamless integration of NARCliM2.0 climate projections into a stable, cloud-based environment. This collaboration culminated in a successful live launch, providing a verified, end-to-end system that delivers reliable and secure climate intelligence to the public and government stakeholders alike.

How Intersect can empower your research:

Our Services

  • Analytics Services: Delivering data science, advanced analytics, artificial intelligence, and statistical solutions to empower research endeavours.
  • Technology Services: Offering digital storage systems, large-scale computing platforms, data services, software, and more to support research needs.
  • Training & Education: Helping researchers develop the essential skills through targeted programs.
  • Research Support: Our Digital Research Analysts (DRAs) work with researchers to find the best technologies and apply their expertise effectively.

Membership

Membership involves a commitment to creating better research capability for the communal benefit of all members. Each member can draw on over 50 years of shared, on-campus experience in digital research technology; backed by research-specific technical specialists in IT systems, operations, and engineering. This suite of services is optimally delivered through an on-campus professional: a core component of Intersect membership.

Training

Intersect Australia offers hands-on training tailored to researchers and higher-degree research (HDR) students, covering a wide range of research-relevant digital tools and technologies, such as Excel, Python, R, SPSS, NVivo, REDCap and Qualtrics. These courses are available at beginner, intermediate, and advanced levels.

All members can access these training sessions. For details on all Intersect training courses across all member universities, click here.