Intersect’s Largest Ever Data Migration Project

by lyndsay Posted on

Under the National Cloud Refresh initiative, Intersect has partnered with the Queensland Cyber Infrastructure Foundation (QCIF) to provide new infrastructure for the Australian Research Data Commons’ (ARDC) Nectar Research Cloud. This will further build capability and capacity, to ensure that our member’s and national researcher’s demands for cloud resources are being met. As a part of this initiative, Intersect has migrated its TIME and SPACE infrastructure from its data centre in Sydney to the QCIF facility.

This migration project commenced in Dec 2020 and thus far remains one of the largest-scale migration projects ever undertaken by Intersect. 

At the beginning of this project, the Intersect Research Cloud hosted the following at its Sydney based data centre: 

  • 3.1 PB of data across 158 data collections
  • 291 Intersect & Member Virtual Machines
  • 137 National Allocation Virtual Machines (non-member) which required migration by the researchers

By the end of the migration project, following an in-depth analysis of all resources,  Intersect has successfully migrated the following to the QCIF facility:

  • 1.7PB of data across 158 data collections
  • 193 Intersect & Member Virtual Machines

In total, the migration ran for 6 months and was successfully concluded by the end of May 2021.

Important tasks undertaken by Intersect to achieve the timely migration of SPACE and TIME include:

  • Analysis and identification of 98 Intersect & Members VMs and 1.4PB of redundant or no longer required data, that could be decommissioned or did not need to be migrated, thereby providing cost savings and further data retention opportunities to our customers.
  • Physical relocation of some Intersect servers to Brisbane and a complete rebuild of these servers in the QCIF Data Centre
  • Removal and resale or responsible e-Wasting of equipment and networking from the Sydney Data Centre which itself took approx 40 effort days- it should also be noted that ITC Asset Management provided seamless and greatly appreciated cooperation with the e-waste disposal. 
  • Consistent and on-message communication with all of our customers, researchers and impacted staff in the run-up to, during, and post-migration
  • Production of user documentation for our customers outlining any new access mechanisms to their data
  • Troubleshooting and managing customer tickets raised during this migration process

Notwithstanding the complexity of moving an entire cloud infrastructure to another Data Centre in another state, the project also had to contend with the following challenges, some of which are unique to the times we are living in:

  • COVID-19 and travel bans thereby restricting staff movements into and out of QLD to install equipment in the new data centre
  • Infrastructure upgrades at the new data centre and working around these upgrades, the new processes, and associated outages when they occurred 

Pleasingly, the migration of our infrastructure to QCIF has resulted in additional unforeseen benefits for our customers. By virtue of analysing and identifying which resources had to migrate to the new data centre, Intersect has freed up between 25-45% of some customer’s SPACE allocations, resulting in cost savings for these customers and further data retention potential for more of our customer’s researchers.

The move to the new data centre has brought many new and exciting opportunities and advantages for our existing and new customers: 

  • Lower costing model due to economy of scale advantages gained from partnering with QCIF
  • Simpler costing models, including ‘Community pricing’ options for members
  • Safer data control, all data now having an off-site copy.
  • New higher-performance compute capability
  • Better data management (Idea)

In conclusion, Intersect looks forward to a dynamic future with our QCIF partner, resulting in further flexibility for our customers in our data storage and virtual computing offerings. 

Back to Articles