Shazam! Data Teleporting at La Trobe

Craig Hamilton
10 Mar 2019

MELBOURNE, Victoria - La Trobe University Institute for Molecular Science 'teleports' serial crystallography data into Space with Shazam!

Around Christmas 2018, when research activities tend to wind down for the holidays, in Melbourne at La Trobe University and in the Intersect Sydney office, data began moving faster.

Dr. Connie Darmanin‘s team of experimental physics scientists captures serial crystallography data at very fast speeds and – amongst many other outputs – generates molecular movies of proteins. Experimental data comes from sources including Australian and international synchrotrons and X-ray Free Electron Laser (XFEL) facilities.

The challenge is that this data is highly variable, voluminous, and its acquisition can be long running. This is compounded by some source instruments being offsite and direct transfer of data from these facilities is time consuming, resulting in batches of data arriving on physical removable storage. It’s also fragile in that missing files or observations can compromise terabytes of results and require repeating a physical experiment, consuming more precious instrument time. Past manual methods of acquiring, transferring and cataloging the data had been tedious at best, and physical storage methods were not meeting researcher access or risk management requirements.

Fortunately, with the foresight of CIO Peter Powell, leadership of John Parry, and teaming with Manager, Communications and Security Jason Schimming and Monitoring Specialist Michael Filippidis, La Trobe had already commissioned Intersect to address research data capture, transmission, storage and access holistically a few months earlier. The result was a collaborative effort to enhance and deploy the Shazam! Data Teleporter service along with Space storage and Time compute.

In December, on-campus Intersect eResearch Analyst Dr. Ghulam Murtaza led a blended La Trobe/Intersect team to deploy Shazam! in the La Trobe Institute for Molecular Science (LIMS). Over Christmas it began acquiring over 15TB of molecular data in thousands of files from many external USB disks. Consisting of historical data from the Australian and international Synchrotrons, plus various XFEL sources, every last byte of data was captured, safely transmitted via the Internet, validated for assurance, and stored. This data is now held securely in Intersect Space, and available as needed to analytics system hosted in Time. Because this is directly related to observations, batches and experiments – not just filesystems – and because Shazam! automates so many tedious manual processes and points where errors could be introduced, we call this “data teleporting”.

Today, Dr Connie Darmanin’s team realises more benefits than just those from teleporting and peace of mind.  Instead of swapping among unreliable, clumsy and slow USB drives to access experimental data, every single observation is now available in real time, at the same time. Analytics software in the cloud has access to all of this data, and this is available to any team member, anywhere, any time. 

Shazam! can teleport data from any source, including direct-from-instrument, and is not limited to organisational boundaries. In the future the combined La Trobe and Intersect team will look to explore deploying Shazam! onsite at the Australian Synchrotron, creating a generic and reusable data teleporting model for other researchers and scientists. Beyond serial crystallography, the La Trobe University ICT team has plans to expand Shazam! to other research projects and instruments.

For more information about Shazam! or to talk about your research storage protocols please get in touch with Craig.

Craig Hamilton

Engineering Manager
T: 02 8079 2500
M: 0422 389 590
craig.hamilton@intersect.org.au
Back to news
Your browser is not supported. Please upgrade your browser.