Enhancing Cancer Research Efficiency: CellProfiler on an HPC Cluster

Abdullah Shaikh | Feb 28th 2024

The Cancer Systems Microscopy (CSM) Lab at the University of New South Wales focuses on improving cancer treatment by researching precision diagnostics, targeted therapies and fundamental insights. Their research involves analysing thousands of cell images, and they use an open-source software, CellProfiler, for this purpose. 

The lab uses the Graphical User Interface (GUI) version of CellProfiler on their local machines (laptops or computers) for image analysis. However, this approach requires leaving small-scale computers running for extended periods—sometimes up to 30 hours—to complete the analysis and get results. This process is not only time-consuming but also inefficient and laptop-dependent. The researchers reported:

“There have been some times where our machines are busy for more than 30hrs. We’ve had jobs that run for days (and even months though these jobs are far rarer)”. 

To address these challenges, we collaborated with the CSM Lab to help them run their analysis on a large-scale computing cluster, Katana, the local HPC Cluster of UNSW. This involved installing the CellProfiler modules on Katana and drafting scripts to ensure the execution of their analysis headlessly in a command-line environment. Researchers can now submit their images and pipelines as batch jobs. They can take advantage of the HPC’s queuing system and efficiently utilise the computational resources to speed up their analysis.

According to Andrew Gunawan, a researcher at the CSM Lab, headed by Dr John Lock, the benefits are clear:

“Having CellProfiler pipelines run on Katana is great. It allows our jobs to run in the background on Katana (our local HPC Cluster). The new workflow has freed up our laptops for other tasks and increased efficiency”.

This project involved training and support to the researchers to ensure that all their CellProfiler pipelines and plugins integrate into the HPC cluster.