By Glen Charlton
Background:
The Climate Branch of NSW Department of Primary Industries (DPI) engaged Intersect to collaborate on the development of an automated data processing workflow. The aim was to process sensor data and provide insights for end-users of the Seasonal Conditions Monitoring Network (SCMN), which is part of the Enhanced Drought Indication Scheme (EDIS)
Dr Ian Marang (Remote Sensing Specialist at Climate Applications, DPI) worked in collaboration with Glen Charlton (Research Data Scientist at Intersect’s Advanced Analytics and AI Platform) on this project. According to Dr. Marang,
”Glen and Anastasios from Intersect’s 3AI team were professional, engaged and knowledgeable on all points of the project delivery pipeline. They identified a solution from our general objective and laid out a workplan and reasonable budget to accomplish it.”
The pilot SCMN network consists of several farms across NSW equipped with sensor arrays to collect weather and soil conditions data. Data is wirelessly transmitted from the farms, and with the support of Intersect’s Software Engineering team, the agreed solution consists of an automated sensor data collection, processing (including but not limited to cleaning, validation check, quality control, and transformations), storage, and an interactive real-time dashboard for data visualisation and reporting.
Challenge:
The Future Ready Regions EDIS Development (FRR-ED) project team at the NSW DPI Climate Branch faced the challenge of automating real-time sensor data processing, storage and visualisation of weather and soil conditions. Dr Marang explains,
“When collecting data remotely across Australia’s wide expanse, reliably communicating over the limited available infrastructure is challenging. With our in-house deployment team’s expertise in telemetry and sensor design, allied with Intersect’s 3AI team we have built a robust solution for collecting research-grade data remotely with solid ingestion, QA, extraction processes.”
Solution:
The solution involved an automated data processing workflow that utlises custom Python functions powered by Apache Airflow to process incoming sensor data and store it in TimescaleDB. Meta-data for the sensor data is stored in PostgreSQL tables alongside the TimescaleDB hypertable. This enables in-depth analysis, visualisation and reporting of the sensor data using Google Looker Studio.
Apache Airflow is used to develop the workflow by defining each step as a task (or Python function) arranged in a directed acyclic graph (DAG) to create a workflow that runs automatically. Airflow’s scheduler ensures that the tasks are executed in the correct order and that the workflow runs efficiently. The scheduler is triggered by Intersect’s Databolt to collect the data from the DPI communication nodes for storage and processing within a specialised virtual machine. Additionally, Apache Airflow offers a suite of tools for monitoring, logging, and debugging the workflow. This makes it easier for the researchers at the DPI to monitor the whole process using the web-based user interface alongside email alerts.
Finally, a custom dashboard was developed using Google Looker Studio to display the processed sensor data. The dashboard provides real-time insights into weather and soil conditions at each of the sites where sensors are deployed, increasing the engagement with farmers and land holders who use the system.
Result:
This end-to-end solution has allowed researchers to increase engagement with participants (farmers and land-holders) and has provided a rich dataset ready for further analysis and research as part of the broader EDIS project. Dr Marang emphasises,
“Getting the data for our Science team to improve our models of drought, agronomic stress and yield is only half the job. Another critical component to this project’s success is the land-holder’s seeing value in using AgTech to make better business decisions. When a farmer can open up their phone or computer and see how much moisture their paddock has, what the temperature or humidity is and compare it to historical trends they can start to plan their operations more effectively, building their and our state’s resilience in the face of future climate extremes.”
