Speaker
Description
The ASKAP Telescope is a cm-band synthesis imaging telescope operated by CSIRO at Inyarrimanha Ilgari Bundara, the CSIRO Murchison Radio-astronomy Observatory in Western Australia. Its innovative phased-array-feed (PAF) receivers give ASKAP the wide field-of-view that makes it excellent at conducting large-scale surveys of the sky. This comes, however, at the cost of high data rates. High-performance computing is required to create the calibrated images and catalogues quickly enough to keep up with the incoming data rate, and allow the data products to be deposited in the CSIRO ASKAP Science Data Archive (CASDA) for use by astronomers.
The data processing that creates the science products is orchestrated by the ASKAP Pipeline, a highly-parallelised, scripted workflow that leverages the Slurm workload manager to execute all the required calibration, imaging and source-finding tasks. These tasks are realised through the custom-written calibration & imaging package ASKAPsoft, which has been designed to address challenges presented by the scale of data ASKAP produces. This Pipeline is in near constant use, with regular survey observing resulting in large amounts of data being deposited into the CASDA archive and made publicly available. Data rates into the archive can be around 70 TB per week. The Pipeline is nearly autonomous - most observations are processed in a completely hands-off fashion, with only a small fraction requiring manual attention (perhaps due to data quality issues).
This talk will describe the Pipeline and key design decisions that have shaped its workflow, demonstrating its capabilities as a large & complex supercomputing workflow. Building on the experience of many years of operations through pilot and full-scale surveys, this talk will detail the current performance and describe recent additions to the processing that enhance the data quality, while reflecting on lessons learned from managing the large data rates and I/O loads, looking ahead to future large-scale processing for facilities such as the SKA.
