Abstract
The Magnetospheric Multiscale (MMS) Science Data Center (SDC) at the
Laboratory for Atmospheric and Space Physics (LASP), at the University
of Colorado, has managed MMS science and ancillary data processing and
distribution since MMS launched in March 2015. The MMS SDC employs
automation in nearly every part of its operations. Automation is used to
start up processing “runners” that listen on queues for new processing
jobs, which are triggered by configurable timing rules, including cron
and operational events, or certain data/data files being available. A
separate set of SDC code then automatically creates processing jobs and
tracks its progress. The MMS SDC runs processing jobs for each
instrument (47 different job types in total), ranging from levels l1a to
l3, for “survey” and “burst” modes, plotting, and cdf creation. The
SDC runs anywhere from a few hundred to over 2,000 jobs per day (on
average, 1,000 jobs per day). Processing jobs are scheduled in a few
different ways, from running based on a fixed schedule in cron, to being
triggered by certain mission events, to being triggered by the
appearance of new files. Several fail-safes have been added into the
code over time to ensure failures are caught and handled, however,
situations do arise where failures occur and are dealt with when
something in the SDC does not work as expected. Added to these
complexities is the fact that the MMS mission is incredibly
time-sensitive, and requires the SDC to be available and ready to handle
issues 24/7/365, which can be challenging due to the limited staffing on
MMS a few years into the mission. The importance of automation in MMS
SDC processing is clear. Not only does automated processing relieve some
of the load from the software engineers working on the SDC, but it
ensures continued smooth operation of the MMS SDC. This then allows
scientists to continue their research efforts unhindered. As time has
progressed, various areas for improvement, and extra automation, in this
process have been implemented. This poster will focus on automation
improvements to keep the system running smoothly with almost no human
involvement.