The Cherenkov Telescope Array Observatory (CTAO) is the next-generation atmospheric Cherenkov gamma-ray project. CTAO will be deployed at two sites, one in the Northern and the other in the Southern Hemisphere, containing telescopes of three different sizes for covering different energy domains. The commissioning of the first CTAO Large-sized Telescope (LST-1) is being finalized at the CTAO Northern site. Additional calibration and environmental monitoring instruments such as laser imaging detection and ranging (LIDAR) instruments and weather stations will support the telescope operations. The Array Control and Data Acquisition (ACADA) system is the central element for onsite CTAO operations. ACADA controls, supervises, and handles the data generated by the telescopes and the auxiliary instruments. It will drive the efficient planning and execution of observations while handling the several Gb/s camera data generated by each CTAO telescope. The ACADA system contains the CTAO Science Alert Generation Pipeline – a real-time data processing and analysis pipeline, dedicated to the automatic generation of science alert candidates as data are being acquired. These science alerts, together with external alerts arriving from other scientific instruments, will be managed by the Transients Handler (TH) component. The TH informs the Short-term Scheduler of ACADA about interesting science alerts, enabling the modification of ongoing observations at sub-minute timescales. The capacity for such fast reactions – together with the fast movement of CTAO telescopes – makes CTAO an excellent instrument for studying high-impact astronomical transient phenomena. The ACADA software is based on the Alma Common Software (ACS) framework, and written in C++, Java, Python, and Javascript. The first release of the ACADA software, ACADA REL1, was finalized in July 2023, and integrated after a testing campaign with the LST-1 finalized in October 2023. This contribution describes the design and status of the ACADA software system.
The Cherenkov Telescope Array Observatory (CTAO) is the next-generation atmospheric Cherenkov gammaray Observatory. CTAO will be constructed on two sites, one array in the Northern and the other in the Southern hemisphere, containing telescopes of three different sizes, for covering different energy domains. To combine and orchestrate the different telescopes and auxiliary instruments (array elements), the Array Control and Data Acquisition (ACADA) system is the central element for the Observatory on-site operations: it controls, supervises, and handles the data generated by the array elements. Considering the criticality of the ACADA system for future Observatory operations, corresponding quality assurance provisions have been made at the different steps of the software development lifecycle, with focus on continuous integration and testing at all levels. To enable higher-level tests of the software deployed on a distributed system, an ACADA test cluster has been set up to facilitate testing and debugging of issues in a more realistic environment. Furthermore, a separate software integration and test cluster has also been established that allows for the off-site testing of the integrated software packages of ACADA and of the corresponding array elements. Here the software integration can be prepared, interfaces and interactions can be tested, and on-site procedures that are required later in the process can be checked beforehand, only limited by the simulation capabilities that are delivered as part of the software packages. Once preparations and testing with the off-site test cluster are completed, the integrated software can be deployed at the target site. The software packages and setup parameters are kept under configuration control at all stages, and deployment steps are documented to ensure that installations are reproducible. This methodology has been applied for the first time in the context of the integration of ACADA with the first CTAO Large-sized Telescope (LST-1) in October 2023.
KEYWORDS: Atmospheric Cherenkov telescopes, Control software, Software development, Telescopes, Observatories, Data acquisition, Astronomy, Lanthanum, Interfaces, Control systems
The Cherenkov Telescope Array Observatory (CTAO) is the next-generation ground-based instrument for gamma-ray astronomy. CTAO will be located at two sites, one in the Northern (La Palma, Spain) and the other in the Southern Hemisphere (Paranal, Chile), with telescopes in three different sizes to cover different energy ranges. The commissioning of the first CTAO Large-Sized Telescope (LST-1) is being finalized at the CTAO-North site. The Array Control and Data Acquisition (ACADA) software is a central element of on-site CTAO operations. ACADA comprises subsystems for central control, the short-term scheduler, monitoring systems, and data handling at rates of GB/s. Consequently, it is a very complex software that requires many developers with different expertise, such as control software, data acquisition, data analysis, scheduling, configuration, and human interfaces. To implement such complex software, ACADA has been broken down into subsystems, which CTAO delegates to expert developer teams around the world through in-kind contributions. All the software is under version control exploiting a dedicated installation of GitLab. We have created at least one repository for each subsystem and a final one for the integration. We have defined the software development and integration procedures so that all phases of the Software Development Life Cycle (SDLC) are supported. Particular attention has been paid to the critical time when a software version is in operation on site and, bug-fixing and new features need to be kept under version control in parallel. The goal is to manage bug fixes without adding new features out of the scope of the release, but at the same time to guarantee the distribution of bug fixes for future releases. This contribution presents our strategy to manage multiple software versions according to the CTAO development plan.
To achieve the data quantity and quality required by increasingly demanding science drivers astronomical facilities have grown in size and complexity. This trend not only creates new challenges for technological aspects, but also creates the need for an advanced operations management approach to effectively and efficiently operate and maintain these large facilities throughout their life-cycle. In the context of astronomical observatories, operations management usually involves processes that are related to science operations, maintenance management, obsolescence, upgrades, enhancement, quality assurance, planning, and performance monitoring, among others. Starting with the experience acquired at the ALMA observatory, this paper presents the authors' thoughts on new factors and on apparent differences with respect to operations management of traditional large observatory facilities; and how these new challenges could be addressed via best practices and what the related key concepts are. The methodology used is to first identify those areas that contribute to increased complexity or more stringent operational constraints, in order to elaborate on possible resolution strategies. Rather than aiming at delivering turnkey solutions, this paper is intended to explore the interest within the community to gather and validate related operations management concepts and "best practices" in a collaborative manner.
The problem reporting and tracking system (PRTS) is the ALMA system to register operational problems, track unplanned corrective operational maintenance activities and follow the investigations of all problems or possible issues arisen in operation activities. After the PRTS implementation appeared several issues that finally produced a lack in the management of the investigations, problems to produce KPIs, loss of information, among others. In order to improve PRTS, we carried out a process to review the status of system, define a set of modifications and implement a solution; all according to the stakeholder requirements. In this work, we shall present the methodology applied to define a set of concrete actions at the basis of understanding the complexity of the problem, which finally got to improve the interactions between different subsystems and enhance the communication at different levels.
The Atacama Large Millimeter/submillimeter Array (ALMA) is a joint project between astronomical organizations in Europe, North America, and East Asia, in collaboration with the Republic of Chile. ALMA consists of 54 twelve-meter antennas and 12 seven-meter antennas operating as an aperture synthesis array in the (sub)millimeter wavelength range. Since the inauguration of the observatory back in March 2013 there has been a continuous effort to establish solid operations processes for effective and efficient management of technical and administrative tasks on site. Here a key aspect had been the centralized maintenance and operations planning: input is collected from science stakeholders, the computerized maintenance management system (CMMS) and from the technical teams spread around the world, then this information is analyzed and consolidated based on the established maintenance strategy, the observatory long-term plan and the short-term priorities definitions. This paper presents the high-level process that has been developed for the planning and scheduling of planned- and unplanned maintenance tasks, and for site operations like the telescope array reconfiguration campaigns. We focus on the centralized planning approach by presenting its genesis, its current implementation for the observatory operations including related planning products, and we explore the necessary next steps in order to fully achieve a comprehensive centralized planning approach for ALMA in steady-state operations.
The Atacama Large Millimeter/submillimeter Array (ALMA) is a joint project between astronomical organizations in
Europe, North America, and East Asia, in collaboration with the Republic of Chile. ALMA consists of 54 twelve-meter
antennas and 12 seven-meter antennas operating as an aperture synthesis array in the (sub)millimeter wavelength range.
Assembly, Integration, and Verification (AIV) of the antennas was completed at the end of the year 2013, while the final
optimization and complete expansion to validate all planned observing modes will continue. This paper compares the
actually obtained results of the period 2008-2013 with the baselines that had been laid out in the early project-planning
phase (2005-2007).
First plans made for ALMA AIV had already established a two-phased project life-cycle: phase 1 for setting up
necessary infrastructure and common facilities, and taking the first three antennas to the start of commissioning; and
phase 2 focused on the steady state processing of the remaining units. Throughout the execution of the project this lifecycle
was refined and two additional phases were added, namely a transition phase between phases 1 and 2, and a closing
phase to address the project ramp-down. A sub-project called Accelerated Commissioning and Science Verification (ACSV)
was carried out during the year 2009 in order to provide focus to the whole ALMA organization, and to
accomplish the start-of-commissioning milestone. Early phases of CSV focused on validating the basic performance and
calibration. Over time additional observing modes have been validated as capabilities expanded both in hardware and
software.
This retrospective analysis describes the originally presented project staffing plans and schedules, the underlying
assumptions, identified risks and operational models, among others. For comparison actual data on staffing levels, the
resultant schedule, additional risks identified and those that actually materialized, are presented. The observed
similarities and differences are then analyzed and explained, and corresponding lessons learned are presented.
The Atacama Large Millimeter/submillimeter Array (ALMA) is a joint project between astronomical organizations in
Europe, North America, and East Asia, in collaboration with the Republic of Chile. ALMA will consist of at least 54
twelve-meter antennas and 12 seven-meter antennas operating as an aperture synthesis array in the (sub)millimeter
wavelength range. It is the responsibility of ALMA AIV to deliver the fully assembled, integrated, and verified antennas
(array elements) to the telescope array.
After an initial phase of infrastructure setup AIV activities began when the first ALMA antenna and subsystems became
available in mid 2008. During the second semester of 2009 a project-wide effort was made to put in operation a first 3-
antenna interferometer at the Array Operations Site (AOS). In 2010 the AIV focus was the transition from event-driven
activities towards routine series production. Also, due to the ramp-up of operations activities, AIV underwent an
organizational change from an autonomous department into a project within a strong matrix management structure.
When the subsystem deliveries stabilized in early 2011, steady-state series processing could be achieved in an efficient
and reliable manner. The challenge today is to maintain this production pace until completion towards the end of 2013.
This paper describes the way ALMA AIV evolved successfully from the initial phase to the present steady-state of array
element series processing. It elaborates on the different project phases and their relationships, presents processing
statistics, illustrates the lessons learned and relevant best practices, and concludes with an outlook of the path towards
completion.
S. Asayama, L. B. Knee, P. Calisse, P. Cortés, R. Jager, B. López, C. López, T. Nakos, N. Phillips, M. Radiszcz, R. Simon, I. Toledo, N. Whyborn, H. Yatagai, J. McMullin, P. Planesas
The Atacama Large Millimeter/submillimeter Array (ALMA) will consist of at least 54 twelve-meter antennas and 12
seven-meter antennas operating as an aperture synthesis array in the (sub)millimeter wavelength range. The ALMA
System Integration Science Team (SIST) is a group of scientists and data analysts whose primary task is to verify and
characterize the astronomical performance of array elements as single dish and interferometric systems. The full set of
tasks is required for the initial construction phase verification of every array element, and these can be divided roughly
into fundamental antenna performance tests (verification of antenna surface accuracy, basic tracking, switching, and on-the-fly rastering) and astronomical radio verification tasks (radio pointing, focus, basic interferometry, and end-to-end
spectroscopic verification). These activities occur both at the Operations Support Facility (just below 3000 m elevation)
and at the Array Operations Site at 5000 m.
The Atacama Large Millimeter/submillimeter Array (ALMA) is a joint project between astronomical organizations in
Europe, North America, and East Asia, in collaboration with the Republic of Chile. ALMA will consist of at least 54
twelve-meter antennas and 12 seven-meter antennas operating as an interferometer in the millimeter and sub-millimeter
wavelength range. It will be located at an altitude above 5000m in the Chilean Atacama desert. As part of the ALMA
construction phase the Assembly, Verification and Integration (AIV) team receives antennas and instrumentation from
Integrated Product Teams (IPTs), verifies that the sub-systems perform as expected, performs the assembly and
integration of the scientific instrumentation and verifies that functional and performance requirements are met. This
paper aims to describe those aspects related to the AIV Engineering team, its role within the 4-station AIV process, the
different phases the group underwent, lessons learned and potential space for improvement.
AIV Engineering initially focused on the preparation of the necessary site infrastructure for AIV activities, on the
purchase of tools and equipment and on the first ALMA system installations. With the first antennas arriving on site the
team started to gather experience with AIV Station 1 beacon holography measurements for the assessment of the overall
antenna surface quality, and with optical pointing to confirm the antenna pointing and tracking capabilities. With the
arrival of the first receiver AIV Station 2 was developed which focuses on the installation of electrical and cryogenic
systems and incrementally establishes the full connectivity of the antenna as an observing platform. Further antenna
deliveries then allowed to refine the related procedures, develop staff expertise and to transition towards a more routine
production process. Stations 3 and 4 deal with verification of the antenna with integrated electronics by the AIV Science
Team and is not covered directly in this paper. It is believed that both continuous improvement and the clear definition of
the AIV 4-station model were key factors in achieving the goal of bringing the antennas into a state that is well enough
characterized in order to smoothly start commissioning activities.
The Atacama Large Millimeter Array (ALMA) is a joint project between astronomical organizations in Europe, North
America, and Japan. ALMA will consist of at least 50 twelve meter antennas operating in the millimeter and submillimeter
wavelength range. It will be located at an altitude above 5000m in the Chilean Atacama desert. The ALMA
Test Facility (ATF), located in New Mexico, USA, is a proving ground for development and testing of hardware,
software, commissioning and operational procedure.
At the ATF emphasis has shifted from hardware testing to software and operational functionality. The support of the
varied goals of the ATF requires stable control software and at the same time flexibility for integrating newly developed
features. For this purpose regression testing has been introduced in the form of a semi-automated procedure. This
supplements the established offline testing and focuses on operational functionality as well as verifying that previously
fixed faults did not re-emerge.
The regression tests are carried out on a weekly basis as a compromise between the developers' response- and the
available technical time. The frequent feedback allows the validation of submitted fixes and the prompt detection of sideeffects
and reappearing issues. Results of nine months are presented that show the evolution of test outcomes, supporting
the conclusion that the regression testing helped to improve the speed of convergence towards stable releases at the ATF.
They also provided an opportunity to validate newly developed or re-factored software at an early stage at the test
facility, supporting its eventual integration. Hopefully this regression test procedure will be adapted to commissioning
operations in Chile.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.