A high-throughput big-data orchestration and processing system for the High Energy Photon Source

High-data-throughput and multimodal-acquisition experiments will prevail in next-generation synchrotron beamlines. Orchestrating dataflow pipelines connecting the data acquisition, processing, visualization and storage ends are becoming increasingly complex and essential for enhancing beamline performance. Mamba Data Worker (MDW) has been developed to address the data challenges for the forthcoming High Energy Photon Source (HEPS). It is an important component of the Mamba experimental control and data acquisition software ecosystem, which enables fast data acquisition and transmission, dynamic configuration of data processing pipelines, data multiplex in streaming, and customized data and metadata assembly. This paper presents the architecture and development plan of MDW, outlines the essential technologies involved, and illustrates its current application at the Beijing Synchrotron Radiation Facility (BSRF).