FPGA-accelerated streaming data reduction achieving an average compression ratio over 8000 in a 17.4 kHz, 840 kpixel...

We present a data-acquisition and -analysis framework for quasi-elastic gamma-ray scattering (QEGS) experiments at BL35XU of SPring-8, equipped with an 840 kpixel CITIUS X-ray detector operating at 17.4 kHz. The detector produces data at 27 GB s^-1 (216 Gbps), and typical experiments involve acquisition over beam-time periods longer than 24 h, generating datasets of 2.3 PB per day. To handle this volume, we constructed a data-handling pipeline consisting of the detector, data reduction at the beamline and analysis tools at the data center. The data reduction employs field-programmable gate array (FPGA)-accelerated per-pixel processing to reduce data entropy, followed by Zstandard compression on CPUs, achieving an average compression ratio of over 8000. The compressed data are transferred to the SPring-8 data center within two to three minutes of data acquisition. At the data center, analysis tools are provided via the Open OnDemand platform, enabling incremental integration and spectral analysis through a web-based interface without the need for high-performance-computing command-line interaction. This data-handling pipeline has been applied in QEGS user experiments, where it enabled timely feedback on experimental data, with integrated results available within six minutes and spectral analysis within seven minutes of integration.