TERSE/PROLIX (TRPX) - a new algorithm for fast and lossless compression and decompression of diffraction and cryo-EM...

Area of Science:

• Crystallography
• Data Science
• Biophysics

Background:

• High-throughput crystallography generates massive datasets, posing significant data handling challenges.
• Efficient compression is crucial for storage, transmission, and analysis of crystallographic data.

Purpose of the Study:

• Introduce TERSE/PROLIX (TRPX), a novel lossless compression algorithm tailored for crystallographic diffraction and cryo-electron microscopy (cryo-EM) data.
• Evaluate TRPX's performance against established compression algorithms.

Main Methods:

• Developed TERSE/PROLIX (TRPX), a C++20 lossless compression algorithm.
• Compared TRPX with gzip, bzip2, CBF, Zstandard, LZ4, and HDF5 filters using continuous-rotation electron diffraction and raw cryo-EM data.
• Assessed compression efficiency (rate) and speed.

Main Results:

• TRPX significantly outperformed all tested algorithms in both compression rate and speed.
• TRPX was 60x faster than bzip2 and over 3x faster than LZ4.
• TRPX files are byte-order independent and require minimal memory, enabling hardware implementation.

Conclusions:

• TRPX offers a superior solution for compressing crystallographic and cryo-EM data, enhancing data analysis and interpretation.
• The algorithm addresses storage and transmission concerns effectively.
• TRPX code and associated tools are open-sourced on GitHub.