Best practices for high data-rate macromolecular crystallography (HDRMX).

Area of Science:

• Macromolecular crystallography
• X-ray science
• Data science

Background:

• Advancements in X-ray sources (synchrotrons, X-ray free-electron lasers) and detectors enable experiments with numerous small samples.
• These experiments can reveal crucial information about polymorphs and molecular dynamics.
• Current data handling methods are insufficient for the increased volume and complexity of experimental data.

Purpose of the Study:

• To propose re-engineered approaches for image clustering in macromolecular crystallography.
• To address the challenges of managing large datasets and metadata from modern X-ray experiments.
• To present solutions for efficient data validation and analysis.

Main Methods:

• Discussing improved algorithms for image clustering to support polymorph and dynamics studies.
• Examining recent and upcoming changes in metadata standards for Eiger detectors.
• Introducing software for rapid validation of images in the updated Eiger format.

Main Results:

• The transition to image container systems like HDF5 is favored over traditional file systems due to data volume.
• New metadata formats and validation tools are essential for handling increased data loads.
• Optimized image clustering is critical for extracting polymorph and dynamics information.

Conclusions:

• Re-engineering data processing and management is crucial for leveraging advanced capabilities in macromolecular crystallography.
• Efficient metadata handling and image validation are key to successful high-throughput experiments.
• The proposed approaches facilitate the analysis of small sample datasets for polymorph and dynamics research.