On the Readiness of Scientific Data Papers for a Fair and Transparent Use in Machine Learning

Area of Science:

• Data Science
• Machine Learning
• Scientific Publishing

Background:

• Growing demand for fair and trustworthy machine learning (ML) systems necessitates comprehensive data documentation.
• Scientific communities increasingly adopt data-sharing practices for reproducibility, publishing data and documentation in data papers.
• Existing data documentation practices may not fully align with the needs of ML applications and regulatory requirements.

Purpose of the Study:

• To analyze the extent to which scientific data papers meet the documentation needs of the machine learning community and regulatory bodies.
• To assess the coverage and trends in data documentation within scientific data papers across various domains.
• To compare the documentation standards in general scientific data papers with those from an ML-specific venue.

Main Methods:

• Analysis of a sample of 4041 data papers from diverse scientific domains.
• Assessment of data coverage and documentation trends relevant to ML applications.
• Comparative analysis with datasets published in an ML-focused venue (NeurIPS D&B).

Main Results:

• Identification of gaps and trends in data documentation within scientific data papers.
• Evaluation of the suitability of current data paper standards for ML use cases.
• Benchmarking against documentation practices in ML-specific dataset publications.

Conclusions:

• Scientific data papers show varying degrees of preparedness for ML applications.
• Recommendations are proposed for data creators and publishers to enhance data documentation.
• Guidelines aim to improve data transparency, fairness, and trustworthiness in ML technologies.