Evaluation of a large-scale biomedical data annotation initiative

Area of Science:

• Bioinformatics
• Genomics
• Data Science

Background:

• Gene Expression Omnibus (GEO) data requires robust annotation for effective analysis.
• Manual re-annotation using standardized thesauri enhances data quality.
• Developing scalable annotation frameworks is crucial for large biological repositories.

Purpose of the Study:

• To describe a large-scale manual re-annotation of GEO samples.
• To develop and evaluate a flexible, comprehensive, and scalable annotation scheme for diseases.
• To assess the reliability of manual re-annotation through coverage and inter-annotator agreement.

Main Methods:

• Manual re-annotation of 12,500 samples across six disease categories (breast cancer, colon cancer, inflammatory bowel disease, rheumatoid arthritis, systemic lupus erythematosus, Type 1 diabetes mellitus).
• Utilized variables and values derived from the National Cancer Institute thesaurus.
• Evaluated annotation structure by measuring variable coverage and inter-annotator agreement.

Main Results:

• Annotated 12,500 samples with approximately 30 variables per disease category.
• Achieved over 98% coverage for critical variables: disease state, tissue, and sample type.
• Demonstrated 89% strict inter-annotator agreement and 92% agreement with similarity measures.

Conclusions:

• Manual re-annotation of large biological data repositories can be performed reliably.
• The developed framework provides a scalable and effective method for data annotation.
• High inter-annotator agreement validates the quality and consistency of the annotation process.