TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19

Area of Science:

• Information Retrieval
• Medical Informatics
• Pandemic Response

Background:

• The COVID-19 pandemic necessitated specialized information retrieval (IR) approaches.
• Traditional IR evaluation methods required adaptation for pandemic-specific research needs.

Purpose of the Study:

• To describe the development and considerations of the TREC-COVID shared task.
• To outline how IR systems were adapted for pandemic information needs.

Main Methods:

• Examined 9 fundamental IR research questions pertinent to pandemic scenarios.
• Modified traditional IR shared task evaluations, considering users, modalities, topics, participants, assessment, relevance, metrics, timelines, and data use.
• Documented the process of addressing these specific requirements.

Main Results:

• TREC-COVID successfully adapted IR evaluation for pandemic research.
• Initial participation numbers indicated significant community engagement with the task.

Conclusions:

• The TREC-COVID task provided a framework for developing effective IR systems during health crises.
• The effort highlighted the IR community's commitment to supporting pandemic research.