Related Experiment Videos
Towards new measures of information retrieval evaluation
W R Hersh1, D L Elliot, D H Hickam
1Oregon Health Sciences University, Portland.
Proceedings. Symposium on Computer Applications in Medical Care
|January 1, 1994
Summary
A new short-answer test method effectively measures information acquisition from information retrieval (IR) systems. This approach shows promise for evaluating how well users obtain information, especially in medical education.
Area of Science:
- Information Science
- Medical Education
- Human-Computer Interaction
Background:
- Current information retrieval (IR) system assessment methods have limitations in measuring user information acquisition.
- Evaluating the effectiveness of IR systems in enabling users to obtain information is crucial.
Purpose of the Study:
- To introduce and validate a novel approach for assessing information obtained from IR systems.
- To evaluate the efficacy of this new method using senior medical students.
Main Methods:
- A short-answer test was administered to senior medical students.
- Students used one of two IR systems to answer questions they were uncertain about.
- Performance was assessed based on pre- and post-search test scores.
Main Results:
- Initial test scores indicated low certainty among students before searching.
- Information retrieval system usage resulted in a high proportion of correctly answered questions.
- The new assessment method demonstrated its capability to measure information acquisition.
Conclusions:
- The developed short-answer test is a viable method for assessing information obtained from IR systems.
- This approach can effectively measure user success in acquiring information.
- The method will be employed in future studies to compare different IR systems.