A comparison of two methods for indexing and retrieval from a full-text medical database

The objective of this study was to compare how well medical professionals are able to retrieve relevant literature references using two computerized literature searching systems that provide automated (non-human) indexing of content. The first program was SAPHIRE, which features concept-based indexing, free-text input of queries, and ranking of retrieved references for relevance. The second program was SWORD, which provides single-word searching using Boolean operators (AND, OR). Sixteen fourth-year medical students participated in the study. The database for searching was six volumes from the 1989 Yearbook series. The queries were ten questions generated on teaching rounds. All subjects searched half the queries with each program. After the searching, each subject was given a questionnaire about prior experience and preferences about the two programs. Recall (proportion of relevant articles retrieved from the database) and precision (proportion of relevant articles in the retrieved set) were measured for each search done by each participant. Mean recall was 57.6% with SAPHIRE; it was 58.6% with SWORD. Precision was 48.1% with SAPHIRE vs 57.6% with SWORD. Each program was rated easier to use than the other by half of the searchers, and preferences were associated with better searching performance for that program. Both systems achieved recall and precision comparable to existing systems and may represent effective alternatives to MEDLINE and other retrieval systems based on human indexing for searching medical literature.