Conducting the NLM/AHCPR Large Scale Vocabulary Test: a distributed Internet-based experiment

The Large Scale Vocabulary Test, sponsored by the National Library of Medicine (NLM) and the Agency for Health Care Policy and Research (AHCPR), was conducted to determine the extent to which a combination of existing health-related terminologies cover vocabulary needed in health care information systems. The test was conducted over the Internet using a sophisticated World Wide Web interface with over 60 participants and over 40,000 terms submitted. This paper discusses the issues encountered in the design and execution of the experiment, including the design of the interface and the issues of recruitment, training, and guidance of remote participants over the Internet. Test data are currently undergoing expert review. Upon completion of the expert review, the results of the test will be fully reported.