The effects and quality of IT evaluation studies: Trends in 1982 - 2002

During the last years the significance of evaluation studies as well as the interest in adequate methods and approaches for evaluation has grown in medical informatics. In order to put this discussion into historical perspective of evaluation research, we conducted a systematic review on trends in evaluation research of information technology in health care from 1982 to 2002. In a first step of our inventory, we concentrated on describing long-term developments e.g. in study questions and methods used of IT evaluation studies as described in the abstracts of the identified 1.035 publications. In the second step of our inventory, we now analysed 64 randomly selected full papers on evaluation studies, in order to answer two questions: Did IT evaluation studies show a positive effect of IT on quality of processes or outcome quality of patient care, and how did the quality of evaluation studies themselves develop in the last 20 years. Appropriateness of the care process was the most common and mostly positively evaluated aspect. There was a trend towards more multi-centre studies. The quality of the studies remained stable over time. Randomized controlled trials had significantly higher quality than non-RCT studies.