A systematic review of technical evaluation in telemedicine systems

We conducted a systematic review of the literature to critically analyse the evaluation and assessment frameworks that have been applied to telemedicine systems. Subjective methods were predominantly used for technical evaluation (59 %), e.g. Likert scale. Those including objective measurements (41%) were restricted to simple metrics such as network time delays. Only three papers included a rigorous standards based objective approach. Our investigation has been unable to determine a definitive standards-based telemedicine evaluation framework that exists in the literature that may be applied systematically to assess and compare telemedicine systems. We conclude that work needs to be done to address this deficiency. We have therefore developed a framework that has been used to evaluate videoconferencing systems telemedicine applications. Our method seeks to be simple to allow relatively inexperienced users to make measurements, is objective and repeatable, is standards based, is inexpensive and requires little specialist equipment. We use the EIA 1956 broadcast test card to assess resolution, grey scale and for astigmatism. Colour discrimination is assessed with the TE 106 and Ishihara 24 colour scale chart. Network protocol analysis software is used to assess network performance (throughput, delay, jitter, packet loss).