Formative evaluation of a prototype system for automated analysis of mass spectrometry data

Mass Spectrometry (MS) is emerging as a breakthrough mass-throughput technology capable of producing powerful clinical diagnostic and prognostic models and of identifying important disease biomarkers. Few individuals possess the necessary skills to carry out MS analyses competently, and access to such individuals is limited in most settings, hindering progress in this field. We seek to ease this burden by creating a fully automated system (FAST-AIMS) capable of analyzing mass spectra to produce high-quality diagnostic and outcome prediction models and identify related biomarkers. In the present report we introduce the system and conduct a formative evaluation in which 6 users apply it to a challenging dataset. FAST-AIMS' performance is compared to that of an expert statistician as well as to a previously published analysis by an independent group. In our experiments FAST-AIMS when used by both MS-sophisticated users (n=4) and naïve users (n=2) achieves performance (a) comparable to our human expert, and (b) superior to the previously published manual analysis; in addition (c) the system's estimates future performance accurately, thus avoiding overfitting.