The necessity of external validation in exhaled breath research: a case study of sarcoidosis

As in other disciplines of 'omics' research, reproducibility is a major problem in exhaled breath research. Many studies report discriminatory volatiles in the same disease, yet the similarity between lists of identified compounds is low. This can occur due to many factors including the lack of internal and, in particular, external validation. In an ideal situation, an external validation-sampled at, for example, a different location-is always included to ensure generalization of the observed findings to a general population. In this study, we hypothesized that sarcoidosis patients and healthy controls could be discriminated based on a group of volatile organic compounds (VOCs) in exhaled breath and that these discriminating VOCs could be validated in an external population. The first dataset consisted of 87 sarcoidosis patients and 27 healthy controls, whereas the validation dataset consisted of 25 patients and 29 controls. Using the first dataset, nine VOCs were found that could predict sarcoidosis with 79.4% accuracy. Different types of internal and external validation were tested to assess the validity of the nine VOCs. Of the internal validations, randomly setting aside part of the data achieved the most accurate predictions while external validation was only possible by building a new prediction model that yielded a promising yet not entirely convincing accuracy of 74% due to the indirect approach. In conclusion, the initial results of this study are very promising but, as the results of our validation set already indicated, may not be reproducible in other studies. In order to achieve a reliable diagnostic breath fingerprint for sarcoidosis, we encourage other scientists to validate the presented findings.

NCT00741572 & NCT02361281.