We only see what we already know--a modified Bayes' formula to explain inherent limitations of diagnostic tests

In medical tests involving human judgment, human testers are inclined to diagnose only symptoms and signs that they already know. The present analysis delineates the relationship between prior knowledge and test characteristics. The relationship between test outcome and a given diagnosis is generally described in terms of sensitivity and specificity. In the present analysis, this relationship is broken down into two separate contributions: First, the association between the test measure and the presence of disease and second, the association between the actual test outcome and the presence of the test measure. The second relationship describes the performance of a human tester in assessing the presence of a test measure irrespective of the disease status. Bayes' formula can be adjusted to include contributions by both relationships. The ability to increase a diagnostic probability through testing depends on a tester's familiarity with the test measure and reliability in eliciting its presence. Testing yields the most reliable results if the test measure falls well within a tester's own level of competence. Test measures outside the tester's competence level lead to tests with worse outcome than no testing at all.