When the Algorithm Speaks First: Trust, Reliance and the Reconstruction of Clinical Metacognition

This paper addresses three of the aims of the Journal: clinical decision making, health philosophy and person-centred healthcare. Retinal imaging serves as the analytic case. The question at issue, what epistemic attitude a clinician may properly hold towards an algorithmic output issued for an individual patient, arises in every discipline and every health system in which such systems are deployed, and the argument is tested throughout against systems that read no images.

Diagnostic artificial intelligence increasingly delivers a conclusion before the clinician has formed one, and medical education has responded by adding modules on the critical appraisal of algorithmic output. Such curricula rest upon two unsound foundations: they extend a form of metacognitive instruction shown not to improve diagnostic accuracy and they ask clinicians to cultivate an attitude whose defining feature is the relaxation of validation. This paper establishes what epistemic attitude a clinician may hold towards an algorithmic output and derives the form that clinical metacognition must take in consequence.

Conceptual analysis, drawing upon the philosophy of trust and epistemic dependence, the cognitive science of clinical reasoning and empirical studies of artificial intelligence in clinical practice. Retinal medicine serves as the analytic case, chosen because it is the domain in which the position under contest is strongest; the argument is tested throughout against systems that process no images.

Trust and reliance are distinct attitudes. On the account adopted here, to trust a medical algorithm is to rely upon it while relaxing the monitoring of the conditions that make it reliable; trust is reliance from which validation has been withdrawn. Regulatory clearance, institutional adoption and published performance supply legitimate grounds for trusting the assemblage that produces a system, yet none of them, singly or together, warrants reliance upon its output for an individual patient. Three asymmetries separate algorithmic from human epistemic dependence: the algorithm can be neither interrogated nor held to account; its errors fall on whole classes of patients at once; and its warrant holds of a population while the decision is owed to a person. The metacognition this situation demands is knowledge, not disposition, and stands untouched by the evidence against generic metacognitive instruction.

Clinical metacognition requires reconstruction around three competences: calibrated reliance, awareness of distributed epistemic labour, and vigilance towards confidence-masked uncertainty. Each is content-dependent, teachable and assessable. Since none can be exercised where institutions supply neither the means of interrogation nor the time it takes, the reform of education and the reform of clinical governance are, in this respect, a single undertaking.