The inconclusive category, entropy, and forensic firearm identification

Area of Science:

• Forensic Science
• Pattern Evidence Analysis
• Machine Learning Applications

Background:

• Recent discussions highlight challenges in estimating error rates for forensic disciplines like firearms examination.
• The President's Council of Advisors on Science and Technology (PCAST) report criticized the lack of error rate studies in many forensic fields.
• Consensus is lacking on measuring error rates in disciplines that include an "inconclusive" category, such as firearm and tool mark examination.

Purpose of the Study:

• To explore methods for measuring error rates in forensic pattern evidence analysis, particularly when an "inconclusive" category is used.
• To adapt error rate calculations from binary decision models to scientific fields with a range of conclusions.
• To present a model system for examining the performance of various error metrics in such fields.

Main Methods:

• Developed and trained three neural networks of varying complexity to classify ejector mark outlines on cartridge cases.
• Utilized ejector marks from firearms as a model system for error rate analysis.
• Discussed an entropy-based method for assessing classification similarity to ground truth, applicable to scales with inconclusive outcomes.

Main Results:

• Neural networks were trained to classify ejector mark patterns, serving as a testbed for error metric evaluation.
• The study demonstrates a model system for analyzing performance metrics in forensic pattern analysis.
• An entropy-based approach was discussed for evaluating classification accuracy in systems with inconclusive results.

Conclusions:

• The study provides a framework for evaluating error rates in forensic disciplines that incorporate an "inconclusive" conclusion.
• Proposed methods offer a more nuanced approach to error rate assessment beyond binary decision models.
• The findings contribute to the ongoing scientific discussion on improving the reliability and objectivity of forensic evidence analysis.