AI detectors are poor western blot classifiers: a study of accuracy and predictive values

Area of Science:

• Biotechnology
• Academic Integrity
• Artificial Intelligence in Science

Background:

• Generative artificial intelligence (AI) can create realistic scientific images, challenging academic fraud detection.
• Western blots are a common biological technique frequently targeted for image manipulation.

Purpose of the Study:

• To evaluate the effectiveness of three free, web-based AI detectors in identifying AI-generated western blot images.
• To assess the impact of image size reduction on detector performance.

Main Methods:

• Tested three AI detectors (Is It AI?, Hive Moderation, Illuminarty) on 48 AI-generated and 48 authentic western blot images.
• Analyzed sensitivity, specificity, and positive predictive value (PPV) of each detector.
• Investigated the effect of reducing western blot image file size on detection accuracy.

Main Results:

• Detector performance varied significantly, with low positive predictive values for all.
• Is It AI? showed high sensitivity (0.9583) but moderate specificity (0.5417).
• Hive Moderation had high specificity (0.8750) but low sensitivity (0.1875).
• Illuminarty demonstrated moderate sensitivity (0.7083) and low specificity (0.4167).
• Reducing image size impacted sensitivity and specificity differently across detectors, with minimal PPV improvement.

Conclusions:

• Current free AI detectors are unreliable for authenticating scientific images like western blots.
• Generic detectors lack the necessary robustness and specificity for scientific content.
• There is a critical need for specialized AI detection tools trained on scientific imagery to combat academic fraud.