Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content.

Area of Science:

• Computer Vision
• Image Processing
• Human-Computer Interaction

Background:

• Developing robust no-reference (blind) visual quality metrics is crucial for consumer applications where cameras create original media and impairments simultaneously.
• The integrity of subjective data in such uncontrolled environments is not fully understood.
• Existing subjective study designs may not be suitable for consumer-generated content.

Purpose of the Study:

• To introduce two new datasets (CCRIQ2 and VIME1) of consumer camera images with subjective scores.
• To analyze the integrity of these datasets and propose a new analysis approach.
• To evaluate the suitability of different experimental designs for subjective quality assessment.

Main Methods:

• Collected subjective scores from 24 participants using the Absolute Category Rating method.
• Created CCRIQ2 with a strict experimental design for camera performance evaluation.
• Created VIME1 with a loose experimental design mimicking consumer photographer behavior.

Main Results:

• The loose experimental design (VIME1) produced unreliable subjective data, contradicting international recommendations.
• The strict experimental design (CCRIQ2) is more suitable for camera performance evaluation.
• The Hoßfeld-Schatz-Egger α metric failed to differentiate between the two datasets.

Conclusions:

• Classical subjective study designs may be inadequate for consumer content due to uncontrolled variables.
• Loose experimental designs, even if compliant with standards, can yield unreliable data.
• New approaches are needed for subjective quality assessment in consumer imaging contexts.