From pairwise comparisons and rating to a unified quality scale

Area of Science:

• Psychometrics and psychophysics
• Data fusion and analysis
• Subjective quality assessment

Background:

• Psychometric scaling quantifies perceptual experiences by linking stimuli, internal representations, and responses.
• Existing subjective datasets often use disparate experimental protocols like rating and pairwise comparisons.
• Integrating these diverse datasets is challenging but crucial for comprehensive analysis.

Purpose of the Study:

• To propose a probabilistic framework for fusing outcomes from different psychophysical experimental protocols.
• To enable the merging of existing subjective datasets and facilitate experiments collecting multiple measurement types.
• To analyze and compare the efficiency and accuracy of rating versus pairwise comparison protocols.

Main Methods:

• Developed a probabilistic framework to integrate data from rating and pairwise comparison experiments.
• Conducted simulations to evaluate the proposed fusion method.
• Utilized benchmark and real-world image quality assessment datasets for empirical validation.

Main Results:

• The proposed framework effectively fuses data from different psychophysical protocols.
• Analysis revealed the distinct advantages and trade-offs (time vs. accuracy) of rating and pairwise comparison methods.
• Fusion of protocols demonstrated improved overall accuracy and efficiency in subjective assessments.

Conclusions:

• The probabilistic fusion framework offers a robust method for combining diverse psychophysical data.
• The findings highlight the necessity of psychometric scaling and the benefits of protocol mixing for subjective tasks.
• This approach is broadly applicable to various quality-of-experience assessments beyond image quality.