A systematic approach to normalization in probabilistic models.

Area of Science:

• Information Retrieval
• Natural Language Processing
• Computer Science

Background:

• Term frequency (TF) quantification is fundamental to information retrieval (IR) models.
• Current TF normalization primarily relies on document length, potentially overlooking other document characteristics.
• Recent data suggests document verboseness, in addition to length, influences retrieval effectiveness.

Purpose of the Study:

• To propose and investigate a novel TF normalization combining document verboseness and length.
• To theoretically justify the proposed normalization by exploring the duality between verboseness and length.
• To evaluate the impact of this combined normalization on IR model performance across diverse domains.

Main Methods:

• Systematic combination of document verboseness and length in TF normalization.
• Theoretical analysis of the duality between document verboseness and length, and other IR model components.
• Empirical testing of new TF normalizations on four distinct test collections using a spectrum of TF quantifications.

Main Results:

• The proposed TF normalization, integrating document verboseness and length, was tested on multiple datasets.
• Experimental results showed that the new models consistently performed as well as or better than existing models.
• The interaction between document verboseness and length within the normalization was analyzed.

Conclusions:

• Combining document verboseness and length offers a theoretically sound and empirically effective approach to TF normalization.
• The novel normalization strategy enhances information retrieval performance without additional computational overhead.
• This research provides a more nuanced understanding of TF quantification in IR systems.