A method for semantic textual similarity on long texts.

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Information Retrieval

Background:

• Assessing semantic similarity in long documents is computationally challenging.
• Existing methods often require large datasets or struggle with text truncation.

Purpose of the Study:

• To introduce a novel method for semantic similarity detection in long documents.
• To leverage sentence transformers and large language models for efficient information retrieval.
• To develop an analytical fuzzy strategy for selective iterative retrieval.

Main Methods:

• Text preprocessing involves splitting documents into sentences.
• Utilizes pre-trained sentence transformers and large language models (LLMs) of any token capacity.
• Employs an analytical fuzzy strategy with parameter tuning for classifying similarity into four classes: identical, same topic, concept related, and non-related.
• Avoids text truncation by processing pairs of sentences.

Main Results:

• The method effectively detects semantic similarity in long documents without prior training.
• Smaller sentence transformers demonstrate reliability and cost-effectiveness for long text analysis.
• The fuzzy strategy enables selective retrieval even under noisy conditions.

Conclusions:

• The proposed method offers an economical alternative to large language models for long document similarity.
• Sentence transformers are viable tools for semantic similarity analysis of extensive texts.
• The approach facilitates accurate information extraction and document comparison.