Statistical inferences for polarity identification in natural language

Area of Science:

• Computational social science
• Natural language processing
• Statistical modeling

Background:

• Human behavior is fundamentally driven by information processing and decision-making.
• Understanding how individuals process information to make decisions is a key research objective.
• Existing methods for analyzing textual data often rely on heuristics, limiting interpretability.

Purpose of the Study:

• To introduce LASSO regularization as a novel statistical tool for extracting decisive words from text.
• To facilitate the study of granular expressions and their reception in natural language.
• To provide a statistically rigorous method for generating domain-specific dictionaries.

Main Methods:

• Application of LASSO (Least Absolute Shrinkage and Selection Operator) regularization.
• Utilizing LASSO not as a predictive model, but for statistical inference.
• Analyzing the relationship between word occurrences and an outcome variable.

Main Results:

• LASSO regularization effectively extracts statistically decisive words from textual content.
• The method yields highly interpretable statistical inferences, differing from standard predictive uses.
• Identified word choices and text segments are statistically significant to authors or readers.

Conclusions:

• The proposed LASSO regularization method offers a powerful tool for social science research.
• Enables the creation of objective, domain-specific dictionaries based on statistical evidence.
• Facilitates hypothesis testing in behavioral research by identifying key linguistic elements influencing authors and readers.