Ordered quantile normalization: a semiparametric transformation built for the cross-validation era.

Area of Science:

• Machine Learning
• Data Science
• Statistical Modeling

Background:

• Normalization is crucial for machine learning data preprocessing.
• Classical normalization methods struggle with cross-validation and diverse data distributions.
• Existing techniques are not consistently effective across various underlying data distributions.

Purpose of the Study:

• Introduce Ordered Quantile (ORQ) normalization, a novel transformation method.
• Develop a one-to-one transformation to map arbitrary distributions to a normal (Gaussian) distribution.
• Evaluate ORQ normalization's effectiveness compared to existing methods.

Main Methods:

• Developed Ordered Quantile (ORQ) normalization, a transformation guaranteeing normal distribution in the absence of ties.
• Conducted a simulation study comparing ORQ with popular normalization techniques using known data generating distributions.
• Utilized repeated cross-validation to identify optimal transformations when the underlying distribution is unknown.
• Implemented and tested methods using the bestNormalize R package on car pricing data.

Main Results:

• ORQ normalization consistently and effectively transforms data to a normal distribution across all tested underlying distributions.
• ORQ was the only method demonstrating consistent effectiveness irrespective of the data's original distribution.
• The bestNormalize R package facilitates the evaluation of multiple normalization transformations.

Conclusions:

• Ordered Quantile (ORQ) normalization offers a robust and reliable solution for data preprocessing in machine learning.
• ORQ normalization is a valuable tool for transforming data to a normal distribution, enhancing model performance.
• The bestNormalize R package provides a practical framework for applying and comparing normalization techniques.