Eleven quick tips for data cleaning and feature engineering

Davide Chicco1, Luca Oneto2,3, Erica Tavazzi4

  • 1Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada.

Plos Computational Biology
|December 15, 2022
PubMed

Related Concept Videos

Quantifying and Rejecting Outliers: The Grubbs Test01:02

Quantifying and Rejecting Outliers: The Grubbs Test

Sometimes, a data set can have a recorded numerical observation that greatly  deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier.  To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.8K
Extraction: Advanced Methods00:56

Extraction: Advanced Methods

Metal ions can be separated from one another by complexation with organic ligands–the chelating agent– to form uncharged chelates. Here, the chelating agent must contain hydrophobic groups and behave as a weak acid, losing a proton to bind with the metal. Since most organic ligands used in this process are insoluble or undergo oxidation in the aqueous phase, the chelating agent is initially added to the organic phase and extracted into the aqueous phase. The metal-ligand complex is...
506
Data Validation01:15

Data Validation

Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
207
Data Collection by Observations01:08

Data Collection by Observations

Data collection refers to a systematic way of obtaining, observing, measuring, and analyzing accurate information. Observational studies are one of the most widely used methods of data collection. It involves collecting data by observing the behavior and physical characteristics of a sample without making any modifications to the sample.
An astronomer viewing the motion and brightness of stars in the sky and recording the data is an example of observational data collection. A botanist recording...
12.4K
Cleaning, Sterilization, and Disinfection01:30

Cleaning, Sterilization, and Disinfection

Cleaning, disinfection, and sterilization are the methods that help to break the infection chain and prevent disease.
Cleaning
The cleaning process usually involves using water with detergents or enzymatic cleaner and removing foreign material from objects and surfaces, including organic material such as body fluids or inorganic material like soil. Cleaning is performed before high-level disinfection and sterilization because foreign materials on the cover of the devices interfere with process...
7.3K
Detection of Gross Error: The Q Test01:00

Detection of Gross Error: The Q Test

When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.3K