Related Experiment Video
Updated: Jul 16, 2026
06:55
Inverse Probability of Treatment Weighting (Propensity Score) using the Military Health System Data Repository and National Death Index
Published on: January 8, 2020
Using lowess to remove systematic trends over time in predictor variables prior to logistic regression with quantile
Craig B Borkowf1, Paul S Albert, Christian C Abnet
1National Cancer Institute, Center for Cancer Research, Cancer Prevention Studies Branch, 6116 Executive Blvd., Suite 705, MSC 8314, Bethesda, MD 20892-8314, USA. CBorkowf@cdc.gov
Statistics in Medicine
|April 22, 2003
Summary
Systematic trends in laboratory data can bias case-control studies. Locally weighted robust regression (lowess) effectively removes these trends, improving logistic regression model accuracy for risk assessment.
Area of Science:
- Epidemiology
- Biostatistics
- Medical Research
Background:
- Case-control studies often use logistic regression to analyze binary outcomes against continuous predictors.
- Laboratory measurements of predictor variables can be affected by systematic trends over time, introducing bias.
- Accurate risk assessment relies on reliable predictor variable data in epidemiological studies.
Purpose of the Study:
- To introduce and evaluate the use of locally weighted robust regression (lowess) for correcting systematic trends in predictor variables within case-control studies.
- To assess the impact of trend-contaminated data versus trend-adjusted data on logistic regression model parameters and statistical power.
- To demonstrate the application of lowess for trend removal in a real-world case-control study of oesophageal cancer risk.
Main Methods:
- Employed logistic regression to model relationships between binary responses and categorized continuous predictors.
- Utilized locally weighted robust regression (lowess) to estimate and remove systematic temporal trends from laboratory measurements.
- Applied lowess-adjusted data to logistic regression models and compared results with analyses using raw, trend-contaminated data.
Main Results:
- Trend-contaminated data led to attenuated parameter estimates, reduced statistical significance, and lower power in logistic regression models.
- Lowess-adjusted data yielded nearly unbiased parameter estimates, restored nominal significance levels, and improved statistical power.
- The study successfully demonstrated lowess's efficacy in removing a day-of-analysis trend from sphinganine measurements in an oesophageal cancer study.
Conclusions:
- Locally weighted robust regression (lowess) is a valuable method for correcting systematic trends in predictor variables in case-control studies.
- Adjusting for laboratory measurement trends using lowess enhances the reliability and accuracy of logistic regression-based risk estimates.
- The application of lowess improves the statistical validity and power of epidemiological analyses affected by time-dependent laboratory drifts.
Related Concept Videos
01:52Regression Toward the Mean
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when researchers try to extrapolate results...
01:18Percentile
A percentile indicates the relative standing of a data value when data are sorted into numerical order from smallest to largest. It represents the percentages of data values that are less than or equal to the pth percentile. For example, 15% of data values are less than or equal to the 15th percentile. Low percentiles always correspond to lower data values. High percentiles always correspond to higher data values.Percentiles divide ordered data into hundredths. To score in the...
01:11Residuals and Least-Squares Property
The vertical distance between the actual value of y and the estimated value of y. In other words, it measures the vertical distance between the actual data point and the predicted point on the line
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
01:00Detection of Gross Error: The Q Test
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
01:02Quantifying and Rejecting Outliers: The Grubbs Test
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This number is...
01:09Truncation in Survival Analysis
Truncation in survival analysis refers to the exclusion of individuals or events from the dataset based on specific criteria related to the time of the event. This exclusion can happen in two primary forms: left truncation and right truncation.
Left truncation occurs when individuals who experienced the event of interest before a certain time are not included in the study. This is often due to a "delayed entry" into the study where only those who survive until a certain entry point are observed.
Left truncation occurs when individuals who experienced the event of interest before a certain time are not included in the study. This is often due to a "delayed entry" into the study where only those who survive until a certain entry point are observed.