Related Experiment Video
Updated: Jun 28, 2025
09:18
Author Spotlight: Assessing the Reliability of Doppler Ultrasound in Measuring Leg Blood Flow
Published on: December 15, 2023
2.8K
Split-sample reliability estimation in health care quality measurement: Once is not enough
Kenneth J Nieser1,2, Alex H S Harris1,2
1Center for Innovation to Implementation, VA Palo Alto Health Care System, Menlo Park, California, USA.
Health Services Research
|April 25, 2024
Summary
Split-sample reliability estimates are sensitive to random data splits, particularly with small sample sizes. Averaging estimates from multiple data resamples improves reliability and stability for measure developers.
Area of Science:
- Statistical Methods
- Psychometrics
- Data Analysis
Background:
- Split-sample reliability estimation is a common method.
- This method's sensitivity to random data splitting can affect results.
- Stability issues are pronounced in low sample size and low variability scenarios.
Purpose of the Study:
- To assess the impact of random data splits on split-sample reliability estimates.
- To propose methods for enhancing the stability of split-sample reliability.
- To provide guidance for measure developers and evaluators.
Main Methods:
- Simulated data reflecting diverse quality measure distributions and scenarios.
- Conducted simulation studies on split-sample reliability estimation.
- Utilized R functions for data simulation and analysis.
Main Results:
- Split-sample reliability estimates demonstrate high sensitivity to random data partitioning.
- This dependency is particularly notable in settings with limited sample size and low data variability.
- Averaging estimates across numerous data splits significantly enhances reliability stability.
Conclusions:
- Measure developers and evaluators should be aware of split-sample method's sensitivity.
- Averaging multiple reliability estimates from resampled data without replacement is recommended.
- This approach yields more stable and dependable reliability estimates.
Related Concept Videos
01:16Contaminants and Errors
89
Effective sample preparation is crucial for accurate and reliable laboratory analysis. During this process, two significant sources of error can arise: concentration bias from improper sample splitting and contamination caused by methods used to reduce particle size, such as grinding or homogenization. Identifying and minimizing these potential errors is crucial to ensuring the validity of the analysis.
Another key consideration is determining the appropriate number of samples required to...
Another key consideration is determining the appropriate number of samples required to...
89
01:11Statistical Analysis: Overview
6.6K
When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
6.6K
01:19Testing a Claim about Standard Deviation
2.4K
A complete procedure to test a claim about population standard deviation or population variance is explained here.
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
2.4K
01:15One-Way ANOVA: Equal Sample Sizes
3.3K
One-Way ANOVA can be performed on three or more samples with equal or unequal sample sizes. When one-way ANOVA is performed on two datasets with samples of equal sizes, it can be easily observed that the computed F statistic is highly sensitive to the sample mean.
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
3.3K
01:17Cochran's Q Test
311
Cochran's Q Test is a nonparametric statistical test used to determine if there are potential differences in the outcomes of three or more related groups on a binary (yes/no) or dichotomous outcome. It is essentially an extension of the McNemar Test, which is limited to two related samples - Cochran's Q test can handle three or more related samples, making it more versatile in scenarios where subjects are measured under multiple conditions. The test statistic follows a Chi-Square...
311
01:29Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K