Related Experiment Video
Updated: Jul 19, 2025
Imaging Features of Systemic Sclerosis-Associated Interstitial Lung Disease
Published on: June 16, 2020
Learning more from the inter-rater reliability of interstitial fibrosis assessment beyond just a statistic
Peir-In Liang1, Wei-Chou Lin2, Mei-Chin Wen3
1Department of Pathology, Kaohsiung Medical University Hospital, Kaohsiung Medical University, Kaohsiung, Taiwan.
Abstract:
Interstitial fibrosis assessment by renal pathologists lacks good agreement, and we aimed to investigate its hidden properties and infer possible clinical impact. Fifty kidney biopsies were assessed by 9 renal pathologists and evaluated by intraclass correlation coefficients (ICCs) and kappa statistics. Probabilities of pathologists' assessments that would deviate far from true values were derived from quadratic regression and multilayer perceptron nonlinear regression. Likely causes of variation in interstitial fibrosis assessment were investigated. Possible misclassification rates were inferred on reported large cohorts. We found inter-rater reliabilities ranged from poor to good (ICCs 0.48 to 0.90), and pathologists' assessments had the worst agreements when the extent of interstitial fibrosis was moderate. 33.5% of pathologists' assessments were expected to deviate far from the true values. Variation in interstitial fibrosis assessment was found to be correlated with variation in interstitial inflammation assessment (r2 = 32.1%). Taking IgA nephropathy as an example, the Oxford T scores for interstitial fibrosis were expected to be misclassified in 21.9% of patients. This study demonstrated the complexity of the inter-rater reliability of interstitial fibrosis assessment, and our proposed approaches discovered previously unknown properties in pathologists' practice and inferred a possible clinical impact on patients.
Insights
Renal pathologists show poor agreement in assessing interstitial fibrosis, with significant deviations expected in over a third of cases. This variability impacts patient classification, especially in conditions like IgA nephropathy.
Area of Science:
- Nephrology
- Pathology
- Medical Statistics
Background:
- Interstitial fibrosis assessment by renal pathologists lacks consistent agreement.
- Variability in assessment may lead to misclassification and impact patient management.
Purpose of the Study:
- To investigate the properties of interstitial fibrosis assessment variability.
- To infer the potential clinical impact of this variability.
Main Methods:
- Assessed 50 kidney biopsies with 9 renal pathologists.
- Used intraclass correlation coefficients (ICCs) and kappa statistics.
- Applied regression models to estimate deviation probabilities and identify causes of variation.
Main Results:
- Inter-rater reliability ranged from poor to good (ICCs 0.48–0.90).
- Agreement was lowest for moderate interstitial fibrosis.
- 33.5% of assessments were expected to deviate significantly from true values.
- Assessment variation correlated with interstitial inflammation variation (r²=32.1%).
- IgA nephropathy interstitial fibrosis scores had an expected misclassification rate of 21.9%.
Conclusions:
- Interstitial fibrosis assessment exhibits significant inter-rater variability.
- Novel analytical approaches revealed previously unrecognized patterns in pathologist practice.
- This variability has potential clinical implications for patient diagnosis and treatment stratification.
