Related Experiment Video
Updated: Aug 5, 2026
Improving IV Insulin Administration in a Community Hospital
Published on: June 11, 2012
INSIGHT: Ingest and Standardize Glucose Harmonization Tool
Ben Ehlert1,2, Krish Shah1, Dev Katarya1
1Department of Genetics, Stanford University School of Medicine, CA, USA.
Background:
Publicly available continuous glucose monitoring (CGM) datasets are fragmented across schemas, limiting reuse, pooling, and machine learning development. Although large biobanks provide harmonized resources, many smaller CGM studies remain difficult to access because ingestion and standardization require substantial manual effort.
Methods:
We developed INSIGHT (Ingest and Standardize Glucose Harmonization Tool), an open-source large language model (LLM)-assisted harmonization pipeline that standardizes public CGM datasets into a minimal common schema: Subject_ID, Timestamp, and Glucose. INSIGHT combines deterministic rules with structured LLM assistance to identify CGM files, classify file roles, generate schema-specific read plans, normalize timestamps and glucose units, resolve subject identifiers, detect overlapping exports, merge outputs, and retain auditable JSON artifacts. We evaluated INSIGHT on 14 public CGM datasets with curated reference outputs using pre-specified training and held-out testing partitions. Performance was assessed using file-level precision/recall, subject recovery, temporal alignment, glucose fidelity, scalar CGM feature accuracy, and composite INSIGHT scores.
Results:
INSIGHT is open-source at https://github.com/benehlert/CGMHarmonization. Across 90 model-dataset evaluations spanning 9 model variants and 10 datasets, top configurations achieved near-perfect held-out harmonization fidelity. GPT-5.4 and Gemini 3.1 Pro Preview achieved mean INSIGHT scores of 0.990, with glucose mean absolute error (MAE) below 0.03 mg/dL and within 5 mg/dL agreement of 0.9996. Errors primarily reflected complex subject-timestamp recovery rather than glucose conversion, with failures concentrated in deeply nested datasets containing overlapping or co-located non-CGM exports.
Conclusion:
INSIGHT demonstrates that LLM-assisted code generation can generalize CGM harmonization across heterogeneous datasets while producing auditable loaders. By reducing reuse costs, INSIGHT complements curated repositories and supports large-scale pooled analyses and foundation model development for glycemic physiology.
