A widely used corporate digital-transformation index is not fully measurement-invariant: a construct-validity audit of a text-based composite

Text-derived composite scores are widely used across the social sciences, but their comparability is usually assumed rather than tested.

We audit whether a widely used corporate digital-transformation index supports the comparisons drawn from it.

Using 46,799 firm-year observations from 5,267 Chinese A-share firms (2011-2024), we treat the six published sub-dimensions as indicators of a latent construct. With robust maximum likelihood and firm-clustered standard errors we fit the configural-metric-scalar ladder across ownership and industry, add a longitudinal two-wave model for time, revalidate the partial-invariance search in a held-out half of firms, and compare the index with an independent technology-keyword measure.

A one-factor model fits moderately (CFI = 0.959, RMSEA = 0.064) but represents three sub-dimensions weakly (omega = 0.70), and no admissible lower-dimensional alternative improves on it. Across ownership, metric invariance is retained (change in CFI = -0.009) and a partial-invariance model restores comparability (-0.007); ignoring it overstates the state-vs.-private gap by about a fifth. Industry shows the largest failure (metric -0.036), localized to one loading (-0.008); partial scalar invariance then holds against that model (-0.006) but not cumulatively (-0.014), so the comparison is conditional and fragile. Under it the raw composite and the partially invariant reflective latent model yield sector contrasts of opposite sign (+0.06 vs. -0.13 SD), a divergence produced by one sub-dimension and reproduced in a held-out sample and in a MIMIC model. Over time, the longitudinal model retains partial metric invariance (-0.003); scalar invariance requires freeing four of six intercepts, and the implied change in level then ranges from -0.07 to +2.41 SD across the fifteen two-anchor specifications, three of which satisfy the criterion. The two text-based measures correlate 0.78 as latent constructs and 0.53 as composites, a gap consistent with the composites' limited reliability.

Cross-sectional comparability can sometimes be recovered under explicit partial-invariance models; comparability of levels over time is not identified, because the specifications that fit disagree about the change by more than the change itself. Validity depends on the comparison, the grouping dimension, the preprocessing choice, and the aggregation rule; each should be tested rather than assumed.