Inconsistency was more prevalent than reported: an empirical study of 57 networks with multiple treatments using the...

Inconsistency has been reported to be ubiquitous in network meta-analysis. However, this evidence is based on statistical tests for inconsistency with well-documented power limitations. A novel interpretation index was developed that is founded on the Kullback-Leibler divergence measure and warrants a semi-objective decision about the extent of inconsistency as acceptably low or material when statistical tests are underpowered. The prevalence of local inconsistency was investigated using the synergy of the Bayesian node-splitting approach with the newly proposed interpretation index. The results were also contrasted with inferences drawn from the 'standard decision-making approach' about the presence of inconsistency.

The nmadb R package was considered to obtain the sample of 57 networks on a binary outcome. The Bayesian node-splitting approach was initially applied to each network to estimate the posterior distribution of direct and corresponding indirect effects for each split node alongside the inconsistency factor and between-study standard deviation ( $\tau$ ). Then, the interpretation index was applied to each split node to quantify the average divergence between the direct and indirect effects and determine whether inconsistency was acceptably low or material based on a semi-objectively derived threshold.

The interpretation index revealed material inconsistency in 58% of the split nodes and 81% of the networks compared to the 'standard decision-making approach' (whether the 95% credible interval of the inconsistency factor excluded zero inconsistency) that indicated conclusive inconsistency in 4% of the split nodes and 18% of the networks. Material inconsistency was less prevalent for large $\tau$ values and single-study split nodes. Networks with single-study split nodes yielded larger and more imprecise inconsistencies than split nodes with more studies, making inconsistency subject to small-study effects. Such networks were also prone to a spuriously acceptable low inconsistency. Splitting single-study nodes were associated with larger inconsistency with increasing $\tau$ values than splitting nodes with more studies.

Inconsistency should be interpreted cautiously in the presence of single-study comparisons and substantial statistical heterogeneity, as a true inconsistency may be concealed. Local inconsistency should be expected and quantified using a method that aligns with the evidence structure of the network.