Advancing evaluation of bioassessment methods: A reply to Liu and Cao

A series of three papers was written about the development of multimetric indices (MMIs) using diatoms in rivers, streams and lakes for transcontinental surveys conducted by the United States Environmental Protection Agency. Stevenson et al. (2013) used the surface sediment diatom data from the 2007 National Lake Assessment to develop national scale site specific models for MMIs to account for natural variation in condition among sites. Liu and Stevenson (2017) also used the 2007 lakes data to evaluate performance of MMIs by grouping sites by ecoregions or typologies (naturally similar types of lakes defined by similarity in diatom species composition) with site specific metric models (SSMMs) that adjust metrics for natural variability among sites. Tang et al. (2016) used benthic diatom data from the 2008-2009 National River and Stream Assessment to develop SSMMs and MMIs by ecoregion and typology. All three studies showed that SSMMs improved performance of diatom MMIs by accounting for natural variation among sites. None of the studies provided consistent evidence that grouping sites by typologies produced better MMI performance than grouping sites by ecoregions. Liu and Cao (2018) criticized the Tang et al. (2016) paper for using means and standard errors to evaluate relative performance of MMI calculation methods at the site group scale, however, their criticism is incorrect. Actually, Tang et al. (2016) only used means to summarize and report relative performance of MMI calculation methods in the body of the paper. Tang et al. (2016) appropriately used non-parametric rank sum approaches to evaluate the probability that the multiple MMI calculations for separate site groups were the same for ecoregion (n = 9) and typology (n = 7) site groups. Liu and Stevenson (2017) used this same non-parametric approach for tests of lake diatom MMIs. Liu and Cao's (2018) concerns can be addressed by distinguishing between the goals and methods used for testing and evaluation of MMI calculation methods at the national and site-group scales. Tang et al. (2016) did not aggregate data across site groups to test MMI performance at the national scale because they were following standard EPA methods that develop separate MMIs for each site group. In conclusion, Liu and Cao (2018) misunderstood the MMI evaluation in Tang et al. (2016) and added no new information to this body of work, because all the concerns they raised were discussed in Liu and Stevenson (2017).