Assessing the quality of single particle reconstructions by atomic model building

The 2015/2016 Map Challenge challenged cryo-EM practitioners to process a series of publicly available cryo-EM datasets. As part of the challenge, metrics needed to be developed to assess and compare the quality of the different map submissions. The most common metric for assessing maps is determining the resolution by Fourier shell correlation (FSC), but there are well known instances where the resolution can be misleading. In this manuscript, we present a new approach for assessing the quality of a map by determining the map "modelability" rather than on resolution. We used the automated map tracing and modeling algorithms in Rosetta to generate populations of models, and then compared the populations between different map entries by the Rosetta score, RMSD to a reference model provided by the map challenge, and by pair-wise RMSDs between different models in the population. These metrics were used to determine statistically significant rankings for the map challengers for each dataset. The rankings revealed inconsistencies between the resolution by FSC, emphasized the importance of the interplay between number of particles contributing to a map and map quality, and revealed the importance of software familiarity on single particle reconstruction results. However, because multiple variables changed between map entries, it was challenging to derive best practices from the map challenge results.