Map Challenge assessment: Fair comparison of single particle cryoEM reconstructions

Cryo-electron microscopy (cryoEM) is capable of achieving near-atomic resolution of biomolecular structures due to recent advances in hardware. Despite the long history of image processing software development for cryoEM, uncertainty about best practices and validation remains. The Map Challenge was therefore designed to test the current state of single particle reconstruction. As the first such challenge, the participants were given the freedom to analyze the cases in whichever way they wanted. Therefore, the maps submitted feature different sizes, sampling and orientations, making assessment non-trivial. To be fair, I developed a method to pose all maps in each case in the same configuration with minimal interpolation. I assessed the quality of these maps by visual inspection and Fourier shell correlation (FSC). Comparing the even-odd FSC with an FSC calculated against a reference structure analysis, I concluded that the quality of the maps related more to the user than to other factors, such as the software package used. Poor quality maps suffer either from lack of data or poor choices made by the user. Some maps appear significantly better than a reference or consensus of other maps, indicating overfitting. Best practices to avoid problems include an understanding of the effects of reference map modifications on particle image alignment, and generating appropriate masks. Ultimately, none of the issues revealed in the Map Challenge is insurmountable, as underscored by the excellent quality of reconstructions achieved by a significant number of participants.