The conundrum of diffuse basis sets: A blessing for accuracy yet a curse for sparsity

Diffuse atomic orbital basis sets have proven to be essential to obtain accurate interaction energies, especially in regard to non-covalent interactions. However, they also have a detrimental impact on the sparsity of the one-particle density matrix (1-PDM), to a degree stronger than the spatial extent of the basis functions alone could explain. This is despite the fact that the matrix elements of the 1-PDM of insulators (systems with significant highest occupied molecular orbital-lowest unoccupied molecular orbital gaps) are expected to decay exponentially with increasing real-space distance from the diagonal. The observed low sparsity of the 1-PDM appears to be independent of representation and even persists after projecting the 1-PDM onto a real-space grid, leading to the conclusion that this "curse of sparsity" is solely a basis set artifact, which, counterintuitively, becomes worse for larger basis sets, seemingly contradicting the notion of a well-defined basis set limit. We show that this is a consequence of the low locality of the contra-variant basis functions as quantified by the inverse overlap matrix S-1 being significantly less sparse than its co-variant dual. Introducing the model system of an infinite non-interacting chain of helium atoms, we are able to quantify the exponential decay rate to be proportional to the diffuseness as well as local incompleteness of the basis set, meaning small and diffuse basis sets are affected the most. Finally, we propose one solution to the conundrum in the form of the complementary auxiliary basis set singles correction in combination with compact, low l-quantum-number basis sets, showing promising results for non-covalent interactions.