The Dark Data Quandary

The digital universe remains a black box. Despite attaining high-technology capabilities like artificial intelligence and cognitive computing, "Big Data" analytics have failed to keep pace with surging data production. At the same time, the falling costs of cloud storage and distributed systems have made mass data storage cheaper and more accessible. These effects have produced a chasm between data that is stored and data that can be readily analyzed and understood. Enticed by the promise of extracting future value from rising data stockpiles, organizations now retain massive quantities of data that they cannot presently know or effectively manage. This rising sea of "dark data" now represents the vast majority of the digital universe. Dark data presents a quandary for organizations and the judicial system. For organizations, the inability to know the contents of retained dark data produces invisible risk under a spreading patchwork of digital privacy and data governance laws, most notably in the medical and consumer protection areas. For courts increasingly confronted with Big Data-derived evidence, dark data may shield critical information from judicial view while embedding subjective influences within seemingly objective methods. To avoid obscuring organizational risk and producing erroneous outcomes in the courtroom, decision-makers must achieve a new awareness of dark data’s presence and its ability to undermine Big Data’s vaunted advantages.