Digital Forensics Experimentation: Analysis and Recommendations

Digital forensics (DF) is becoming one of the most prestigious research areas in computer science due to its inherent nature of providing a means to acquire, examine, analyze, and report evidence to be used in legal processes. To successfully perform it, novel techniques, approaches, and tools have been proposed, experimented on, and evaluated by researchers. However, the experimentation process is not a trivial task in this area as substantial evidence is not accepted in court. Therefore, the experimentation process has to be improved in DF, especially its documentation and data sharing to enable its reproducibility. The objective of this paper is to characterize the state-of-the-art research on DF experiments. We conducted a Systematic Mapping Study (SMS), analyzing 107 primary studies reporting DF experiments. We demonstrate that DF experimentation somehow fails at documenting the most essential elements of an experiment, such as hypothesis, variables, design, instrumentation, validity evaluation, setup, training, datasets and benchmarks, statistical techniques (descriptive, hypothesis, and effect-size test), limitations, and data sharing. In this work, we also propose a set of recommendations to improve experimentation in DF, especially regarding its replication and reproducibility. DF experimentation should evolve if the community intends to provide reliable and reproducible studies. By embracing this, both academicians and practitioners might benefit from such experiments and evidence.