Addressing the Problem of Hard-to-Reach Unpublished Data from Theses in University Repositories

Researchers frequently encounter challenges in accessing valuable data encapsulated within university theses, which are predominantly archived in PDF format and remain unpublished in repositories. These documents often encompass original research, including vital environmental and hydrological data, yet they pose difficulties for searching or analysis due to inconsistent formatting and inefficient repository search tools such as keyword searches, which lead to an overwhelming list of documents. Our research team, engaged in developing a groundwater database for the Arequipa region of Peru, encountered this issue directly, with numerous relevant theses dispersed across local university repositories. The manual review process proved excessively time-consuming, necessitating the development of an innovative, automated solution. Our multi-step methodology commenced with optical character recognition (OCR) and Python scripts for keyword scoring, followed by the employment of Large Language Models (LLMs), notably Google's Gemini and the locally hosted Ollama, to semantically analyze content. This facilitated the identification and extraction of pertinent data (e.g., water quality parameters, well locations) and its organization into usable formats such as Excel spreadsheets; subsequent manual checks confirmed a high level of accuracy. The final system enables users to query an extensive number of documents swiftly and contextually, effectively overcoming traditional keyword search limitations. The tool is presently being disseminated among local researchers and institutions, offering a robust solution for accessing and managing regional groundwater data. This methodology possesses the potential for global scaling and adaptation, thereby enhancing access to gray literature and expediting scientific discovery across various disciplines.