Integrating query of relational and textual data in clinical databases: a case study

Area of Science:

• Biomedical Informatics
• Database Management
• Information Retrieval

Background:

• Clinical data presents challenges for traditional database systems due to malformed text.
• Existing relational database management systems (RDBMS) often lack adequate features for searching unstructured medical text.
• Information retrieval (IR) techniques are crucial for processing and searching large volumes of clinical documents.

Purpose of the Study:

• To design and implement a clinical data mart integrating information retrieval (IR) and relational database management systems (RDBMS).
• To evaluate the effectiveness of an integrated IR+RDBMS system in handling malformed clinical text.
• To explore solutions for improving text indexing and searching in clinical data environments.

Main Methods:

• Developed a clinical data mart using commodity software, integrating IR and RDBMS functionalities.
• Housed 2.8 million documents spanning five years, supporting features like Boolean, stemming, proximity, and fuzzy searching.
• Implemented relevance ranking based on "total documents per patient" or "report type weighting."

Main Results:

• Non-curated medical text contains significant spelling and punctuation errors, hindering text indexing and searching.
• Standard RDBMS packages' IR facilities are insufficient for adequately handling malformed text.
• The integrated IR+RDBMS system demonstrated capability in managing and searching the clinical data.

Conclusions:

• A robust integrated information retrieval and relational database management system can be developed by combining RDBMS with third-party IR software.
• There is a need for RDBMS vendors to enhance the accessibility of their IR features for non-programmers.
• The study highlights the potential of integrated systems for improving clinical data management and retrieval.