Development Parameters of the Decision Aid Rule-Based Evaluation and Support Tool (REST) for Optimizing the Resources...

We aimed to develop and test a decision aid tool for NLP agnostic developers to assess the most relevant information extraction (IE) method. Because of non-inferior IE performance under specific conditions and of better carbon footprint, transferability and interpretability, we set up rules as the default IE method of the REST tool. We hypothesized that a hybrid IE method might optimize both rules and supervised ML method constraints. REST had to help the developer choose between rules (default option) and supervised ML methods (when necessary) for each entity. We defined the feasibility of rules according to IE performance metrics, entity linguistic homogeneity and entity occurrence frequency in the dataset. Entity by entity, if the sensitivity and positive predictive value thresholds are > 75%, if the entity linguistic homogeneity is > 10%, and if the entity occurs in > 25% of the highlighted records, REST suggests rules as the IE option in a user-friendly manner. Otherwise, supervised ML is suggested. A YouTube video tutorial https://youtu.be/v58IqJxVnCo summarize the specifications of REST. We assessed the feasibility and we tested REST on 12 entities of a minimal dataset in 35 free text medical records, by 2 oncologists. Its external validity resulted in an agreement rate of 91.7%. We developed and tested the visualization and decision aid REST which could help NLP agnostic developers to assess the feasibility of rules for a dedicated IE task. Supervised ML is considered as a backup option and LLM option is under investigation.