Domain-specific embedding models for hydrology and environmental sciences: enhancing semantic retrieval and question...

Area of Science:

• Environmental Science
• Hydrology
• Natural Language Processing

Background:

• General-purpose Large Language Models (LLMs) embeddings struggle with specialized scientific terminology and syntax.
• Effective information retrieval and question answering in scientific domains require domain-specific language understanding.

Purpose of the Study:

• Introduce HydroEmbed, open-source sentence embedding models fine-tuned for hydrology and environmental science.
• Adapt LLMs for four question-answering formats: multiple-choice (MCQ), true/false (TF), fill-in-the-blank (FITB), and open-ended questions.

Main Methods:

• Trained embedding models on the HydroLLM Benchmark, a dataset of textbook and scientific article content.
• Employed fine-tuning strategies like MultipleNegativesRankingLoss, CosineSimilarityLoss, and TripletLoss.
• Evaluated models using similarity-based context retrieval and GPT-4o-mini for answer generation on 400 QA pairs.

Main Results:

• HydroEmbed models matched or surpassed proprietary and open-source baselines.
• Significant performance gains observed in FITB and open-ended tasks due to domain alignment.
• Competitive accuracy achieved by the MCQ/TF model.

Conclusions:

• Task- and domain-specific embedding models are crucial for robust retrieval-augmented generation (RAG) and QA systems in science.
• HydroEmbed represents a foundational step towards a domain-specialized language model ecosystem (HydroLLM) for environmental sciences.