Stopwords in technical language processing

Area of Science:

• Engineering
• Computer Science
• Natural Language Processing

Background:

• Natural Language Processing (NLP) is increasingly used in engineering for tasks like information retrieval and text classification.
• Standard stopword lists, used to remove uninformative words, are often derived from general text and may not effectively handle engineering jargon.
• A lack of specialized stopword lists for technical domains hinders the performance of NLP applications in engineering.

Purpose of the Study:

• To address the gap in specialized stopwords for engineering text.
• To identify and curate a dataset of generic, insignificant, and uninformative stopwords within engineering contexts.
• To provide a resource for enhancing technical language processing applications.

Main Methods:

• Utilized statistical measures including term frequency (TF), inverse document frequency (IDF), and entropy.
• Analyzed engineering texts to identify frequent yet uninformative terms specific to the field.
• Synthesized findings from multiple statistical measures to rigorously identify stopwords.

Main Results:

• Successfully identified a set of generic stopwords pertinent to engineering texts, distinct from general stopwords.
• Developed a curated dataset of engineering-specific stopwords.
• Demonstrated a method for creating domain-specific stopword lists using statistical analysis.

Conclusions:

• The developed stopword dataset is ready for use in technical language processing applications.
• This resource can improve the accuracy and efficiency of NLP tasks in engineering.
• Future work could involve expanding this methodology to other specialized technical domains.