Water treatment research is expanding rapidly, but much of the knowledge acquired from this research remains scattered across unstructured literature.
The field still lacks a dedicated language model that can efficiently capture water treatment-specific domain semantics for large-scale literature mining.
WaterBERT Development
Here, we address this by developing WaterBERT, a domain-adapted encoder model designed for semantic representation and structured information extraction from water treatment texts.
WaterBERT was developed by continual pretraining on a large-scale water treatment corpus comprising about 2.97 billion tokens.
Model Evaluation
Three fine-tuned models based on WaterBERT were systematically evaluated on downstream tasks, achieving the best overall performance among general-purpose and domain-specific BERT models.
With F1 scores of 90.12% for multiclass treatment process classification, 79.50% for named entity recognition, and 74.04% for relation extraction.
Applications and Benefits
Beyond these benchmark tasks, we further demonstrated WaterBERT's advantages for large-scale literature processing.
Applied to 5,144 Environmental Science & Technology articles, WaterBERT-BERTopic identified coherent, diverse, and domain-specific research topics without predefined categories.
Building on WaterBERT, we processed 693,211 abstracts at substantially lower cost than commercial LLMs while retaining competitive extraction performance to construct a structured water treatment knowledge graph.
Water Knowledge-Enhanced Retrieval System (WaterKERS)
The knowledge graph was then integrated with lexical and dense retrieval to develop a Water Knowledge-Enhanced Retrieval System (WaterKERS), which achieved a relevance score of 77.7.
Substantially outperforming text-based retrieval baselines (54.7-64.5).
Contribution of WaterBERT
Through WaterBERT, this study provides a compact and scalable semantic foundation for large-scale information processing and evidence mapping in water treatment research.