Many sentiment datasets use one annotator for each item, so the test labels show the conventions of that person.
We build a Nepali benchmark from two annotators who labelled the same sentences blind.
We keep only the 3,189 sentences where both give the same polarity.
Model Performance
On the three-class benchmark, NepBERTa with a fixed tokenizer gets a macro-F1 of 0.877 and IndicBERTv2 0.867.
Three commercial large language models (LLMs) get 0.882 to 0.896 zero-shot, and these differences are not significant.
Five-Point Scale Analysis
On the five-point scale, 766 sentences have two adjacent labels, for example Pos and SPos.
We accept both labels and train on both.
On this task, Claude Opus 5.5 beats the best encoders by 0.10 to 0.12 macro-F1.
Tokenizer Audit
A tokenizer audit adds a second result.
The default tokenizers of three Nepali BERT models delete the non-spacing Devanagari vowel signs and the virama.
When we restore the marks, macro-F1 increases by 0.084 for NepBERTa but decreases by 0.095 for a model pretrained on stripped text.