Improving a sentiment classifier with limited resources
Improving a sentiment classifier with limited resources involves more than simply deciding how much data to collect or which model to train.
A labelled collection may lack examples of an important class.
Once trained, a classifier may benefit from another model even when that model is weaker on its own.
Scope: document-level Entity-Level Sentiment Analysis (ELSA)
We study these decisions in document-level Entity-Level Sentiment Analysis (ELSA), primarily using South Slavic news.
Acquisition order and uncertainty-driven revisiting
We first compare acquisition orders as collections grow.
We then use uncertainty to choose which predictions to revisit, how many to process, and whether an alternative encoder or local LLM can correct them.
Selecting routing policies for held-out evaluation
Finally, we select complete routing policies on development data for held-out evaluation.
Findings: acquisition, correction, and policy trade-offs
Minority-prioritised acquisition helps at some very small training sizes, but not consistently thereafter.
On SentiCoref, an oracle correcting all errors in the least-confident 10% without harming correct predictions would add 9.14 Macro-F1 points; however the best evaluated action gains only 0.90.
Encoder averaging helps on AspectBench and NewsMTSC but harms SentiCoref, where selective LLM assistance is more promising.
Both development-selected policies route 30% of records to the encoder-assisted LLM, yielding observed gains of 1.02 points on SentiCoref (interval includes zero) and 1.71 on NewsMTSC, at substantially different inference costs.