Can Reasoning Make Sentiment Models More Robust?

Sentiment classification looks like a simple task: take a financial statement and return a positive or negative label. It is used in research and production for financial-news monitoring, reporting workflows and downstream analysis.

Clean accuracy, however, tells us only how a model behaves on the wording it was given. If the same financial relationship is expressed with different words, a different sentence structure or an added concessive clause, the classification should remain stable. This is important because sentiment labels often become inputs to later processes; when a label is fragile, its errors can propagate into downstream analysis and decisions. That is the robustness question explored here.

Dataset

We began with 20 clean financial statements, each with an expected positive or negative label. These examples establish performance on a baseline financial domain before any changes are introduced.

Figure 1. The primary dataset of clean financial statements.

Each clean example was paired with perturbed versions designed to preserve the underlying financial direction and expected sentiment. The perturbed dataset adds coverage and surfaces invariance failures: cases where the label changes even though the meaning relevant to the task does not.

Figure 2. Controlled variants of the primary financial statements.

Testing four sentiment agents

We compared four fixed configurations:

• Amazon sentiment agent — Amazon Comprehend with a fixed wrapper that returns positive or negative; no natural-language rationale.

• Google sentiment agent — Google Cloud Natural Language sentiment analysis with the same fixed label wrapper; no natural-language rationale.

• Non-reasoning GPT — GPT-4o mini prompted to return one classification label.

• Reasoning-configured GPT — GPT-5.4 mini configured to return a brief rationale followed by the label.

This comparison examines four deployed agent configurations rather than claiming to isolate reasoning alone.

From clean accuracy to invariance

The main financial-language test uses four label-preserving perturbations. Lexical paraphrases replace words or phrases while preserving the financial direction. Sentence-level paraphrases restructure the sentence. Professional Persona rewrites change the register into polished executive language. Concessive clauses add a contrasting statement that should not overturn the main outcome.

These tests target related but distinct capabilities: lexical-semantic invariance, syntactic and paraphrastic invariance, semantic stability across register shifts, and discourse compositionality. Their corresponding failure patterns can be described as lexical sensitivity, structural sensitivity, register sensitivity and clause interference.

Perturbation	Failure label	Capability tested
Lexical paraphrase	Lexical sensitivity	Lexical-semantic invariance
Sentence-level paraphrase	Structural sensitivity	Syntactic and paraphrastic invariance
Professional Persona	Register sensitivity	Semantic invariance across register shifts
Concessive clause	Clause interference	Discourse compositionality
Keyboard typos	Orthographic sensitivity	Orthographic and noisy-input robustness

The design is inspired by FLUKE (Otmakhova et al., 2026) and a longer line of computational-linguistics work on controlled perturbations, including Semantically Equivalent Adversarial Rules (Ribeiro et al., 2018), CheckList (Ribeiro et al., 2020), contrast sets (Gardner et al., 2020), TextFooler (Jin et al., 2020) and BERT-Attack (Li et al., 2020). FLUKE is one of the latest broad frameworks in this tradition rather than the sole origin of the approach.

Clean accuracy was only the starting point

The four agents initially looked similar: clean accuracy ranged from 95% to 100%. Robust accuracy separated them. The reasoning-configured GPT scored 90%, the non-reasoning GPT 75%, the Google sentiment agent 75%, and the Amazon sentiment agent 45%. Here, robust accuracy is the weakest perturbation score across the 20 base cases.

The GPT robustness gap was concentrated in lexical paraphrases, at 90% versus 75%. That makes the matched examples important: they show what changed inside individual decisions rather than reducing the comparison to one headline number.

Figure 3. The main financial-language test across all four agent configurations.

What the failures looked like

Clause interference

The Amazon sentiment agent changed a positive cash-flow example to negative after a negative background clause was added. Cash flow still increased from EUR97m to EUR121m, but the phrase “Although the global economy was struggling” was enough to change the final prediction.

Figure 4. Amazon sentiment-agent failure after a concessive clause is added.

The Google sentiment agent showed a related failure in the opposite direction. The core result moved from profit to net loss, but appending “although revenues increased” changed the final prediction to positive.

Figure 5a. Google sentiment-agent failure on a profit-to-loss statement with a positive distractor clause.

On the exact same input, the reasoning-configured GPT identified the move from profit to net loss as the decisive evidence and treated the revenue increase as secondary. It returned the expected negative label.

Figure 5b. Reasoning-configured GPT recovery on the matched concessive-clause input.

Concessive relations require readers to integrate propositions that create and defeat an expectation, as described by Robaldo and Miltsakaki (2014). Here, the added revenue clause introduces genuine counter-evidence that should be weighed against—but should not overturn—the profit-to-loss transition.

Lexical sensitivity

The non-reasoning GPT failed when awkward substitutions preserved the EUR97m to EUR121m increase. The directional evidence remained available, but the expected positive label became negative.

Figure 6. Non-reasoning GPT failure on an awkward lexical paraphrase.

On the exact same input, the reasoning-configured GPT explicitly identified the increase and returned positive. The matched pair is consistent with explicit evidence integration improving stability in this case, although one recovery does not establish universal robustness.

Figure 7. Reasoning-configured GPT recovery on the matched lexical paraphrase.

Additional lexical recoveries

The same pattern appeared in three further examples from the main test. First, the reasoning-configured model recovered an after-tax income increase from EUR0.01m to EUR0.9m after the non-reasoning configuration failed on the awkward wording.

Figure 8. Recovery of the after-tax income increase.

In a second matched case, the rationale combined higher operating earnings with higher sales and returned the expected positive label.

Figure 9. Recovery using both operating-earnings and sales growth.

In a third case, it reconstructed an awkward continued-market-share-growth statement and treated the growth outlook as positive. Together, the examples show the model reconnecting unfamiliar wording to the financial relationship that determines the label.

Figure 10. Recovery of a continued market-share-growth statement.

A separate check on keyboard typos

Published financial language is heavily vetted, so typos were not part of the main threat model. We ran a separate typo-focused follow-up to examine noisy-input recovery. The examples below illustrate recovery mechanisms.

In the first pair, corruption changed “EBIT margin was up from 1.4% to 5.1%” into “EBIY iafgin was up from 1.4% to 5.1%.” The Amazon sentiment agent returned negative even though the numeric increase remained visible.

Figure 11. Amazon sentiment-agent failure on the corrupted EBIT-margin statement.

On the same input, the reasoning-configured GPT anchored its decision to the increase from 1.4% to 5.1% and returned positive.

Figure 12. Reasoning-configured GPT recovery on the matched EBIT-margin typo.

In the second pair, a corrupted sentence still described an operating loss replacing a profit. The Google sentiment agent returned positive despite the recoverable loss direction.

Figure 13. Google sentiment-agent failure on the corrupted profit-to-loss statement.

The reasoning-configured GPT reconstructed the spelling corruption, identified the significant loss and returned negative.

Figure 14. Reasoning-configured GPT recovery on the matched profit-to-loss typo.

In the final pair, the non-reasoning GPT changed a corrupted version of “net sales would be higher than the year before” to negative.

Figure 15. Non-reasoning GPT failure on the corrupted net-sales statement.

On the exact same input, the reasoning-configured GPT stated that the sentence still expressed an increase despite the typos and returned positive.

Figure 16. Reasoning-configured GPT recovery on the matched net-sales typo.

Limitations and uncertainty

The main test contains 20 base cases and one run per configuration, so the percentages should be interpreted descriptively rather than as precise population estimates. This is enough for a quick robustness check with useful coverage, but not for narrow uncertainty claims.

LLM systems can produce different outputs when the same prompt is repeated, including under settings intended to be deterministic (Atıl et al., 2025). The appropriate number of base cases (n) and repeated trials (k) depends on the coverage and uncertainty bounds required by the tester: n broadens sampling of the target distribution, while k measures run-to-run variability.

If evaluation uses an LLM judge, score variation can come from either the tested agent or the judge. Repeating judge evaluations over fixed agent outputs is one way to measure judge variability separately. Aggressive paraphrases and apparent failures also benefit from human review because semantic bounds are not oracles.

Improving performance and assessment confidence

One route to better robustness is to allocate more inference-time compute to cases that benefit from reasoning. Zaremba et al. (2025) found that, across several adversarial attacks and with important exceptions, greater inference-time compute improved the robustness of reasoning models. The practical trade-off is higher latency and cost, so the appropriate configuration depends on the severity of the failures it is intended to prevent.

Performance can also be improved without changing model weights. GEPA (Agrawal et al., 2025) uses execution trajectories and natural-language reflection to propose and test prompt updates, while Meta-Harness (Lee et al., 2026) searches over the wider code that stores, retrieves and presents context to the model. In a workflow like this one, clean and adversarial cases could form a development set for prompt or harness optimization, with separate held-out cases reserved for the final assessment.

Assessment confidence improves through broader evidence: more independent clean cases, more adversaries per case and, where run-to-run variability matters, repeated trials. The added cases should cover the financial language and failure modes that matter in deployment. Counts alone are not enough: a rare high-consequence error may deserve more attention than several low-impact mistakes.

Conclusion

Clean accuracy told only part of the story. Controlled, meaning-preserving variations exposed large differences in how consistently the four sentiment agents tracked the same financial evidence.

In this experiment, the reasoning model recovered more lexical changes and corrupted inputs, and its rationales made those recoveries inspectable. Reasoning alone cannot explain the whole gap: the two GPT configurations use different underlying models and the reasoning model is newer. The potential gain also carries a cost-and-speed trade-off: reasoning inference generally uses more test-time compute and tokens, increasing latency and expense relative to a one-label classification call.

The broader lesson is therefore not that reasoning guarantees robustness. It is that robustness must be tested directly, with variations that reflect the language and failure modes a deployed agent will encounter, to provide stronger, defensible evidence of the system’s robustness.

References

Otmakhova et al. (2026). FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation.

Ribeiro, Singh and Guestrin (2018). Semantically Equivalent Adversarial Rules for Debugging NLP Models.

Ribeiro et al. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.

Gardner et al. (2020). Evaluating Models’ Local Decision Boundaries via Contrast Sets.

Jin et al. (2020). Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment (TextFooler).

Li et al. (2020). BERT-ATTACK: Adversarial Attack Against BERT Using BERT.

Robaldo and Miltsakaki (2014). Corpus-driven Semantics of Concession: Where do Expectations Come from?

Atıl et al. (2025). Non-Determinism of “Deterministic” LLM System Settings in Hosted Environments.

Zaremba et al. (2025). Trading Inference-Time Compute for Adversarial Robustness.

Agrawal et al. (2025). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning.

Lee et al. (2026). Meta-Harness: End-to-End Optimization of Model Harnesses.