The Word, Not the Number: Counterfactual Sweeps Expose a Status Shortcut in Laya

Preprint (arXiv submission pending): the fine-tuned Laya encoder’s urgency decisions follow the word “overdue”, not the number of days. Counterfactual sweeps show what held-out accuracy can’t.

Preprint status: This paper has been submitted to arXiv (cs.AI), and the submission is currently on hold in arXiv moderation. I’m sharing it here so the work is available in the meantime. Once it’s live on arXiv, I’ll update this page with the link. It has not been peer reviewed yet.

Download the PDF (8 pages, 5 figures, 4 tables)

The short version

Laya is a ModernBERT-large encoder fine-tuned to make typed decisions over structured agent state. Fine-tuning on the typed-decisions benchmark lifts agreement with the teacher labels from 0.36 to 0.77. Accuracy looks great. I wanted to know whether the model actually reads the numbers its decisions are supposed to depend on.

So I swept the numbers counterfactually (147,136 evaluation records) and ablated the fields that co-vary with them. Here’s what I found:

• Urgency follows the word, not the number. The model flips urgency near days_until_due = 0, but that flip is carried by the word “overdue”. With the word held fixed, the step at zero vanishes (0.60 → −0.02). An invoice labelled overdue stays urgent even when it’s due in 75 days.
• Price reconciliation mostly checks equality. Discrepancies of 0.1% or less get called “Material” at 93% of evaluated points.
• The benchmark can’t tell the difference. Status and the sign of days_until_due agree on all 400 invoices, and the smallest nonzero price variance is about 3%. On-distribution accuracy can’t see these shortcuts.

The broader point: high held-out accuracy doesn’t tell you a model learned the rule you think it learned. Counterfactual sweeps plus dependency ablations are a cheap way to check, and the paper lays out the protocol as a template for auditing numerical decisions.

Abstract

Encoders fine-tuned to make typed decisions over structured agent state offer a non-generative alternative to prompting a large language model. Many of these decisions should depend on the size of a number: how many days until an invoice is due, or how far an invoiced price deviates from the purchase order. We use counterfactual sweeps and dependency ablations to examine numerical behaviour that aggregate held-out accuracy does not explain. We study Laya, a ModernBERT-large decision encoder, before and after fine-tuning on the typed-decisions benchmark, where fine-tuning raises agreement with teacher labels from 0.36 to 0.77. Across 147,136 counterfactual evaluation records, the fine-tuned model shows sharp, largely context-stable boundaries. Urgency switches near days_until_due = 0, but the switch is carried by the co-varying word “overdue”. When that word is held fixed, the sweep’s effect on expected urgency drops from 0.71 to 0.11 levels and the step at zero disappears. An invoice labelled overdue stays urgent even when it is due in 75 days. Price reconciliation is predominantly equality-sensitive: all 68 states change decision during the sweep, and nonzero discrepancies of at most 0.1% are called “Material” at 93% of evaluated points. Status and the sign of days_until_due agree on all 400 invoices, so on-distribution evaluation cannot distinguish the two cues, and the smallest nonzero price variance in the benchmark is about 3%, leaving smaller tolerances untested. Two single-seed retrains retain the near-zero urgency step, and a written-rule probe shows incomplete rule adherence. These findings support a status-dominated urgency shortcut and limited price-magnitude sensitivity, without establishing an exclusively string-based mechanism. We present the protocol as a template for auditing numerical decisions.

Cite

@misc{varghese2026wordnotnumber,
  title  = {The Word, Not the Number: Counterfactual Sweeps Expose a Status Shortcut in Laya, a Fine-Tuned Typed-Decision Encoder},
  author = {Varghese, Mathew},
  year   = {2026},
  note   = {Preprint. arXiv submission pending},
  url    = {https://mathewvarghese.space/wp-content/uploads/2026/10/varghese-2026-word-not-number-laya.pdf}
}

This is independent work. Nothing here has ties to my employer.