TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification

Rida, Adam

doi:10.48550/arXiv.2604.14531

Tracer Research / Open-source paper 02

TRACER.A classifier that learns when to ask the LLM.

Many AI systems pay a large model to choose from the same fixed set of labels on every request. TRACER learns from those past decisions. A fast local model handles the cases it can match reliably. Uncertain inputs still go to the LLM.

Adam RidaApril 16, 202614 minute readMIT licensed

Read the article Read the paper View GitHub

Abstract

Every LLM classification call creates a labeled example: the input and the model's answer. TRACER turns those traces into a growing training set for a lightweight surrogate. An acceptor estimates when the surrogate will agree with the teacher LLM, and a held-out parity gate decides whether the resulting policy is allowed to handle traffic. The paper measures how much work can move local at four agreement targets. Banking77 reaches 83.2% test coverage with .959 teacher agreement at the strictest target. CLINC150 reaches full coverage, though its test agreement falls below that target. MNLI stays at zero because the representation cannot support reliable routing. The refusal is part of the result.

Three observed regimes See the evidence

Banking77: 83.2%
CLINC150: 100%
MNLI: 0%

Figures above are from arXiv v1. The current open-source package has continued to evolve, and its later benchmarks should be read under their own setup.

In brief

A routing policy learned from the work your LLM already did.

01
Observe classification calls.
Each production request leaves an input and the fixed label returned by the teacher LLM.
02
Fit a local surrogate.
TRACER trains several small classifiers to reproduce those labels from text embeddings.
03
Defer uncertain cases.
An acceptor decides which predictions can stay local and which requests should return to the LLM.

The operating idea

Choose the teacher-agreement bar. TRACER searches for the policy that handles the largest share of traffic while clearing that bar on held-out data.

The paper reports empirical agreement on fixed benchmarks. Live distributions can shift, so production monitoring remains necessary.

01 / Classification

Classification is a fixed choice repeated at scale.

Intent detection, moderation, ticket routing, document tagging, and lead qualification all ask the same kind of question: which label from a known set fits this input?

A large language model can perform these jobs with little setup. It reads the input, follows a label schema in the prompt, and returns one answer. That convenience makes LLM classification common even when most requests follow familiar patterns.

As traffic accumulates, the system begins paying the same model to resolve many variations of decisions it has already made. The production log now contains supervision. Each pair of input and output says how the teacher handled one point in the workload.

Customer message

"I returned an item but don't see it on my account?"

Fixed labelrefund_not_showing_up

02 / Traces

Past answers become the training set.

TRACER collects these pairs and embeds each input into a numeric representation. It trains a pool of traditional classifiers, including logistic regression, linear SGD models, small multilayer perceptrons, and optional tree models. The candidate with the strongest teacher-label macro-F1 moves forward.

The target is teacher behavior. If the teacher assigns a wrong label or invents a label outside the requested schema, the trace records that answer. This distinction matters: a surrogate can agree closely with its teacher while both remain wrong against ground truth.

Trace 001

"I returned an item but don't see it on my account?"

refund_not_showing_up

Trace 002

"What is the procedure for activating this card?"

activate_my_card

Trace 003

"Are there ATM fees?"

atm_support

03 / Learning to defer

The local model predicts. The acceptor decides whether to trust it.

Learning to defer, or L2D, adds a routing decision to classification. Each input can be answered by the surrogate or handed back to the teacher LLM.

The surrogate produces a probability for every label. A separate logistic-regression acceptor reads four signals from that distribution: the highest probability, the second-highest probability, the margin between them, and normalized entropy. Together, those signals estimate whether the surrogate will agree with the teacher on this input.

The acceptor score is compared with a threshold called tau. A score above tau returns the local label. A score below tau invokes the teacher. Held-out data determines the threshold.

One input, two possible destinationsPaper method

01Input

A new classification request arrives.

02Embedding

The text becomes a fixed numeric representation.

03Surrogate

A small classifier proposes a label and probabilities.

04Acceptor

A learned score estimates teacher agreement.

Score clears tauReturn local label

The request avoids an LLM call.

Score below tauAsk the teacher

The LLM answers and creates a new trace.

The hybrid classifierWho answers x?

In words

Use the local surrogate when the acceptor score clears the threshold. Otherwise, send the input to the teacher LLM.

Maximize coverageCov = Pr[a(x) ≥ τ]

Subject to teacher agreementTA = Pr[f(x) = T(x) | a(x) ≥ τ] ≥ α

T(x)

The teacher LLM's label for input x.

f(x)

The lightweight surrogate's label.

a(x)

The estimated chance that surrogate and teacher agree.

τ

The routing threshold selected from held-out scores.

α

The operator's target teacher agreement.

Cov

The share of inputs handled by the surrogate.

04 / The parity gate

Raise the quality target and the system may defer more work.

The paper sweeps alpha across four targets. The policy with the highest coverage can serve only after it clears the target on a held-out shadow split and exceeds a 5% coverage floor.

Two policy families compete at every refit. A Global policy lets one surrogate handle all traffic when its overall agreement is high enough. An L2D policy keeps the acceptor and serves only the subset above tau. If every candidate fails, the teacher continues to receive every request.

The control below uses the four Banking77 test points reported in arXiv v1. Alpha is the calibration target. The agreement shown is the observed test result, which can land above or below that target.

Choose the paper's target alpha.

Higher targets ask the surrogate to match the teacher more closely on the handled slice.

α .80

α .85

α .90

α .95

Banking77 test coverage83.2%

Observed teacher agreement.959

Deferred to teacher16.8%

At target alpha .95, the L2D policy handled 83.2% of test inputs and deferred 16.8% to the teacher.

05 / Results

Three tasks reveal three valid routing outcomes.

Full local handling, selective deferral, and refusal all appear in the experiments. The last outcome establishes a useful boundary: some tasks need a richer representation before local routing is safe.

Full

CLINC150

100%

All test traffic was handled locally. At alpha .95, calibration agreement was .952, while test agreement was .930. The paper flags that gap.

Selective

Banking77

83.2%

At alpha .95, the L2D policy handled most test traffic locally and reached .959 teacher agreement on the handled slice.

Refused

MNLI

No candidate cleared the gate across nine configurations. Frozen sentence-pair embeddings could not separate entailment, neutrality, and contradiction reliably.

Paper results on held-out test sets
Task	Target α	Coverage	Teacher agreement	Ground-truth accuracy
Banking77	.80	100%	.894	.804
Banking77	.85	100%	.894	.804
Banking77	.90	96.1%	.912	.804
Banking77	.95	83.2%	.959	.816
CLINC150	.80 to .95	100%	.930	.924
MNLI	.85 to .95	0%	Not applicable	Not applicable

Dated cost illustration

Under the token prices and prompt lengths used in the paper, 10,000 Banking77 calls per day cost about $9,500 per year. The 83.2% coverage result reduces that illustrative teacher cost by about $7,900 per year.

06 / The flywheel

Deferred calls become new examples near the decision boundary.

Every request sent back to the teacher produces another labeled trace. Later refits gain evidence about cases the surrogate found difficult.

The paper divides training traces into five daily batches. Day 1 begins with 2,001 Banking77 traces. Each update merges the new batch and refits the policy from scratch. Coverage ends higher than it began, though it does not rise smoothly.

The dip on Day 3 and the smaller decline on Day 5 are important. A new batch can contain harder traffic, and recalibration can move the threshold. The gate protects the agreement target while the amount of traffic handled locally changes.

Day 12,001

73.4%TA .953

Day 24,001

78.2%TA .951

Day 36,002

74.5%TA .951

Day 48,002

83.4%TA .950

Day 510,003

82.6%TA .951

07 / Interpretability

The routing boundary should be inspectable.

TRACER produces artifacts about the partition between locally handled and deferred traffic. Operators can see which labels are stable, which inputs remain ambiguous, and how that boundary moves after a refit.

Contrastive boundary pairs give the quickest intuition. The two inputs below share the same teacher label in Banking77. The first is served by the surrogate at a high acceptor score. The second returns to the teacher.

Handled locally / score .95

"I returned an item but don't see it on my account?"

Teacher label: refund_not_showing_up

Deferred / score .00

"Why hasn't my return cleared my account?"

Teacher label: refund_not_showing_up

Slice summaries

Handled rate and teacher agreement for each class or length range.

Representative examples

Typical inputs from the handled and deferred groups within each label.

Boundary pairs

Similar teacher-labeled inputs that receive opposite routing decisions.

Temporal deltas

Per-label changes in handled rate after each refit.

Disagreements

Held-out cases where surrogate and teacher produce different labels.

Evidence boundary

The paper inspects these artifacts qualitatively. Their usefulness to practitioners has not yet been measured in a user study.

08 / Since the paper

The open-source package has kept moving.

The arXiv paper records the v1 experiment. Later repository releases strengthened the gate and added tools for checking a workload before training.

Exact lower-bound gating

The current package uses a Clopper-Pearson lower bound plus a selection and verification split.

Distance-based OOD deferral

Inputs far from the training distribution return to the teacher even when classifier confidence is high.

Trace scanning

tracer scan gives a conservative first look at whether the workload contains locally routable regions.

Install TRACER

pip install tracer-llm

tracer demo
tracer fit traces.jsonl --target 0.95
tracer serve .tracer --port 8000

View GitHub ↗Read the docs ↗

09 / Limits

Where the evidence stops.

The strongest reading of the paper is practical and specific: intent-classification traces can support large local offload under some representations, and the gate can reject a task that fails. Broader claims need broader experiments.

Teacher imitation

The surrogate copies teacher behavior, including errors and labels that fall outside the requested schema. It does not establish better ground-truth accuracy.

Calibration gap

CLINC150 passed alpha .95 on calibration at .952 and reached .930 on test. Empirical gating did not fully transfer to unseen data.

Task breadth

The positive results cover two intent datasets with separable embedding clusters. MNLI failed on compositional reasoning.

One representation

All paper experiments use BGE-large-en-v1.5 embeddings. Different encoders may change the coverage and quality frontier.

Simple baseline

The comparison uses a confidence-threshold logistic regression with full hindsight. Stronger learned routers and distillation methods remain future comparisons.

Operational evidence

Artifacts received qualitative inspection, refits start from scratch, and live distribution shift still requires monitoring.

Open paper and open code