Decision models: when they replace a large LLM

⏱️ Reading time: 13 min

For one million labeled tickets, Amazon went from paying around $22,200 with Claude Sonnet 4.6 to paying close to $500 with its own decision model, according to the Strands Agents team. Same work, same ticket volume, 44 times lower cost, and latency that dropped from more than 4 seconds to less than 1. Two Amazon engineers, identified as Carr and Mehul, made that change to the company’s internal ticketing system and documented it with exact figures.

📑 En este artículo

1. TL;DR
2. What is a decision model?
3. Why it matters
4. How a decision model works
5. Real-world use cases
6. Common mistakes and best practices
7. Comparison with alternatives
8. Going deeper
9. Frequently Asked Questions
   1. Does a decision model always replace an LLM like Sonnet?
   2. What happens if a specialized classifier’s catalog of options changes frequently?
   3. How is a model like Strands Decider deployed in production?
   4. Does a classification model need to be retrained with every new ticket?
   5. What happens to a decision model’s latency if traffic spikes suddenly?
   6. Is an LLM like Sonnet still necessary in the same system?
10. References

This article explains what makes a decision model different from a general LLM like Sonnet, and in which cases it makes sense to replace one with the other.

TL;DR

• A decision model pays off with a fixed catalog and high volume: it costs up to 44 times less than a large LLM.
• Amazon switched from Sonnet 4.6 to Strands Decider and cut the cost per ticket from $0.0222 to close to $0.0005.
• Average time per call dropped from 4,215 ms with Sonnet to 897 ms with the specialized model: a 78% decrease.
• Replacing a large LLM only makes sense if the catalog of options is closed and stable, not if it changes every week.
• Sending JSON keys without a fixed order shifts the output probabilities by up to 0.19 points between otherwise identical runs.

What is a decision model?

A decision model is a machine learning model that, given a state and a question with a closed set of options, returns a probability for each option plus a confidence score. It doesn’t generate free text: it chooses among predefined alternatives, like a multiclass classifier optimized for speed and cost.

A general LLM like Sonnet can answer almost any question: explain a concept, write code, summarize a long document. To do that, it needs to generate text token by token and reason over a long prompt on every call. That flexibility costs compute time and money on every request.

A decision model gives up that flexibility in exchange for speed. Amazon first tried Strands Decider 2B and later moved to an E4B version based on Gemma 4, which performed better on its ticket test set. Strands Decider doesn’t generate a natural-language response; instead, it scores each option in the 20-category catalog Amazon uses.

Why it matters

Many companies use a large LLM for tasks that are fundamentally classification: labeling tickets, routing a support message, deciding whether a comment is spam. It’s the simplest option to implement, because the same model you already use to generate text can also return a label in JSON format. The problem shows up when volume grows.

In Amazon’s case, each ticket with Sonnet 4.6 consumed 6,554 input tokens and 170 output tokens, a total of 6,724 tokens per ticket, according to the Strands Agents report. With Strands Decider E4B, those numbers dropped to 2,009 input tokens and just 1 output token, 2,010 total, a 70% reduction in tokens processed per ticket.

That drop in tokens translates directly into cost. At Bedrock’s public prices for Sonnet ($3.00 per million input tokens and $15.00 per million output tokens) and SageMaker’s price for the Strands Decider endpoint ($1.408 per hour for an ml.g5.xlarge instance), the cost per ticket went from $0.0222 to close to $0.0005. Multiplied by one million tickets, that’s the difference between $22,200 and close to $500, a 98% reduction.

Latency dropped by the same proportion. Average time per call went from 4,215 ms with Sonnet to 897 ms with the specialized model, a 78% decrease. The day Amazon turned off traffic to Sonnet, the average duration of its Lambda function fell 74%.

This pattern isn’t exclusive to Amazon. Any team running a classification task over millions of events (tickets, transactions, messages) pays the same cost for using a generalist model on a task that, fundamentally, has an answer among a handful of fixed options. The LLM can almost always solve it. What you need to decide is whether it’s worth paying the price of a generalist model for a task that doesn’t need to generate text.

How a decision model works

The architecture Amazon built is almost entirely standard AWS infrastructure. Each ticket event arrives in an SQS queue, which triggers a Lambda function. The Lambda reads the ticket data and calls an Amazon SageMaker endpoint running Strands Decider. SageMaker copies the model weights from S3 before starting the container, and the endpoint’s network stays isolated, with no outbound access.

flowchart TD
    A["New ticket"] --> B["SQS queue"]
    B --> C["Lambda function"]
    C --> D["SageMaker endpoint: Strands Decider"]
    D --> E["Probability per label"]
    E --> F["Label applied to ticket"]
    C --> G["AppConfig: BEDROCK / DUAL / DECIDER"]
    G --> C

To avoid betting everything on an abrupt switch, the Lambda reads a three-state value from AWS AppConfig on every message. In BEDROCK mode only Sonnet responds. In DECIDER mode only the classification model responds. In DUAL mode, both models receive the same ticket, Amazon applies Sonnet’s label, and saves Strands Decider’s label to compare later.

sequenceDiagram
    participant T as Ticket
    participant L as Lambda
    participant S as Sonnet on Bedrock
    participant D as Strands Decider
    T->>L: event from SQS queue
    L->>S: requests label in DUAL mode
    L->>D: requests label in DUAL mode
    S-->>L: label applied plus tokens used
    D-->>L: probability for each option
    Note over L: compares both responses before deciding the switch

Running both models on real traffic, in parallel, for weeks, gave Amazon more information than any offline evaluation. And if something went wrong, reverting to BEDROCK was a 45-second configuration change, without touching code or redeploying.

For the classifier model to work, it needs a well-defined catalog of options. Strands Decider’s context window is 4,096 tokens, and the descriptions of the 20 labels Amazon used in production didn’t fit in there. The team rewrote each label as a one- or two-sentence criterion, sent them as the options of a single multiple-choice question, and added a “none” option for cases that don’t fit.

Here’s what the input/output contract a decision model follows looks like, in a simplified version written in plain Python with no AI library: a state, a set of options, and a probability for each one.

import math

def decision_model(scores):
    exps = {label: math.exp(score) for label, score in scores.items()}
    total = sum(exps.values())
    return {label: round(value / total, 3) for label, value in exps.items()}

resultado = decision_model({
    "facturacion": 2,
    "soporte_tecnico": 1,
    "ninguna": 0
})
print(resultado)

The literal output of that script is {'facturacion': 0.665, 'soporte_tecnico': 0.245, 'ninguna': 0.09}. A real decision model replaces those three fixed numbers with a trained network that computes each option’s score from the ticket, but the output contract, a probability per option that sums to 1, is the same.

⚠️ Heads up: in Amazon’s tests, sending the same ticket with the JSON keys in a different order shifted the output probabilities by up to 0.19 points. If you’re serving a specialized classifier behind an API, always send the keys in the same order before comparing results across runs.

Real-world use cases

Amazon’s pattern repeats in any system that receives high volume and has to choose among a few known categories. These are the scenarios where a classifier model tends to outperform a general LLM:

• Support ticket routing: deciding which team a case goes to (billing, bugs, accounts) before a person sees it.
• Content moderation: flagging a comment or image as spam, appropriate, or needing human review, with a fixed catalog of categories.
• Transaction fraud detection: scoring each transaction against a closed set of known risk patterns.
• Routing between agents in a multi-agent system: choosing which specialized agent or tool should handle a query, instead of having a large LLM reason through that decision every time.
• Monitoring alert triage: classifying an alert as critical, informational, or duplicate before waking up a human.

All these cases share three conditions: a catalog of options that doesn’t change every day, enough volume for the savings per ticket to multiply, and a context window where each case’s state fits entirely, or nearly entirely, within the chosen model’s limit.

Common mistakes and best practices

The most common mistake is assuming a decision model can replace any classification task without adjustments. If the labels need a long explanation to tell them apart, or the catalog changes every week, cramming all that context into a window of a few thousand tokens doesn’t work: you need to simplify the labels first, as Amazon did.

Another mistake is skipping the parallel comparison phase. Without a mode like DUAL, there’s no way to know if the new model fails exactly on the rare cases that matter most to the business. Running both models on real traffic, even if just for a few weeks, is the only evaluation that truly matters.

Tuning the confidence threshold is also part of the work, not a minor detail. Amazon had to calibrate how confident Strands Decider needed to be before applying a label without review. A threshold set too low applies wrong labels, and one set too high sends too many cases back to human review or to Sonnet, if it’s still running in parallel.

It’s also a mistake to confuse a decision model with a traditional rules engine. A rules engine applies exact, hand-written conditions, while a decision model learns from examples and gives a probability, which lets it handle ambiguous text that an if/else can’t interpret.

As for whether Amazon’s DUAL mode is still active right now: there’s no direct way to verify this from the outside, because the flag’s value lives inside Amazon’s private AWS account, within its own AppConfig. The real verification is done by someone on Carr and Mehul’s team checking that flag from the inside, not by a third party from the internet.

Comparison with alternatives

A classification model isn’t the only way to solve a classification problem. This table summarizes when each approach makes sense:

Going deeper

The math behind why Strands Decider is so cheap isn’t complicated: an ml.g5.xlarge instance costs $1.408 an hour and, at roughly 640 ms per request, handles close to 5,600 tickets in that hour. Dividing that out comes to about $0.0002514 per ticket at full utilization, around $250 per million tickets. Amazon reports close to $500 per million in real production, double that, because it leaves capacity headroom for traffic spikes instead of pushing each instance to 100%.

That headroom is a design decision, not waste: an instance always running at the limit has no way to absorb a spike without queuing up. The SageMaker endpoint autoscales based on traffic, so that extra margin buys time for new instances to come online before latency spikes.

The other advanced piece is why the jump from Strands Decider 2B to the E4B version, based on Gemma 4, improved results. More parameters in the classifier model give it more capacity to separate similar labels from each other, at the cost of more memory (the endpoint uses close to 19 GiB) and somewhat higher latency per request. Amazon measured both versions against the same test ticket set before deciding which one moved to production, the same principle behind the DUAL mode applied to choosing between two versions of the decision model itself.

💭 Key takeaway: the real savings don’t come only from the smaller model: they also come from the much shorter prompt. The descriptions of the 20 production labels alone already exceeded Strands Decider’s 4,096-token window, so simplifying them was just as important as switching models.

Your next step: take 20 real tickets from your support queue, build a closed catalog of 10 labels with one sentence each, and measure with a simple logistic regression how many it resolves correctly before thinking about replacing a large LLM.

Frequently Asked Questions

Does a decision model always replace an LLM like Sonnet?

No. It only makes sense when the task is choosing among a closed, known catalog of options, with high volume and meaningful latency requirements. If the labels need lengthy explanations, change frequently, or the task requires generating free text, a general LLM is still the better option.

What happens if a specialized classifier’s catalog of options changes frequently?

Every catalog change forces you to rewrite the label descriptions and, in many cases, recalibrate the confidence threshold. If that happens every week, the maintenance cost can outweigh the token savings, and it’s better to stick with a more flexible general LLM.

How is a model like Strands Decider deployed in production?

In Amazon’s case, it runs as an Amazon SageMaker endpoint on ml.g5.xlarge instances, with an SQS queue and a Lambda function in front, and AWS AppConfig controlling the switch between Sonnet and the new model.

Does a classification model need to be retrained with every new ticket?

Not in the normal inference flow. The model is trained, or fine-tuned, once per version, and after that it just classifies new tickets. It only gets retrained when the label catalog changes significantly or when a periodic evaluation shows its accuracy has dropped.

What happens to a decision model’s latency if traffic spikes suddenly?

It depends on the autoscaling of the service hosting it. In Amazon’s case, the SageMaker endpoint adds new instances as demand rises, but that process takes time, which is why the team leaves capacity headroom instead of operating at the limit.

Is an LLM like Sonnet still necessary in the same system?

Yes, for the cases the decision model can’t resolve: tickets that don’t fit any label in the catalog, or that need a text response. Many teams, including Amazon’s, keep the general LLM as a fallback for those cases.

References

• Strands Agents: Amazon’s case study replacing Sonnet 4.6 with Strands Decider, with the cost and latency figures cited in this article.
• Amazon SageMaker documentation: official reference on inference endpoints like the one hosting Strands Decider.
• Amazon Bedrock documentation: official reference for the service where Claude Sonnet runs in the AWS cloud.
• Wikipedia: Multiclass classification: general background on classification among multiple categories, the mathematical basis of a decision model.

📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Featured image: Foto de Raimond Klavins en Unsplash