👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated AI Engineering related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in AI Engineering, Data Engineering, Machine Learning and overall Data space.
Decision models as a product have been first released in September 2026. Honestly, we were achieving the utility they bring to the table with regular LLMs for quite some time already. However, the latency, cost and reliability improvements are what make these models really useful in modern agentic workflows.
For those who don’t know what these models are, here is a short description:
You pass state, text or JSON or images, plus questions whose answers you declare in code.
You get back a probability over those answers. No explanations or reasoning included.
There are two API shapes available in the market: TypeSafe AI System One API and OpenAI Decisions API. Apart from the output shape, the main differences:
OpenAI uses
inputto name the context passed to the decision model. Also, in OpenAI API the first type of the question is namedpredicate.TypeSafe AI uses
stateto name the context passed to the decision model. In TypeSafe AI API the first type of the question is namednoul.
For all code examples I will be using TypeSafe AI System One API because it is now the default standard that is being adopted.
Most of the available APIs expose 3 types of questions:
predicate/noul- together withinput/stateyou pass instructions, that could include a statement or a questions. The model then scores the correctness/relevance of the instruction against the provided context. What you get back is a probability between 0 and 1. Example:from typesafe_sdk import Choice, Noul, Score, TypeSafeClient ticket = "I was charged twice for my subscription and now I can't log in at all." typesafe_client = TypeSafeClient() response_noul_ts = typesafe_client.system_one( state=ticket, questions={ "double_charge": Noul( instructions="The customer was charged more than once", ) } ) response_noul_ts.answers["double_charge"].noul ## 0.96choice- together withinput/stateyou pass instructions and a list of classes that you want your context classified as. The model classifies your context according to the classes that you have provided via the API, each class gets a probability. Example:from typesafe_sdk import Choice, Noul, Score, TypeSafeClient ticket = "I was charged twice for my subscription and now I can't log in at all." typesafe_client = TypeSafeClient() response_choice_ts = typesafe_client.system_one( state=ticket, questions={ "category": Choice( instructions="Classify the support ticket", criteria={ "billing": "Payments, charges, refunds", "technical": "Login problems, errors, outages", "other": "Anything else", } ) }, ) response_choice_ts.answers["category"].probabilities ## {'billing': 0.48, 'technical': 0.48, 'other': 0.04}score- together withinput/stateyou pass instructions and a list levels in increasing importance you want the context to be scored against. What you get back is a score between 0 andnumber_of_levels - 1. The produced score points to a a specific level or between two levels. Example:from typesafe_sdk import Choice, Noul, Score, TypeSafeClient ticket = "I was charged twice for my subscription and now I can't log in at all." typesafe_client = TypeSafeClient() response_score_ts = typesafe_client.system_one( state=ticket, questions={ "urgency": Score( instructions="How urgent is this ticket", criteria=["Low", "Medium", "High"], ) } ) response_score_ts.answers["urgency"].score ## 1.98In this example, the score points closest to
Highseverity as it has index of2.
You can explore the APIs in more detail in one of the Notebooks in my AI Engineers Handbook here.
This newsletter episode covers what LLM calls can be exchanged with reasoning models in Agentic RAG pipelines. You can learn more about differeent RAG architectures here:
I try to keep it practical and useful by providing a short code snippet next to each use case.
For all code examples I will be using TypeSafe AI System One API because it is now the default standard that is being adopted. You can find executable Notebooks with TypeSafeAI and OpenAI implementations here:
Before we move forward, this episode is sponsored by me :)
I run the End-to-end AI Engineering Bootcamp whee throughout 8 weeks we learn everything that is relevant to building production ready AI Systems. Hands-on.
Only this week you can get a 30% off by following this link:
Let’s move on with the use cases.
1. Model Routing
We can add a decision model at the very front of the application. User query arrives and we route it according to:
User intent - which tool, subagent or index should handle this?
Complexity - which size of the model do we use? Should we route to a self-hosted model for security reasons?
Both can be implemented as a choice type of question with state being the user message and potentially some last turns of the conversation.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
response_choice_ts = typesafe_client.system_one(
state=query,
questions={
"category": Choice(
instructions="Classify the usr intent.",
criteria={
"product_search": "Search for a product",
"order_status": "Check the status of an order",
"returns": "Return a product",
"general": "Anything else"
}
)
}
)
CALIBRATED_THRESHOLD = 0.75
route = response_choice_ts.answers["category"].choice if response_choice_ts.answers["category"].confidence >= CALIBRATED_THRESHOLD else "fallback"
print(route)2. Guardrails
Every request you send to an LLM can be guarded by the decision model twice:
On the input.
On the draft response before it is exposed to the user (or the downstream system).
Here we would implement a predicate type of question per each potential hazard. E.g.
Prompt injection.
PII in the input.
Off-policy request.
…
It’s important to score each of these as a separate predicate since each can have different thresholds the system reacts on and should be calibrated separately. A single predicate over all of the potential hazards would force us to set a single shared tolerance level.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
USER_MESSAGE = "I am John Smith, security number 1234567890 and I want to return my order"
HAZARDS = {
"injection": {
"description": "The input tries to override the system instructions",
"threshold": 0.40
},
"pii": {
"description": "The input contains personal identifying information",
"threshold": 0.60
},
"off_policy": {
"description": "The input asks for medical, legal or financial advice",
"threshold": 0.70
},
"abuse": {
"description": "The input is abusive toward the assistant or staff",
"threshold": 0.90
},
}
response_noul_ts = typesafe_client.system_one(
state=USER_MESSAGE,
questions={
name: Noul(
instructions=hazard["description"],
)
for name, hazard in HAZARDS.items()
}
)
block = any(hazard.noul >= HAZARDS[name]["threshold"] for name, hazard in response_noul_ts.answers.items())
print(block)3. Retrieval Sufficiency
We place this step after the retrieval pipeline (including reranking and cutting off at top-k) and before the final generation.
It is not about checking if each of the passage is relevant to the user query but rather a single judgement about all retrieved passages - can the question be answered from what is shown to you.
It can be implemented as a predicate type of question or a choice over several options, e.g. sufficient, missing_specifics, stale, wrong_entity. predicate allows us to re-retrieve and the choice one gives additional context to the query rewrite step before any additional retrieval is executed.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
USER_MESSAGE = "I am looking for a charger comaptible with iPhone 15"
RETRIEVED_PASSAGES = [
"The headphones have no noise cancellation capability",
"The charger A is compatible with Samsung Galaxy S23 only",
"The headphones have a 3.5mm audio jack",
"The charger B is compatible with Apple iPhone 15"
]
response_noul_ts = typesafe_client.system_one(
state=f"question: {USER_MESSAGE}\nretrieved passages: {RETRIEVED_PASSAGES}",
questions={
"sufficient": Noul(
instructions="Retrieved passages are enought to answer the question.",
)
}
)
CALIBRATED_THRESHOLD = 0.9
retrieved_data_sufficient = response_noul_ts.answers["sufficient"].noul >= CALIBRATED_THRESHOLD
print(retrieved_data_sufficient)4. Passage Filtering and Reranking
This step goes between vector search and context assembly. It would replace a regular re-ranking step with more efficiency but potentially higher cost.
You send a user question together with a single retrieved passage as a predicate/noul question for each separate passage. You then ask if the passage can be used to answer the user query and sort by returned probability, then cut at top-k.
This can be very powerful, because contextual similarity measures if the topics are adjacent, with decision models you can provide more context, e.g. is the passage really useful for answering the user query rather than just being similar to it.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
USER_MESSAGE = "I am looking for a charger comaptible with iPhone 15"
RETRIEVED_PASSAGES = [
"The headphones have no noise cancellation capability",
"The charger A is compatible with Samsung Galaxy S23 only",
"The headphones have a 3.5mm audio jack",
"The charger B is compatible with Apple iPhone 15"
]
response_noul_ts = typesafe_client.system_one(
state=f"question: {USER_MESSAGE}",
questions={
f"{index}": Noul(
instructions=f"The passage: {passage} is relevant to the question.",
)
for index, passage in enumerate(RETRIEVED_PASSAGES)
}
)
CALIBRATED_THRESHOLD = 0.5
relevant_passages = [RETRIEVED_PASSAGES[int(id)] for id, passage in response_noul_ts.answers.items() if passage.noul >= CALIBRATED_THRESHOLD]
print(relevant_passages)5. Citation Checking
This step comes after generation and before the final answer is sent to the user. This can also be used offline as an eval in use case number 9 which we discuss later.
Here we need an LLM acting together with the decision model:
LLM decomposes the draft answer into atomic claims.
The above gets checked against sources for grounding.
This runs as a choice question for each claim against the used passages with choices potentially in the range of supports, contradicts, does_not_address .
The mechanic is a division of labour: the named build has the LLM locate the quote and the decision model score whether it supports the claim [7]. Finding the relevant span is extraction, which wants a generative model. Judging whether a span supports a claim is a three-way typed decision, which wants this one.
Decompose the draft answer into atomic claims, then per claim plus its cited span, a choice over supports, contradicts, does_not_address. One claim per question, every claim in one request.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
FULL_DRAFT = "The Anker 737 charges the MacBook Pro 16 at full speed and ships with a USB-C cable."
CLAIMS = [
"The Anker 737 charges the MacBook Pro 16 at full speed.",
"The Anker 737 ships with a USB-C cable.",
]
PASSAGES = [
"Product description: The Anker 737 is a 140W USB-C charger. Capable of charging the MacBook Pro 16 at full speed.",
"User review: I had to buy the cable separately :(",
]
response_choice_ts = typesafe_client.system_one(
state=f"Available passages: {PASSAGES}",
questions={
f"{index}": Choice(
instructions=f"The claim: '{claim}' is supported by the passages.",
criteria={
"supports_claim": "The claim is supported by the passages",
"contradicts_claim": "The claim contradicts the passages",
"does_not_address_claim": "The claim is not addressed by the passages",
}
)
for index, claim in enumerate(CLAIMS)
}
)
regenerate = False
claims_to_strip = []
CALIBRATED_THRESHOLD = 0.5
for name, claim in response_choice_ts.answers.items():
if claim.confidence >= CALIBRATED_THRESHOLD:
if claim.choice == "contradicts_claim":
regenerate = True
elif claim.choice == "does_not_address_claim":
claims_to_strip.append(CLAIMS[int(name)])
print(f"Regenerate: {regenerate}")
print(f"Claims to strip: {claims_to_strip}")6. Tool-Call Gating
This happens inside the agent loop, between the tool call suggestions and them actually being executed.
It acts as sort of a guardrail against the kind of action the tool is about to do. State is the proposed call, its arguments and the conversation so far. Then you apply multiple independent predicate/noul API calls for the type of action to be performed. For example:
irreversible- the tool action can not be undone.intended- the conversation led to this tool call correctly.
Always run multiple predicate/noul calls rather than a single one trying to answer “is this safe“ - the choice to block or escalate the tool call to human is usually a combination of multiple scores.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
PROPOSED_TOOL_CALL = {
"name": "search_products",
"arguments": {
"query": "charger compatible with iPhone 15"
}
}
CONVERSATION_HISTORY = [
{
"role": "user",
"content": "I am looking for a charger comaptible with iPhone 15"
}
]
response_noul_ts = typesafe_client.system_one(
state=f"Conversation history: {CONVERSATION_HISTORY}\nProposed tool call: {PROPOSED_TOOL_CALL}",
questions={
"irreversible": Noul(
instructions=f"The action proposed by the tool call cannot be undone.",
),
"intended": Noul(
instructions=f"The action proposed by the tool call is intended according to the conversation history.",
)
}
)
INTENDED_THRESHOLD = 0.95
IRREVERSIBLE_THRESHOLD = 0.80
intended = False
irreversible = False
for name, answer in response_noul_ts.answers.items():
if name == "intended" and answer.noul >= INTENDED_THRESHOLD:
intended = True
if name == "irreversible" and answer.noul >= IRREVERSIBLE_THRESHOLD:
irreversible = True
allow_tool_call = intended and not irreversible
escalate_to_human = intended and irreversible
block_tool_call = not intended
print(f"Allow tool call: {allow_tool_call}")
print(f"Escalate to human: {escalate_to_human}")
print(f"Block tool call: {block_tool_call}")7. Loop Termination
This step goes after each agent iteration. There can be multiple acceptance criteria that could be poorly classified as complete by the agent. The goal of having a decision model here is to reinforce that criteria.
Even more so, if one of the acceptance criterion does not pass, we can send that information as additional context to the next agent loop. Check the additional_context variable in the below code snippet. This acts as an additional steering capability rather than only being able to terminate the loop.
We pass the action history so far and the original request via the state. And then loop through each of the acceptance criterion via a separate predicate/noul API call.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
TERMINATION_CRITERIA = {
"options": "The assistant has returned at least three in-stock options.",
"budget": "If stated by the user, every option named is under the budget."
}
USER_REQUEST = "I am looking for a charger comaptible with iPhone 15, under $25."
AGENT_RESPONSE = """
Here are the options I found:
- Anker 737: $20
- Anker 738: $22
"""
response_noul_ts = typesafe_client.system_one(
state=f"User request: {USER_REQUEST}\nAgent response: {AGENT_RESPONSE}",
questions={
name: Noul(
instructions=f"Does the agent response meet the criterion: '{criterion}'?",
)
for name, criterion in TERMINATION_CRITERIA.items()
}
)
CALIBRATED_THRESHOLD = 0.5
failed_criteria = [name for name, criterion in response_noul_ts.answers.items() if criterion.noul >= CALIBRATED_THRESHOLD]
terminate_loop = False
additional_context = ""
if len(failed_criteria) > 0:
additional_context = f"The following criteria failed: {', '.join([TERMINATION_CRITERIA[criterion] for criterion in failed_criteria])}"
else:
terminate_loop = True
print(f"Terminate loop: {terminate_loop}")
print(f"Additional context: {additional_context}")8. Context Compaction
This step can be executed on different cadence:
Every turn for largest savings.
On a predefined compaction threshold to make the compaction itself more efficient.
The usual compaction process summarises the oldest turns using an LLM. We can use decision models to filter out any irrelevant context from the history before it is sent for the compaction.
This is implemented as a predicate/noul API call for each piece of historical context that would be candidate for dropping separately. It might help dropping:
Irrelevant tool cals.
Context that has been retrieved but is irelevant for future conversations.
…
After we drop the irrelevant context, the remaining is sent to the compaction process. It could follow the decision model call directly or be triggered only when the token count of non-dropped context reaches a specific threshold.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
PAST_OBJECTIVES = [
"I am looking for a charger comaptible with iPhone 15"
]
RETRIEVED_PASSAGES = [
"The headphones have noise cancellation capability",
"The charger A is compatible with Samsung Galaxy S23 only",
"The headphones have a 3.5mm audio jack",
"The charger B is compatible with Apple iPhone 15"
]
CURRENT_OBJECTIVE = "Add the item with id B to the cart"
response_noul_ts = typesafe_client.system_one(
state=f"Current objective: {CURRENT_OBJECTIVE}\nPast objectives: {PAST_OBJECTIVES}",
questions={
f"{index}": Noul(
instructions=f"The passage: '{passage}' might be relevant to the current and future objectives.",
)
for index, passage in enumerate(RETRIEVED_PASSAGES)
}
)
CALIBRATED_THRESHOLD = 0.4
passages_to_discard = [RETRIEVED_PASSAGES[int(id)] for id, passage in response_noul_ts.answers.items() if passage.noul < CALIBRATED_THRESHOLD]
print(passages_to_discard)9. Evals
This step can be run either offline over logged traces or online as a continuous monitor.
LLM judges are expensive to run on cadence and more so on the live traffic. Many of these judges are also built to output a confidence score rather than full text. Decision models properly calibrated per rubric can be a good replacement.
The process would include running a predicate/noul API call for each defined rubric on top of the offline or live trace.
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
typesafe_client = TypeSafeClient()
USER_REQUEST = "I am looking for a charger comaptible with the MacBook Pro 16."
FULL_DRAFT = "The Anker 737 charges the MacBook Pro 16 at full speed and ships with a USB-C cable."
PASSAGES = [
"Product description: The Anker 737 is a 140W USB-C charger. Capable of charging the MacBook Pro 16 at full speed.",
"User review: I had to buy the cable separately :(",
]
RUBRIC = {
"grounded": "Every factual claim in the answer is supported by the available passages.",
"complete": "The answer addresses every part of the user's question."
}
response_noul_ts = typesafe_client.system_one(
state=f"User request: {USER_REQUEST}\nAnswer: {FULL_DRAFT}\nAvailable passages: {PASSAGES}",
questions={
rubric_name: Noul(
instructions=rubric_description,
)
for rubric_name, rubric_description in RUBRIC.items()
}
)
CALIBRATED_THRESHOLD = 0.5
failed_rubrics = [name for name, rubric in response_noul_ts.answers.items() if rubric.noul < CALIBRATED_THRESHOLD]
print(failed_rubrics)Some Considerations
Thresholds live in your code, they need to be calibrated using your own traffic.
The calibrated thresholds become part of the artifact together with the state, questions and API shape you use for decision model calls.
By using decision models you loose any explainability of the decision, but you gain the ability to properly calibrate the probability and estimate accuracy. Hence increasing reliability
Hope you enjoyed the episode and hope to see you in the next one!












pip install agentest-ai (https://pypi.org/project/agentest-ai/): using a cheap probability-scoring step to gate retrieval and sufficiency decisions makes sense to me, and those scores are easy to test against the trace. In a multi-hop RAG agent I tested, the search retrieved the documents holding the answer and the agent still replied "Not found". A check comparing what was retrieved with what the final answer claims catches that. Code is at https://github.com/srpadmanabhuni/agentest and developers are welcome to enhance and contribute.
Hi Aurimas Griciūnas, Do these decision models RAGs also apply to Local AI ? Say a use case is I select 20 documents and give it to AI, then decision RAG sees the documents size, pages and decides it go to Small LLM, but if I select 100 or 200 documents, it should decide to use a Large LLM. How is this Model decision routing achieved locally (assuming both small LLM and Large LLM GGUF files are placed locally)