pip install agentest-ai (https://pypi.org/project/agentest-ai/): using a cheap probability-scoring step to gate retrieval and sufficiency decisions makes sense to me, and those scores are easy to test against the trace. In a multi-hop RAG agent I tested, the search retrieved the documents holding the answer and the agent still replied "Not found". A check comparing what was retrieved with what the final answer claims catches that. Code is at https://github.com/srpadmanabhuni/agentest and developers are welcome to enhance and contribute.
What did you do with that decision then? I asume you sent additional context to the generation step that the information is indeed there? Did it manage to properly use the context then?
Hi Aurimas Griciūnas, Do these decision models RAGs also apply to Local AI ? Say a use case is I select 20 documents and give it to AI, then decision RAG sees the documents size, pages and decides it go to Small LLM, but if I select 100 or 200 documents, it should decide to use a Large LLM. How is this Model decision routing achieved locally (assuming both small LLM and Large LLM GGUF files are placed locally)
Decision Models as any other Language Models can be deployed locally next to your SLMS/LLMS. In that case, most of the providers adopt TypeSafe AI SDK and you can simply point it to a local URL. Would you find a tutorial on that useful?
tool-call gating is the one I'm thinking about most right now. like we had an agent last week that wanted to fire off a refund before anyone checked the order and it was just sitting there in the loop ready to go, and that moment is exactly why the little probability check before anything irreversible matters more than all the fancy routing stuff. guardrails on the draft response too, that's the other one, cheap and boring and it saves you constantly
Thanx Aurimas, there are 4 bots in the game powered by TypeSafe's Jev model—on the main page, they're Patch, Glitch, Vector, and Chaos. If a user doesn't want to connect their own agent via a prompt, they can just go with these bots instead. Though they think so fast that I'm considering slowing them down a bit to simulate actual thinking :)
Your tool-call gate asks whether the action can be undone before it runs. That's one of the two questions on my approval map. The other is whether the action leaves the company. Money going out does, and so does a message to a customer. I'd add it as a second score, and I'd send both to a named person.
pip install agentest-ai (https://pypi.org/project/agentest-ai/): using a cheap probability-scoring step to gate retrieval and sufficiency decisions makes sense to me, and those scores are easy to test against the trace. In a multi-hop RAG agent I tested, the search retrieved the documents holding the answer and the agent still replied "Not found". A check comparing what was retrieved with what the final answer claims catches that. Code is at https://github.com/srpadmanabhuni/agentest and developers are welcome to enhance and contribute.
Thanks for sharing Srinivas.
What did you do with that decision then? I asume you sent additional context to the generation step that the information is indeed there? Did it manage to properly use the context then?
Hi Aurimas Griciūnas, Do these decision models RAGs also apply to Local AI ? Say a use case is I select 20 documents and give it to AI, then decision RAG sees the documents size, pages and decides it go to Small LLM, but if I select 100 or 200 documents, it should decide to use a Large LLM. How is this Model decision routing achieved locally (assuming both small LLM and Large LLM GGUF files are placed locally)
Hi Rahul,
Decision Models as any other Language Models can be deployed locally next to your SLMS/LLMS. In that case, most of the providers adopt TypeSafe AI SDK and you can simply point it to a local URL. Would you find a tutorial on that useful?
yes please !
tool-call gating is the one I'm thinking about most right now. like we had an agent last week that wanted to fire off a refund before anyone checked the order and it was just sitting there in the loop ready to go, and that moment is exactly why the little probability check before anything irreversible matters more than all the fancy routing stuff. guardrails on the draft response too, that's the other one, cheap and boring and it saves you constantly
Hey Justin,
Are you applying them in production already?
Hey Aurimas, great article!
My bots in the game crashingagents.com run on TypeSafe model :)
Hi Eugen,
Snap, this looks like a nice project! Where are you using Decision Models mostly?
Thanx Aurimas, there are 4 bots in the game powered by TypeSafe's Jev model—on the main page, they're Patch, Glitch, Vector, and Chaos. If a user doesn't want to connect their own agent via a prompt, they can just go with these bots instead. Though they think so fast that I'm considering slowing them down a bit to simulate actual thinking :)
Your tool-call gate asks whether the action can be undone before it runs. That's one of the two questions on my approval map. The other is whether the action leaves the company. Money going out does, and so does a message to a customer. I'd add it as a second score, and I'd send both to a named person.