Context engineering has gone from a niche concern to the core discipline of AI engineering in under a year. In mid-2025, two posts laid much of the groundwork. Manus shared lessons from rebuilding their agent framework four times (July 2025). Anthropic followed with their guide on effective context engineering for agents (September 2025). That was eight months ago. The patterns they described have since matured and were adopted across platforms.

I previously wrote about the foundations of context engineering in one of my previous articles:

This piece focuses on what has changed since then, the patterns that have matured, and the tradeoffs you need to understand as an AI engineer.

Why Context Engineering Matters More Than Model Capability

The core insight is simple. LLMs have a finite attention budget. Every token in the context window competes for that attention. As context grows, precision drops, reasoning weakens, and the model starts missing information it should catch. Research calls this “lost-in-the-middle” and “needle in the haystack“ problems.

Anthropic frames it well: context engineering means finding the smallest possible set of high-signal tokens that maximise the likelihood of desired outcomes. The discipline covers everything that lands in the context window: system instructions, tool definitions, MCP resources, retrieved documents, conversation history, and accumulated action history.

Several patterns have emerged for managing context effectively: progressive disclosure for controlling what loads and when, compression for shrinking accumulated history, routing for directing queries to the right source, evolved retrieval strategies for getting external knowledge on demand, and tool management for controlling the capability surface.

Each addresses a different dimension of the problem. The rest of this piece breaks them down.

I ran a live workshop on this topic where I walked through these patterns.

Watch The Recording

Pattern 1: Progressive Disclosure and Agent Skills

The problem

An agent that handles customer support, billing, refunds, and onboarding needs instructions for all four domains. Loading all instructions upfront wastes most of the context window on irrelevant guidance. The traditional alternative, spinning up separate specialised sub-agents, adds orchestration complexity, duplicates shared logic, and introduces latency from inter-agent communication. Neither scales well as the number of domains grows.

What the pattern does

Progressive disclosure loads information in tiers based on relevance. Discovery first (just names and descriptions), activation when relevant (full instructions), execution only during the task (scripts and reference materials).

Agent Skills are the standard implementation. I covered them in depth in my previous piece:

The short version: a skill is a markdown file with YAML frontmatter. The platform reads only the name and description at startup (~80 tokens median per skill). When the model determines a skill is relevant, the full instruction body loads (275 to 8,000 tokens). Supporting scripts and reference materials load only during execution. The format was released by Anthropic in December 2025 and adopted by OpenAI, Google, GitHub, and Cursor within weeks.

The most interesting application is agent identity management. Rather than routing queries to separate specialised sub-agents, a single agent assumes different identities on demand. At rest, it has a base identity. When a task activates a skill, the agent adopts that skill’s instructions, constraints, tone, and behavioral patterns. When the task completes, it returns to base. This is what Claude Code already does. It does not spin up a separate “PDF agent” and a “spreadsheet agent.” It is one agent that activates the relevant skill, shifting its identity to match.

Agent Skills are not only for coding agents. The pattern generalises to customer support, internal operations, research agents, and any system where agents need broad capability with focused execution. Because skills are plain English markdown, domain experts and team leads can configure agent behavior directly, without engineering expertise.

An interesting extension: agents that write their own skills. When an agent encounters a task it handles repeatedly, it can extract the pattern into a new skill file. Claude Code supports this through its skill-creator skill. The agent observes its own successful behavior, generalises it, and makes it available for future sessions. The quality of self-authored skills varies, but the direction closes the loop: humans write the initial skills, agents extend the library from experience.

Tradeoffs.

• Accuracy: High with a small skill set, but degrades with 100+ as overlapping descriptions cause misactivation.

• Latency: Low. Discovery data is pre-loaded, activation adds a file read, not an LLM call.

• Token cost: Low at rest (all 17 Anthropic skills cost ~1,700 tokens at discovery), but accumulates during a session. The key unsolved question: when does an activated skill get deactivated? Without explicit pruning logic, multiple activated skills destroy the token advantage over time.

• Maintainability: Easy per skill, harder at scale. 50+ skills with non-overlapping descriptions requires governance.

• Reliability: Moderate. Skill selection errors compound downstream, and the entire approach depends on selection accuracy at the discovery layer.

Subscribe now

Pattern 2: Context Compression

The problem

Every tool call, every observation, every reasoning step adds to the context. Each tool result can be hundreds or thousands of tokens: API responses, file contents, search results, error traces. Without intervention, the accumulated action history fills the context window and pushes out the system instructions, tool definitions, and early task context that the model actually needs to reason well.

A good example to showcase this is a simple ReAct agent as displayed in the image below:

1. Construct a prompt from system instructions and available tools.

2. User query added to the initial prompt and passed to a LLM triggers the LLM to output either the final answer to the users query or plan additional actions via tool use in the environment.

3. If tool use is chosen, the tools get executed.

4. The results piped back to the initial prompt by appending them to the reasoning and action history.

5. Then we repeat the loop for N times or until the users query can be answered.

Each of the turns adds to the context as conversation history expands. This is extremely troublesome when the tools are retrieving large amounts of context.

What the pattern does

Context compression shrinks accumulated history while preserving the information the model needs. There are few approaches of how the compressions could be handled:

• Keep only top N turns of interaction, discard the remaining.

• Sliding window compression: keep top N turns unchanged, compress the remaining using LLMs.

• Long-term Memory approach: keep top N turns unchanged, move the remaining history to a durable storage and retrieve only relevant actions on-demand.

The field has converged on sliding window plus summarisation hybrids as the dominant approach: keep recent turns in full detail, compress older context through LLM-based summarisation.

Manus adds two practical details. First, keep the most recent tool calls in raw format so the model maintains its "rhythm" and formatting style. Losing that rhythm leads to subtle degradation. Second, do not compress away error traces. When a tool call fails, leaving the error and stack trace in context helps the model avoid repeating the same mistake. This technique is well-established (libraries like Instructor use it for structured output retries), and it applies broadly to any agent that calls tools.

Tradeoffs.

• Accuracy: Moderate. Summarisation preserves the gist but loses details, and any compression is lossy.

• Latency: Moderate. Each compression step requires an LLM call. You can amortise this by compressing periodically instead of each new turn.

• Token cost: Low in many cases when long running agents are involved.

• Maintainability: Requires experimentation: what to keep raw, how many turns before compacting, what detail level in summaries.

• Reliability: Moderate. Works well for long-horizon tasks, but poorly when critical early details get summarised away.

Learn these patterns hands-on in my End-to-end AI Engineering Bootcamp (3 days left to register before next cohort starts). Apply code LASTCHANCE15 for 15% off.

Register Here

Pattern 3: Context Routing

The problem

A multi-domain agent has access to multiple knowledge bases, tool sets, and instruction sets. Loading all of them for every query wastes context and degrades accuracy. A billing question does not need the onboarding knowledge base. A technical support query does not need the refund policy.

What the pattern does

Context routing classifies the query and directs it to the right context source before anything enters the context window. Several approaches have emerged:

LLM-powered routing uses the model itself to classify the query and select the appropriate context source. More accurate than rule-based approaches, but adds latency and cost.

Hierarchical routing uses a lead agent to triage queries to specialised sub-agents, each with its own focused context window.

Rule-based routing uses keyword matching or pattern detection. Fast and predictable, but rigid and unreliable when queries don’t match expected patterns.

Hybrid routing combines multiple methods.

Tradeoffs.

• Accuracy: High for LLM-based routing, moderate for rule-based. LLM routing understands nuance, rule-based misses anything outside expected patterns.

• Latency: Varied. LLM routing adds an inference call before the main task. Rule-based is near-instant. Most production systems combine both.

• Token cost: Savings come downstream: by loading only relevant context, you reduce tokens for the main inference.

• Maintainability: Rule-based routing requires manual updates for new domains. LLM routing adapts automatically but is harder to debug when it misroutes.

• Reliability: LLM routing can hallucinate routing decisions. Fallback to a human or default agent is necessary.

Pattern 4: Retrieval Evolution

The problem

Agents need knowledge that is not in their training data: company documents, product catalogs, policy updates, real-time data. The naive approach (retrieve similar text, stuff it into the prompt, generate) fails on complex queries. A question like “what themes emerge across this quarter’s customer feedback?” requires connecting information across multiple documents, something vector similarity search cannot do. A question where the first retrieval returns insufficient results needs a second attempt with a reformulated query, something a fixed pipeline cannot do.

What the pattern does

RAG has matured from fixed pipelines to agent-controlled retrieval loops. Three evolutions stand out.

I wrote about evolution of RAG architectures in one of my previous articles:

Agentic RAG puts retrieval under agent control. Instead of a fixed pipeline (query → vector search → inject → generate), the agent decides its own search strategy, can reformulate queries when results are insufficient, and iterates until confident. The retrieval loop replaces the retrieval pipeline.

Graph RAG adds relational reasoning. Standard vector search finds similar text but cannot connect entities across documents. Graph-based approaches build entity-relationship graphs over the corpus, enabling thematic and relational questions that require connecting information across multiple sources.

Self-RAG trains models to decide when to retrieve and to critique their own outputs. The model assesses whether it has enough information before answering, triggers retrieval only when needed, and evaluates the quality of retrieved results before using them.

The most advanced work combines all three:

Tradeoffs.

• Accuracy: The strongest dimension. Agent-controlled retrieval can reformulate queries, try multiple strategies, and iterate until confident.

• Latency: High. A single question might trigger three to five retrieval cycles.

• Token cost: High. Each cycle adds retrieved chunks to context plus the agent’s reasoning about strategy. Cost scales with question complexity.

• Maintainability: Moderate. Debugging “why did the agent choose this retrieval strategy?” is harder than debugging a fixed pipeline.

• Reliability: Needs guardrails. Agentic RAG can over-retrieve on simple questions, so maximum retrieval rounds, confidence thresholds, and fallback to direct generation are essential.

Share

Pattern 5: Tool and Capability Management

The problem

Agents need tools to interact with the world: APIs, databases, file systems, search engines. Each tool requires a JSON schema definition that the model reads to understand what the tool does and how to call it. A single complex schema (nested objects, enums, parameter descriptions) can consume 500+ tokens. Connect a few MCP servers and you might reach 90+ tool definitions, over 50,000 tokens of schemas before the model starts reasoning. This is not a theoretical concern. OpenAI recommends fewer than 20 tools per agent, with accuracy degrading past 10.

What the pattern does

MCP (Model Context Protocol) has become the standard for connecting agents to external tools. Originally released by Anthropic in November 2024, it is now governed by the Agentic AI Foundation under the Linux Foundation.

MCP solves the connection problem. The context cost problem remains unsolved.

You can read more about the protocol in one of my articles here:

Manus found a practical constraint: avoid dynamically adding or removing tools mid-iteration, because tool definitions sit near the front of the context, and any change invalidates the KV-cache for all subsequent actions.

Beyond raw context cost, tool management introduces several open problems:

Description quality. The model selects tools based on descriptions, but most MCP server authors write descriptions for humans, not models. Too vague and the model picks the wrong tool. Too verbose and you waste context on a single schema.

Tool overlap across MCP servers. Two different servers might offer similar capabilities (two search tools, two file readers). Without deduplication or preference logic, the model picks arbitrarily.

No versioning for tool contracts. When an MCP server updates its tool schemas, the agent has no way to know. Stale descriptions in cache cause silent failures.

Security surface scales with tool count. Each connected MCP server is an attack surface. Tool outputs can contain prompt injection attempts, and the more tools available, the larger the exposure.

Tradeoffs.

• Accuracy: Depends entirely on description quality, which is unsolved.

• Latency: Low for discovery, but changing tools mid-iteration invalidates the KV-cache, adding significant latency.

• Token cost: The biggest hidden cost in many agent systems. A single complex JSON schema can consume 500+ tokens. 90 tools means 50K+ tokens before any user interaction.

• Maintainability: MCP standardises the interface, but not description quality, schema conventions, or versioning.

• Reliability: Moderate. Again, MCP standardises the interface but not the quality of the underlying tool. Each connected server expands the attack surface.

Putting It Together

These patterns are not alternatives. In a production agent system, you layer them: progressive disclosure and tool management define what can enter the context window, routing and compression manage what stays during execution, retrieval brings in external knowledge on demand, and evaluation measures whether any of it is working. Each layer addresses a different failure mode.

Where to Start

If your agents run long tasks, add compression first. Hybrid sliding window (keep the latest N turns raw, summarise older ones) is the most practical starting point, and probe-based evaluation will tell you whether your summaries are preserving what matters.

If your agents serve multiple domains, add routing. Even keyword-based rules cut context bloat before you invest in LLM-based classification.

If your agents connect to multiple MCP servers, audit the token cost. Count how many tokens your tool schemas consume before any user interaction. That number is usually higher than expected.

Hope you enjoyed this piece and hope to see you in the next one, cheers!

Subscribe now

Partner with SwirlAI

Share

AI Engineering Bootcamp

I have officially launched a YouTube channel. The first four videos are already up as a free pre-course for the End-to-End AI Engineering Bootcamp.

One thing I see consistently: people jump straight into building with LLMs without a proper development environment. They run everything locally with no containerisation, hardcode API keys, and tangle frontend and backend logic together. It works until it doesn’t. When they need to collaborate, deploy, or debug, everything falls apart.

A solid development setup is not optional for AI engineers. It’s the foundation everything else builds on. These four videos walk you through it from scratch:

Part 1: Setting Up Your Development Environment Configure your development environment and LLM API keys from scratch.

Part 2: Build and Containerise Your First Chatbot Build a Streamlit-based chatbot and deploy it with Docker Compose.

Part 3: Moving the Agent Behind a FastAPI Server Extract the LLM calling logic into a FastAPI REST server and run it as a separate service.

Part 4: Containerising Backend and Frontend Split the application into separate FastAPI and Streamlit containers, each deployed independently via Docker Compose.

By the end of the four videos, you have a clean separation between frontend, backend, and LLM logic, all running in containers. This is the foundation the full bootcamp builds on.

The videos are designed to be accessible to anyone with basic programming knowledge. You don’t need to be a software engineer. If you’re a technical PM, data analyst, or someone moving into AI from an adjacent role, this is a practical starting point to understand how AI applications are built and deployed.

More videos are coming in the upcoming weeks. Subscribe to the channel so you don't miss them.

All code is available on GitHub:

GitHub Repository

The full bootcamp syllabus and registration:

End-to-End AI Engineering Bootcamp

Apply code LASTCHANCE15 at the check-out for 15% off.

Free Workshop: State of Context Engineering in 2026

On Friday, March 20 I’m running a 45-minute workshop about context engineering.

Most agent failures come from poor context engineering, not weak model capability. Teams overload prompts with instructions, tools, and retrieved information. The result: brittle systems that are costly and difficult to scale.

In this session, I’ll cover:

• Core patterns shaping context engineering in 2026: how teams give models the right information at the right time

• Tradeoffs between context strategies, compared across accuracy, latency, token cost, maintainability, and reliability

• Practical application: using routing, progressive disclosure, and modular capabilities to build agents that are more efficient and reliable

If you’ve been following the recent newsletter on Agent Skills and progressive disclosure, this session connects directly to those ideas and goes broader.

Register for the webinar here:

State of Context Engineering in 2026

Subscribe now

Partner with SwirlAI

Share

AI Engineering Bootcamp

Everyone is talking about Agent Skills. On December 18, 2025, Anthropic released them as an open standard. Within weeks, OpenAI, Google, GitHub, and Cursor adopted it. Less than three months later, marketplaces like SkillsMP index over 400,000 skills across platforms.

If you look at a skill file, it’s very simple - a markdown file with some YAML at the top. That’s it.

So why did every major AI platform rush to support it?

The skill file is simple on purpose. It structures information into layers so that platforms like Claude Code, Codex CLI, and Gemini CLI can load context progressively: name and description first, full instructions when relevant, supporting materials only during execution. The file is the contract. The platform implements the interface. The design pattern driving both is progressive disclosure.

Let’s unpack how this works.

What Are Agent Skills?

Before we get to why they matter, let’s look at what they actually are.

An Agent Skill is a directory containing a SKILL.md file. That file has two parts: YAML frontmatter (metadata) and Markdown body (instructions).

Here’s a real example from Anthropic’s official skills repository. This is the pdf skill:

pdf/
├── SKILL.md
├── reference.md
├── forms.md
└── scripts/
    ├── check_bounding_boxes.py
    ├── check_fillable_fields.py
    ├── convert_pdf_to_images.py
    ├── create_validation_image.py
    ├── extract_form_field_info.py
    ├── extract_form_structure.py
    ├── fill_fillable_fields.py
    └── fill_pdf_form_with_annotations.py

And the contents of SKILL.md file:

---
name: pdf
description: Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.
license: Proprietary. LICENSE.txt has complete terms
---

# PDF Processing Guide

## Overview

This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see REFERENCE.md. If you need to fill out a PDF form, read FORMS.md and follow its instructions.

## Quick Start

...

Notice the directory structure. The SKILL.md file contains the skill’s metadata and instructions. The reference docs (reference.md, forms.md) provide supporting documentation. The scripts/ folder holds 8 Python utilities the agent can call during execution.

Skills live in predictable locations. On Claude Code, that’s ~/.claude/skills/ for personal skills or .claude/skills/ inside a project. Codex CLI uses .agents/skills/. Gemini CLI uses .gemini/skills/. The paths differ, but the format is identical across all of them.

The key distinction from system prompts or custom instructions: skills are modular and selectively loaded. A system prompt is always on. A skill sits dormant until the platform decides it’s relevant to the current task. The file organizes information into layers. The platform decides when to load each layer.

The Problem: Context Windows Are Not Free

Best practice recommends fewer than 20 tools available to an agent at once, with accuracy degrading past 10. The same principle applies to instructions. An agent handling customer support, billing, refunds, and onboarding doesn’t need all four workflow guides loaded when the user asks about a refund. Connect a few MCP servers without managing what loads, and you quickly reach 90+ tool definitions, over 50,000 tokens of JSON schemas before the model even starts reasoning. Layer in system prompts, workflow instructions, conversation history, and retrieved documents on top.

As context grows, the model's attention degrades and important information gets buried. Models reliably miss information placed in the middle of long contexts, a well-documented phenomenon called "lost-in-the-middle." The more irrelevant context surrounds the relevant pieces, the worse retrieval accuracy gets.

Join the March Cohort of my End-to-End AI Engineering Bootcamp to learn how to solve the challenges of Context Engineering for production systems in the real world. (Use code KICKOFF15 at the checkout for 15% discount).

AI Engineering Bootcamp

Progressive Disclosure and the Three-Tier Architecture

Progressive disclosure is a well-established design pattern, formalized by the Nielsen Norman Group for user interface design. The principle: show only what is needed for the immediate task and defer everything else. Advanced options live behind a click, rarely-used features hide in secondary menus. This reduces cognitive load, lowers error rates, and makes systems usable by a wider audience.

This is part of a broader trend: design patterns built for human cognition transfer well to agents. Agent memory systems already mirror human memory, separating short-term working memory from long-term storage. Progressive disclosure follows the same logic. The context window is the agent’s cognitive space. Overloading it degrades performance, while keeping it focused lets the agent reason sharply. Agent Skills apply this principle to how agents access knowledge.

I have written more about context engineering and memory management problems here:

The SKILL.md file organizes information into three layers. The platform implements the loading logic, deciding when to promote from one layer to the next.

Layer 1: Discovery

At startup, the platform reads only the skill's name and one-line description from the YAML frontmatter. I measured this across Anthropic's 17 official skills: the median discovery cost is ~80 tokens per skill, ranging from ~55 (webapp-testing) to ~235 (xlsx). All 17 skills together cost ~1,700 tokens, meaning an agent can be aware of dozens of skills for less context than a single activated skill.

Layer 2: Activation

When the platform determines a skill is relevant to the current task, it loads the full SKILL.md markdown body into context. I counted tokens across all 17 skills in Anthropic’s official repository: body size ranges from ~275 tokens (internal-comms) to ~8,000 tokens (skill-creator), with a median around 2,000.

The platform makes this decision using LLM reasoning over the descriptions from the discovery layer. Research shows that Claude selects skills through pure reasoning, with description quality directly determining routing accuracy.

Layer 3: Execution

During execution, the platform pulls in supporting materials on demand: scripts, reference documentation, templates, configuration files. These only enter context when the agent reaches a step that requires them.

Here’s how this works in practice, using Anthropic’s PDF skill as an example.

1. The SKILL.md body contains pointers to execution-layer files:

## Next Steps
- For advanced pypdfium2 usage, see REFERENCE.md
- For JavaScript libraries (pdf-lib), see REFERENCE.md
- If you need to fill out a PDF form, follow the instructions in FORMS.md
- For troubleshooting guides, see REFERENCE.md

When the agent hits a step that needs this context, it pulls in the referenced file. The content it finds falls into three categories:

2. Domain knowledge. REFERENCE.md contains technical context the agent needs to make informed decisions:

### Overview
pypdfium2 is a Python binding for PDFium (Chromium's PDF library).
It's excellent for fast PDF rendering, image generation,
and serves as a PyMuPDF replacement.

3. Executable scripts. The same file also includes code the agent can run directly:

import pypdfium2 as pdfium
from PIL import Image

pdf = pdfium.PdfDocument("document.pdf")
page = pdf[0]
bitmap = page.render(scale=2.0, rotation=0)

img = bitmap.to_pil()
img.save("page_1.png", "PNG")

4. Tool pointers. Some files reference standalone scripts as tools. From FORMS.md:

If you need to fill out a PDF form, first check to see if the PDF
has fillable form fields. Run this script from this file's directory:
`python scripts/check_fillable_fields <file.pdf>`, and depending on
the result go to either the "Fillable fields" or "Non-fillable fields"
and follow those instructions.

Each layer adds context, and none of it loads until it's needed. The unloading side matters too. A naive implementation would discard a skill's context entirely after use, only to reload it minutes later when the next related task arrives. Smarter implementations cache recently used skills or keep frequently activated skills warm, balancing context efficiency with the cost of repeated loading.

This Is Not Just About Coding Agents

Most Agent Skills adoption today is in developer tools like Claude Code, Codex CLI, Cursor, and Gemini CLI. The pattern generalizes well beyond that.

OpenClaw is the clearest example. It’s an open-source autonomous agent that passed 175K GitHub stars in under two weeks, and while it can code, its adoption is driven by non-coding use cases: managing calendars, drafting emails, controlling smart home devices, meal planning in Notion, coordinating across WhatsApp and Telegram. Its community registry, ClawHub, hosts over 13,000 skills, most of them non-technical.

The three-tier pattern works anywhere you need broad capability with focused execution. Customer support agents that know about 200 product features but only discuss 2 per conversation. Internal operations agents managing dozens of workflows. Research agents navigating large knowledge bases. Progressive disclosure is a system design pattern.

This is where the responsibility falls on us as AI Engineers. Coding agent platforms like Claude Code and Cursor already implement the progressive disclosure interface. When we build non-coding agents, customer support bots, internal operations tools, domain-specific assistants, we need to build that same interface ourselves. The SKILL.md contract gives us the structure. Implementing the loading logic, the discovery-to-activation-to-execution pipeline, that’s our job.

Subscribe now

Agent Behavior, Accessible to Non-Technical People

ChatGPT made conversing with AI accessible to everyone. Skills do the same for configuring how agents behave. A skill file is markdown with plain English instructions. A product manager can open one and understand what it does. A domain expert can edit one. A non-technical team lead can create one from scratch, or ask an AI to generate it using a skill-creator skill. Both Anthropic and Google ship built-in skill-creators that generate skill files from natural language descriptions.

On Claude.ai, non-developers can enable pre-built skills in settings, upload custom skill packages as ZIP files, and let Claude select relevant skills automatically. Configuring agent behavior used to require prompt engineering expertise or developer access. Now the people closest to the problem, domain experts, team leads, operators, can directly shape how agents behave.

Skill marketplaces like SkillsMP are already forming with a distribution model similar to browser extensions: discover, install, configure.

The Ecosystem Moved Fast for a Reason

Anthropic released the Agent Skills open standard on December 18, 2025.

Within weeks:

• OpenAI adopted it for Codex CLI and ChatGPT.

• Google added skills to Gemini CLI.

• GitHub Copilot launched skills support the same day as the standard.

• Cursor integrated skills alongside their existing Rules system.

This speed tells you something. Every one of these platforms faces the same two problems: how to give agents broad knowledge without destroying context quality, and how to let users configure agent behavior without requiring engineering expertise. The skills format solves both. Progressive disclosure keeps context lean, and the markdown-based contract makes skills accessible to anyone who can write plain English.

Wrapping Up

Agent Skills are the first mainstream implementation of progressive disclosure applied to agent context management. The pattern is simple: give agents a lightweight index of capabilities, pull in details when needed, and keep context lean.

Context efficiency is only half the value. Skills also make agent behavior configurable by anyone who can write plain English, the same way ChatGPT made conversing with AI accessible to everyone.

Coding agent platforms already implement the progressive disclosure interface. When we build non-coding agents, the responsibility falls on us as AI Engineers to implement the same three-tier loading pattern, with smart caching to avoid naive context churn.

The SKILL.md file defines the contract. The platforms implement the interface. Progressive disclosure as a system design principle is what ties them together. This pattern will outlast the specific file format and show up wherever agents need to choose from many options without drowning in information.

Subscribe now

Partner with SwirlAI

Share

AI Engineering Bootcamp

Few months ago I took my mission to teach AI to the next level - June 23rd marked the start of the first cohort of my End-to-End AI Engineering Bootcamp. Last week was the time to wrap up and reflect on the whole experience.

When I first launched the program, the goal was simple - bridge the gap between just knowing AI concepts and actually building, deploying, evaluating and engineering production ready AI systems end-to-end.

Now, after weeks of working alongside ambitious learners from around the world, I can confidently say that the experience was a lot more than I expected:

• First of all, the goal set was to the point - hands-on learning in a cohort based environment is how you succeed in learning AI Engineering.

• The participants were active beyond my expectations - each of the live sessions would become a club where live discussions would uncover unknown unknowns elevating the learning experience to new levels.

• Full dedication from my side - I’ve spent countless nights in order to deliver according to the quality bar I have raised for myself.

• Students were delivering week after week - capstone submissions were flowing in even if sometimes it required more time than initially thought.

My personal numbers:

• 40 hours of live lectures delivered.

• 30 hours of hands-on coding recorded and available for offline consumption.

• 250 pages written, available as offline reading material.

Success story of the cohort:

• One of the capstone projects was sold even before the demo day!

Join Next Cohort

Use code KICKOFF10 at the check-out for a 10% discount.

Personally, I know I gave my all to the bootcamp. However, because it was the first cohort shaping the future of the program, I never expected this outcome:

Here are some reviews that highlight student experience:

Equally important to me personally is the following acknowledgement. It reassures me that what I am doing is on the right path.

This also gives me confidence that the course is among the best of what you can get on Maven when it comes to hands-on End-to-End AI Engineering.

The feedback I received throughout the bootcamp will be invaluable for the evolution of the program and making sure that the next cohort will bring even more value to the students.

What’s new in the upcoming cohort.

Syllabus.

I have plenty of feedback and my own reflections that will help me bring the bootcamp to the next level. Some of the points that I will be focusing on even more in the upcoming cohort:

• Even deeper focus on Evals: you will get the value of dedicated eval courses and more.

• Context Engineering hands-on deep dives for different levels of Agentic System complexity.

• Additional project that I code up alongside your learning journey.

• …

New Partnerships.

I have already closed a partnership with Modal for the next cohort:

• Each student will receive $500 in Modal credits for cloud compute and deployment.

Potentially, additional partnerships will happen.

Guest Lectures.

The cohort will have a line-up of four or more guest lecturers that will bring unique insights into dedicated parts of the End-to-End AI Engineering process.

• This is an addition to all of the materials and sessions delivered by me.

• List of speakers TBD.

Who is this course for.

You will get most out of the bootcamp if you are:

• Data Scientist / ML Engineer looking to re-skill into AI Engineering.

• A founder looking to build AI Native products.

• Engineering Leader looking to up-skill your teams.

• Software Engineer looking to learn and implement non deterministic LLM based systems.

Reserve Your Spot

Use code KICKOFF10 at the check-out for a 10% discount.

I also run dedicated cohorts and workshops, so if your company is looking to upskill its talent or tackle business challenges with AI, feel free to contact me at aurimas@swirlai.com .

Price changes to the future cohorts

This will be the last time I offer the cohort at this price. Future cohorts will be priced higher due to several factors:

• The curriculum is already more advanced than what’s available elsewhere, and it will continue to expand and improve.

• Demand for the program is strong.

• Enrolling now provides lifetime access to materials from all future cohorts.

Closing Note

I am incredibly grateful to everyone who joined the first cohort and helped shape this program. Cohort Beta is going to be even stronger - I’d love for you to be part of it.

P.S. I’ll be running a free video lesson series on my youtube channel. This will be part of onboarding to the bootcamp. Stay tuned!

Share

Subscribe now

The topic that has been dominating headlines in AI Agent world for the past months - Context Engineering. In this article I will outline my thoughts on the topic and what I have observed while building Agentic Systems for the past few years. Also, we will look closer into all types of Context that Agentic Systems rely on and the challenges that come with managing it.

For Engineers who have been building AI Agents, Context Engineering is a new name but not a new practice - it is how we forced Agents to perform the work we expect to be done in the first place. When many were positioning Prompt Engineering as the hottest carrier path in the upcoming years I was not surprised, to me, the “old school” Prompt Engineering is a subset of Context Engineering.

What is Context Engineering and why it is important.

Let’s quickly remember what an Agentic System is.

In simple terms, it is a topology of LLM calls connected via different patterns. It could be an iteration loop or a more sequential pattern. But the main thing to take out from this is that each output of an LLM node influences the downstream system.

It is clear that quality of the agent is only as good as the context you pass to it via prompts fed to LLM at each step. One could think that ever expanding Context Size limits of the LLMs will be the solution, but there are many inherent problems in pushing as much data to prompts as possible. Few ways the Model can be confused due to too much information have been documented:

• Context Poisoning: When a hallucination makes it into the context

• Context Distraction: When the context overwhelms the training

• Context Confusion: When superfluous context influences the response

• Context Clash: When parts of the context disagreeContext window size.

This is where Context Engineering comes in. It is a practice of trying to provide the minimal amount of focused context to the specific Agentic System step so that it is able to perform work it was designed to.

There are multiple types of context that exists and needs to be managed and orchestrated in modern Agentic Systems. Bellow is an image listing these different types.

Let’s look into each part of the context separately and analyse what kind of challenges we need to solve when engineering/orchestrating it.

Join the September Cohort of my End-to-End AI Engineering Bootcamp to learn how to solve the challenges of Context Engineering for production systems in the real world. (Use code KICKOFF10 at the checkout for 10% discount).

AI Engineering Bootcamp

Instructions / System Prompt.

Where it fits: System instructions (or the system prompt) are rails that define LLMs outputs on a more general level. This is a special prompt that defines the role, behavior, and boundaries of the LLM outputs before any user input is passed to the LLM later. These instructions set the overall tone, personality, scope and response style for the model. They act as policy, ensuring the LLM follows certain rules regardless of what the user asks.

Challenges.

• Alignment - the system prompt must cover all the necessary guidance (from factuality checks to style guidelines) within a limited space. Too little can make the model may go off-track, too much might constrain useful behavior.

• Risk of prompt injection - a malicious user input can override or contradict the system instructions. Current LLMs don’t perfectly separate system instructions from user text. Attackers may embed conflicting instructions that cause the model to ignore the original rules.

• System context can conflict with user needs - if the user asks for something outside the allowed scope, the LLM must refuse per system instructions. Maintaining helpfulness with these constraints is a hard challenge.

User Prompt.

Where it fits: The user prompt is the direct request or query provided by the user, the immediate task the user wants done. This part of context was the main focus of prompt engineering - developers tweaked wording and details to try and get better answers from LLMs.

Challenges.

• Real use cases often involve multi-turn conversations/interactions, follow-up questions or the need for external information.

• A single user query might require multipple steps to be answered. E.g. searching for information, then summarizing, then formatting an answer. Such task can’t be solved by a single Agent turn, it requires multipple turns in sequence. Check out the Deep Research Agent as an example:

• Context engineering addresses this by breaking complex user requests into smaller sub-tasks or chaining prompts. The challenge here is understanding the user’s true intent and managing the agents flow to solve that intent.

• Ensuring that the user’s request is fully understood and fulfilled over multiple turns requires a lot f iteration and comprehensive eval suites covering multiple complex interaction examples.

In short, the user prompt is just the starting point. The context engineering needed to handle intent solution is what goes next.

Retrieved Context.

Where it fits: Retrieved context refers to additional information pulled in from outside sources to help answer the user’s query. This is the main concept on which Retrieval Augmented Generation (RAG) systems are based. When a question requires facts or data not already in the LLM’s parametric knowledge, the system would fetch relevant context from databases, documents, web or other APIs. Once we retrieve the data we inject it to the prompt that is given to the model. This external information becomes part of the model’s context for that turn.

Challenges.

• The primary challenge with retrieved context is finding and choosing the right information. In a large enterprise knowledge base or internet-scale data, there may be thousands of snippets that could be related.

• The system has a limited context window (a maximum number of tokens it can attend to, and also the previously mentioned confusion modes) so we must be selective about what we include there. We need to solve an optimisation problem - from N candidate pieces of data, pick the ones that matter most for this query. Retrieving too little will leave out key facts, retrieving too much will produce noise.

• Mitigating these issues requires hybrid retrieval and clever ranking of retrieved snippets, filtering out low-quality or conflicting data, and sometimes summarising or compressing the retrieved text.

• Properly preparing the corpus for retrieval is also a huge Data Engineering challenge as you need to carefully preprocess each of the chunk before embedding so that each of the chunks does not miss relevant surrounding context and more global metadata.

Subscribe now

State (short-term memory).

Where it fits: There is a blurred line between State and conversation history - the recent messages in the current session. The System State serves as the model’s working memory of the latest state available. Part of this state is also the conversation history. It does not mean though that you always pass the entire conversation history or other elements of the state each time you invoke an LLM as part of the system execution. Context Engineering as a lot about how you manage this evolving state and what minimal parts of the state are passed to LLMs in trying to make them perform exactly the tasks they are meant to perform.

Challenges.

• Obvious limitation is that the context window is finite. In long conversations or long-running AI agent tasks, the amount of conversation can exceed the size of the context window, forcing the system to drop or compress older messages. The challenge is deciding what to remember and what to forget. If you append the entire conversation every time, you will hit token limits. If you forget too aggressively, the model will lose important details and repeat past questions and tool calls.

• Context engineering at this layer involves techniques like summarisation, clipping and context caching. E.g. summarising earlier parts of the conversation and only keeping the summary or selectively carrying forward key facts

• Another issue is context drift - over many turns, if the model’s memory of early instructions or facts gets too diluted, it may start to deviate from the original intent. We need to continuously ensure the our Agents stays on track (sometimes by re-injecting key instructions into the context if they were dropped).

Long-Term Memory.

Where it fits: Long-term memory is a mechanism for Agents to retain information beyond the current session. This can include data like user’s preferences and profile, facts learned in past conversations, or any data that should persist over time. Implementations of long-term memory will vary - from simple databases of past Q&A pairs to vector embeddings of conversation transcripts that can be searched on demand. The key is that this memory is persistent and can be pulled into context when relevant.

Challenges.

• Relevance and retrieval - given months of accumulated interactions or facts, how do we fetch only the pieces that matter for the current conversation? Similar like with retrieved context, it is not viable to load everything - the system needs to search and inject the most relevant memories. You might have hundreds of things the Agent could recall about a user, but only a couple are useful in answering the current question.

• Consistency and updating - long-term memory can become a liability if not managed well - e.g. the user’s situation may change (they get a new job) and the memory store must be updated or old info deprecated. Stale or incorrect memories can lead to responses that could potentially breach trust.

• Privacy and security - storing personal data long-term requires safeguards so that sensitive info is not unintentionally exposed.

• Integrating memory into context without confusion - if our system pulls in a summary of your past chats, that summary becomes part of the prompt which could potentially conflict with other instructions or overwhelm the model if too verbose.

The connection between Long and Short-Term memory can be visualised as follows:

1. Episodic - This type of memory contains past interactions and actions performed by the agent. After an action is taken, the application controlling the agent would store the action in some kind of persistent storage so that it can be retrieved later if needed. A good example would be using a vector Database to store semantic meaning of the interactions.

2. Semantic - Any external information that is available to the agent and any knowledge the agent should have about itself. You can think of this as a context similar to one used in RAG applications. It can be internal knowledge only available to the agent or a grounding context to isolate part of the internet scale data for more accurate answers.

3. Procedural - This is systemic information like the structure of the System Prompt, available tools, guardrails etc. It will usually be stored in Git, Prompt and Tool Registries.

4. Occasionally, the agent application would pull information from long-term memory and store it locally if it is needed for the task at hand.

5. All of the information pulled together from the long-term or stored in local memory is called short-term or working memory. Compiling all of it into a prompt will produce the prompt to be passed to the LLM and it will provide further actions to be taken by the system.

We usually label 1. - 3. as Long-Term memory and 5. as Short-Term memory.

Subscribe now

Tools.

Where it fits: Most advanced AI systems would use external tools - for example, calling APIs, running code, searching the web or simply performing Retrieval via a predefined function. This extends the model’s capabilities by letting it act in the environment or fetch information dynamically. In terms of context, whenever we expose a tool to the LLM, we need to take care of two aspects:

1. Tool definitions - the instructions or documentation describing what tools exist and how to use them (often provided as part of the system prompt).

2. Tool outputs - the results returned by the tool, which the model should incorporate into its next response.

Vanilla Tool use implementation can be described as follows:

1. User Query is passed to the Agent (usually a Python application).

2. All of the available Functions/Tools are defined as part of the Agent code (procedural memory).

3. The list of available Tools is passed together with the User Query to a LLM via a prompt. The LLM figures out which functions need to be invoked and with what parameters.

4. The Agent application directly executes the functions.

5. User Query is sent to the LLM together with the data retrieved after function execution.

6. The answer is constructed and returned to the user via the Agent.

Challenges.

• The model must understand when and how to suggest a tool invocation. This often requires carefully prompting the model with instructions like:

  When making tool calls, use this exact format:
  {
      "name": "tool_name",
      "arguments": {
          "parameter1": "value1",
          "parameter2": "value2",
      }
  }
  
  CRITICAL: All parameters must go inside the "arguments" object, not at the top level of the tool call.
  
  Examples:
  - Remove item from shopping cart:
  {
      "name": "remove_from_shopping_cart",
      "arguments": {
          "product_id": "123",
          "user_id": "123",
          "cart_id": "456"
      }
  }

• Once a tool is used, the output needs to be integrated seamlessly. If the tool returns a lot of data, including that in the context can quickly hit context size limits. Often we need to post-process tool outputs, e.g. extracting parts that are relevant, summarising etc. before appending it to the context.

• State maintenance between tool calls and the model’s reasoning. Sometimes Agent architectures implement patterns similar to ReAct where the model iteratively reasons, calls a tool, analyses the result and continues. Ensuring that the model’s chain of reasoning is properly preserved across these steps is not trivial. Especially when you also need to keep the track of the original user query.

• Error handling - tools can fail or return unexpected results. In those cases we need to stitch safety nets around the non-deterministic part of the system (i.e. guide the model on how it should respond in the case of such errors, e.g. apologise and exit the system).

• From a security standpoint, tool use carries risks in context. E.g. if the System reads from a URL, the page might contain malicious text (prompt injection via tool output) or sensitive data that shouldn’t be revealed. Context engineering for tools also means guard-railing such cases and properly mitigating the risks.

Structured Output.

Where it fits: In most agentic applications, it is not enough for the LLM to respond with free-form text since the output of one LLM call can be an input to the next LLM call, hence we often need the answer in a particular format or structure. This could be a JSON object, an HTML snippet or a SQL query. Context for ensuring correct structured outputs refers to any instructions or mechanisms that enforce a certain format for the model’s response. In most cases you would define these rules as part of the System Prompt, on top of the instructions you should use frameworks like Instructor that help in making sure that structured outputs are properly handled.

Challenges.

• Anyone who has tried to get LLM to output perfectly structured data knows how frustrating it can be. Models often hallucinate formatting or include extra explanations despite instructions.

• A major challenge here is reliability - how to guarantee the output is formatted exactly as it is needed (no missing brackets, no additional text). While there are ways to minimise inconsistencies (like using OpenAI’s function calling and similar APIs that allow developers to specify a JSON schema or grammar that the model must adhere to, constraining the model’s decoding) not all systems support that.

• The good news is that tooling is improving - validators, format enforcement and better prompt techniques. A good example is the Instructor library that I personally use in most of my projects. It enforces the outputs to conform to specific Pydantic schema objects as well as applies advanced retry techniques if initial parsing of the outputs is not successful.

Wrapping up.

Context engineering is an actively evolving practice. We are all learning in the trenches - even though some practices might work today, some of them might become less relevant in the future as the tooling improves, some might stop working due to the changes in behaviours of LLMs. Whatever you think about it, it is the time to participate in defining the best practices and help moving the industry forward.

Happy building!

Hope you enjoyed this article and hope to see you next week!

Subscribe now

Partner with SwirlAI

Share

AI Engineering Bootcamp

👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in AI Engineering, Data Engineering, Machine Learning and overall Data space.

In today’s episode I want to share with you one of the frameworks that I have designed. It helps me think about adding complexity to the infrastructure needed to power Agentic Systems that we are building for enterprises. I hope it will help you as well.

In the post you will find:

• The Cowboy Agentic AI Hierarchy of Needs.

• The Enterprise Agentic AI Hierarchy of Needs.

• Different layers of infrastructure needed to power production grade Agentic Systems.

• Problems that these layers are meant to solve and some notable vendors that are tackling them.

The framework has evolved throughout the time. I will start with the initial version.

The Cowboy Agentic AI Hierarchy of Needs.

I started contemplating the Framework I am about to introduce to you close to when ChatGPT was launched a few years ago. It was inspired by the conversations I had with companies that were building agentic applications back then, on the first available LLM APIs.

The idea behind the Agentic AI Hierarchy of Needs framework is simple but useful:

• While working on Agentic Systems you want to build and ship them fast.

• What are the minimal requirements for Agentic AI specific tooling and infrastructure you can go away with as you progress in the lifecycle of your product:

  • POC → MVP → Beta → GA → …

Here is the image representing the importance of each infrastructure layer I observed back then (it is based on how companies would approach adding additional tooling as they developed the product).

Here is a short description of the diagram, we will go into some layers of the infrastructure in more details as we progress through the blog post.

Model Layer:

• CPU/GPU hardware: In most cases we are taking this for granted as it is being handled for us builders.

• Base Infrastructure: The next layer of infrastructure needed to handle LLM deployment and serving.

• Foundation Models themselves: In most cases provided by the big AI Labs.

We can start prototyping and build the simplest applications and reach POC level by only having this.

You can read more about the best practices of moving from POC to MVP etc. here:

Application Layer:

This is where the most business value is created - continuous development of applications powered by GenAI.

To reach a reliable MVP we usually needed:

• Data Storage.

• LLM Orchestration.

Beta:

• Model Routing.

• LLM Observability.

GA:

• LLM Evaluation.

• LLM Security.

• AI Agent Memory.

• AI Agent Communication Protocols.

I am now calling this the Cowboy Agentic AI Hierarchy of Needs. This is because back then close to none of the startups I worked with cared about the quality of the application compared to how much emphasis was put on the fact that the application had to work at scale and never go down.

Of course, everyone wanted VC money flowing in. The applications would crumble due to quality issues, however new customer inflow would still be higher than churn, so that was ok.

The Enterprise Agentic AI Hierarchy of Needs.

I still consult startups, but a lot of focus has shifted towards enterprise grade agentic applications in the last year plus. In general, the Cowboy Agentic AI Hierarchy of Needs did not stood the test of time and quality needed for the applications being shipped to production.

The shift was expected - LLM Observability and Evaluation had to take the central picture, pushing even Orchestration to the less important place in the Hierarchy of Needs.

Almost in all cases we are setting up the Observability foundations first before adopting any LLM Orchestration framework available in the market. Sometimes the adoption of these frameworks would not happen at all (depending on the complexity of the project/product) as simple wrapper clients like instructor are enough and the chaining can be easier implemented without a dedicated framework.

Also, I’ve noticed a trend of organisations dropping LLM Orchestration frameworks with the application they are building becoming more and more complex. This happens because the need for low level control becomes more important which is in most cases hidden away from the user by the frameworks and can become hard to reach.

Again, you can read more about Evaluation Driven Development shift here:

Subscribe now

The Vendor Landscape.

The good thing about AI hype is that so many platforms have been and are still being created to help with operations in each layer of the infrastructure stack. The picture below displays just a handful of more popular ones in the market. I believe it is less that 10% of the players that exist.

I am planning to start curating an up to date market map, stay tuned for more info and feel free to reach out if you want to be included and you believe I could miss you. You can reach me at aurimas@swirlai.com.

Now, let’s go deeper into some of the infrastructure layers defined and discuss what problems they are trying to solve.

The Model Layer.

The model layer can be split into 3 parts:

• GPU/CPU Hardware.

• Base Infrastructure.

• Foundation Models.

GPU/CPU Hardware.

The problem: Accelerated compute R&D, manufacturing and supply.

Some notable vendors/frameworks:

• R&D and manufactoring: NVIDIA, Groq, Google, AWS.

• Supply: NVIDIA, Groq, Google, AWS, Azure, Coreweave.

Extra notes: interestingly, with the increase in compute requirements for inference (both, regular serving and test time compute) even the big clouds are not capable to keep up with demand.

Base Infrastructure.

The problem: efficient and scalable model deployment on single and multi-node clusters.

Some notable vendors/frameworks:

• vLLM.

• Kubernetes.

• Slurm.

Extra notes: Vendors providing both Proprietary and Open Model APIs are leveraging this infrastructure for serving the models for the public.

Foundation Models.

The problem: the need for general, task specific and multi-modal models increasingly capable of solving difficult problems with high precision.

Some notable vendors/frameworks:

• OpenAI.

• Anthropic.

• Google.

• Mistral.

• Open Source community.

The Application Layer.

Share

Data Storage.

Internal data is considered to be the main differentiator for companies that build Agentic Systems nowadays.

The problem: Most production ready enterprise Agentic Systems rely on internal context available within the enterprise. There is a requirement for integration with variety of data sources and efficient capabilities for retrieval of this data.

Some notable vendors/frameworks:

• Qdrant.

• Weaviate.

• MongoDB.

Extra notes: Vector databases are just a piece of the puzzle. Efficient information retrieval systems should be comprised of relational, graph, key value, document and vector databases. It all depends on the use case and as an AI Engineer you should not always default to vector.

Observability and Evaluation.

Observability.

The problem: Agentic Systems are non-deterministic and can seem like black boxes from the outside. We need to be able to efficiently trace all of the actions that are happening within the application by instrumenting the code and then perform analytics on the traced data. Also, there is a need for LLMOps practice implementation like prompt versioning. It should also happen on the Observability layer.

Some notable vendors/frameworks:

• LangSmith.

• Langfuse.

• Arize.

Extra notes: Observability platforms often come with their own instrumentation SDKs. Some Evaluation as well as Versioning capabilities are part of these platforms too.

Evaluation.

The problem: Agentic Systems are non-deterministic, there is a need manage exact and non-exact evaluation rules that would be ran against the data produced by the system. It is the only way to make sure that the system you are building and evolving behaves as expected.

Some notable vendors/frameworks:

• Ragas.

• Arize.

• Galileo.

Extra notes: While the vendors do provide some out of the box evaluations, in most cases you will have to define your own evaluation rules. Also, you can’t blindly rely on evals defined in different platforms - even though the naming of the eval rules can match, the implementation is most likely different.

Subscribe now

Orchestration and Model Routing.

Orchestration.

The problem: Agentic Systems often take form of complex non-deterministic chains of LLM or other GenAI model calls. There is a need of frameworks that would help developers quickly build these systems and manage the complexity as they are being evolved.

Some notable vendors/frameworks:

• LangGraph.

• CrewAI.

• LlamaIndex.

Extra notes: Very often simple wrapper clients like instructor are enough to start off without the need to adopt any dedicated LLM Orchestration Framework. Also, these frameworks hide some low level implementation details that you would want to tweak to achieve the best performance of your application so it might make sense to drop the framework when your application becomes complex enough. Having said that, frameworks like LangGraph are moving in the right direction by allowing low level tweaks if needed.

Model Routing.

The problem: Not all model APIs that you will be using will be stable enough to handle your production traffic. There is a need for routing layer that would allow falling back to a different model provider if there are issues with the main one.

Some notable vendors/frameworks:

• LiteLLM.

• OrqAI.

• Portkey.

Extra notes: properly switching between models is harder that it might look. As an example, a prompt that works for OpenAI might not work well for Claude family of models. Together with the fallback models you should configure fallback prompts. You should also closely tie Routing with Observability.

LLM Security, Agent Memory and Communication Protocols.

Security.

The problem: Real Agentic Systems have agency over some of our internal systems (e.g. data retrieval, automated ticket creation etc.). Malicious actors can manipulate natural language based interfaces to extract sensitive data or perform unintended actions within your infrastructure. We need safety guardrails that can prevent this and help us identify existing vulnerabilities.

Some notable vendors/frameworks:

• splxAI.

• Lakera.

• WhyLabs.

Extra notes: LLM Security can be split into multiple categories like:

• AI Application Red Teaming - continuous attempt to jailbreak your application.

• Guardrails - making sure that no unexpected data reaches an LLM or is exposed to the user of your application. E.g. PII data.

Agent Memory.

The problem: Effective reasoning and planning capabilities of Agentic Systems strongly rely on the actions that the system has already taken as well as on the context available to the organisations internally and externally. We are used to modelling this memory by splitting it into short-term and long-term. There is a need of a layer that helps efficiently manage and retrieve relevant memories on-demand. You can read more about the types of memories in Agentic Systems here:

Some notable vendors/frameworks:

• mem0.

• Cognee.

• Letta.

Extra notes: These memory layers are not just databases, but rather frameworks of for efficient memory management and retrieval.

Communication Protocols.

The problem: As we are entering the era of IoA (Internet of Agents) where AI Agents are distributed over the network and developed by different organisations, we need standards of how the communication between these systems should be handled.

You can read more about how MCP and A2A protocols work here:

Some notable vendors/frameworks:

• MCP.

• A2A.

• Agentcy.

Extra notes: Open protocols for Agent communication are important but they are just a piece of the picture, there will be a need for standards that govern all of the existing protocols and other missing pieces. E.g.

• How do we standardise tracing and Observability in multi-agent IOA systems?

• How do we retain the identity of the running job of an AI Agent instance if the communication standard is not unified in different parts of the pipeline?

Summary.

• Agentic Systems might look simple from the outside, but it takes a lot of effort to bring them too production reliably.

• Proliferation of Vendors in the AI space due to the heat of the market is an amazing benefit we get as builders.

• You should carefully think about the minimal amount of additional infrastructure elements as you move along the maturity of your AI application.

• Adopting Evaluation Driven development is key. Especially when building for enterprises.

Hope you enjoyed this article and hope to see you next Wednesday!

Subscribe now

Partner with SwirlAI

Share

👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in AI Engineering, Data Engineering, Machine Learning and overall Data space.

Agentic AI and other buzzwords are emerging almost monthly if not more often. In reality they all describe different variations of Agentic Systems, it might be n agentic workflow or multi-agent system, it’s just a different topology under the same umbrella.

If you are considering a career in AI Engineering in 2025, it might feel overwhelming and that is completely normal.

But you need to remember - you are not too late to the game. The role as such has only emerged over the past few years and is still rapidly evolving.

In order to excel in this competitive space, you will need a clear path and focused skills.

Here is a roadmap you should follow if you want to excel as an AI Engineer in today’s landscape.

Join me tomorrow (June 5th) in a free live webinar where I will go through how to Deploy Reliable AI Systems with LLMOps.

Sign up for free

Fundamentals - learn as you go.

I have always been a believer that learning fundamentals is key to your career growth. This has not changed.

However, I have to admit that the game itself has changed with the speed that the industry is moving forward. Staring of with fundamentals before anything else is no longer an option. Hence, you should be continuously learning them as you build out modern AI Engineering skillset.

Here is a list of concepts and technologies I would be learning and applying in my day-to-day if I were to start fresh.

Python and Bash:

• FastAPI - almost all of the backed services implemented in Python are now running as FastAPI servers.

• Pydantic - the go to framework for data type validation. It is now also a Python standard for implementing structured outputs in LLM based applications.

• uv - the next generation Python package manager. I haven’t seen any new projects not using it.

• git - get your software version control fundamentals right.

• Asynchronous programming - extremely important in LLM based applications as your Agentic topologies will often benefit from calling multiple LLM APIs asynchronously without blocking.

• Learn how to wrap your applications into CLI tools that can be then executed as CLI scripts.

Statistics and Machine Learning:

• Understand the non-deterministic nature of Statistical models.

• Types of Machine Learning models - it will help you when LLMs are not the best fit to solve non-deterministic problem.

• General knowledge in statistics will help you in evaluating LLM based systems.

• Don’t get into the trap of thinking that AI Engineering is just Software Engineering with LLMs, some maths and statistics is involved.

LLM and GenAI APIs.

You should start simple, before picking up any LLM Orchestration Framework begin with native client libraries. The most popular is naturally OpenAI’s client, but don’t disregard Google’s genai library, it is not compatible with OpenAI APIs but you will find use cases for Gemini models for sure.

So what should you learn?

Types of LLMs:

• Foundation vs. Fine-tuned.

• Code, conversational, medical etc.

• Reasoning Models.

• Multi-Modal Models.

Structured outputs:

• Learn how OpenAI and Claude enforces structured outputs via function calling and tool use.

• Try out simple abstraction libraries like Instructor - they are enough for most of the use cases and uses pydantic for the structure definition natively.

Prompt Caching:

• Learn how KV caching helps in reducing generation latency and costs.

• Native prompt caching provided by LLM providers.

• How LLM serving frameworks implement it in their APIs (e.g. vLLM).

Model Adaptation.

I love the term Model Adaptation. The first time (and maybe the only time) I’ve seen it in literature was in the book “AI Engineering” by . The term ideally encompasses what we, AI Engineers, do to make LLMs perform actions we expect.

What should you learn?

Prompt Engineering:

• Learn the proper prompt structure. It will differ depending on the provider you are using.

• Understand context size limitations.

• Prompting techniques like Chain of Thought, Tree of Thought, Few-shot.

• Advanced prompting techniques: Self-consistency, Reflection, ReAct.

Tool Use:

• Tool Use is not magic, learn how it is implemented via context manipulation.

• Don’t rush to agents yet, learn how LLMs are augmented with tools first.

• You might want to pick up a simple LLM Orchestrator Framework at this stage.

Finetuning:

• Learn when it is worth to Finetune vs. just Prompt Engineering or implementing RAG. In most cases it is not worth the effort.

• Try out tools like Unsloth for quick learning if you do decide to get your hands dirty.

Subscribe now

Storage and Retrieval.

Vector Databases:

• Learn strengths and weaknesses of vector similarity search.

• Different types of Vector DB indexes: Flat, IVFFlat, HNSW.

• When PostgreSQL pgvector is enough.

Graph Databases:

• High level understanding about Graph Databases.

• Don’t spend too much time here as there is still limited use for Graph DBs even though the promises connected with Graph Retrieval were and still are big.

• Current challenges still revolve around the cost of data preparation for Graph Databases.

Hybrid retrieval:

• Learn how to combine the best from keyword and semantic retrieval to get the most accurate results.

RAG and Agentic RAG.

Data Preprocessing:

• Learn data clean data before computing Embeddings.

• Different chunking strategies.

• Extracting useful metadata to be stored next to the embeddings.

• Advanced techniques like Contextual Embeddings.

Data Retrieval, Generation and Reranking:

• Experiment with amount of data being retrieved.

• Query rewriting strategies.

• Prompting for Generation with retrieved Context.

• Learn how reranking of retrieved results can improve the accuracy of retrieval in your RAG and Agentic RAG systems.

MCP:

• Agentic RAG is where MCP starts to play a role, you can implement different data sources behind MCP Servers. By doing so you decouple the domain responsibility of the data owner.

LLM Orchestration Frameworks:

• You don’t need to rush with choosing Orchestration Framework, most of them hide the low level implementation from you and you would be better off starting out without any Framework whatsoever and using light wrappers like Instructor instead.

• Once you want to pick up and Orchestrator, I would go for the popular ones because that is what you run into in the real world:

  • LangChain/LangGraph.

  • CrewAI.

  • LlamaIndex

  • Test out Agent SDKs of Hyper-scalers and AI Labs.

AI Agents.

AI Agent and Multi-Agent Design Patterns:

• ReAct.

• Task Decomposition.

• Reflexion.

• Planner-Executor.

• Critic-Actor.

• Hierarchical.

• Collaborative.

• …

Memory:

• Learn about Long and Short-Term memory in Agentic Systems and how to implement it in real world.

• Try out mem0 - the leading Framework in the industry for managing memory. It now also has an MCP server that you can plug into your agents.

Human in or on the loop:

• Learn hoe to delegate certain actions back to humans if the Agent is not capable to solve the problem or the problem is too sensitive.

• Human in the loop - a human is always responsible for confirming or performing certain actions.

• Human on the loop - the Agent decides if human intervention is needed.

A2A, ACP, etc.:

• Start learning Agent Communication Protocols like A2A by google or ACP by IBM.

• There are more Protocols popping out each week, but the idea is the same.

• Internet of Agents is becoming a real thing. Agents are implemented by different companies or teams and they will need to be able to communicate with each other in a distributed fashion.

Agent Orchestration Frameworks:

• Put more focus on Agent Orchestration Frameworks defined in the previous section.

Master this roadmap together with me in the End-to-End AI Engineering Bootcamp (𝟭𝟬% 𝗱𝗶𝘀𝗰𝗼𝘂𝗻𝘁 𝗰𝗼𝗱𝗲: Kickoff10 )

AI Engineering Bootcamp

Infrastructure.

Kubernetes:

• Have at least basic understanding of Docker and Kubernetes.

• If your current company does not use K8s, it is more likely you will run into the one that does use it rather than the opposite.

Cloud Services:

• Each of the major cloud providers have their own set of services meant to help AI builders:

  • Azure AI Studio.

  • Google Vertex AI.

  • AWS Bedrock.

CI/CD:

• Learn how to implement Evaluation checks into your CI/CD pipelines.

• Understand how Unit Eval Tests are different from Regression Eval Tests.

• Load test your applications.

Model Routing:

• Learn how to implement Model fallback strategies to make your

• Try tools like liteLLM, Orq or Martian.

LLM Deployment:

• Learn basics of LLM deployment Frameworks like vLLM.

• Don’t focus too much on this as it would be a rare case that you would need to deploy your own models in real world.

Observability and Evaluation.

AI Agent Instrumentation:

• Learn what SDKs exist for instrumenting Agentic applications, some examples:

  • Langsmith SDK.

  • Opik SDK.

  • Openllmetry.

  • …

• Learn Multi-Agent system Instrumentation. How do we connect traces from multiple agents into a single thread.

• You can also dig deeper into OpenTelemetry because most of the modern LLM Instrumentation SDKs are built on top of it.

Observability Platforms:

• There are many Observability platforms available off the shelf, but you nee to learn the fundamentals of LLM Observability:

  • Traces and Spans.

  • Evaluation datasets.

  • Experimenting with changes to your application.

  • Sampling Traces.

  • Prompt versioning and monitoring.

  • Alerting.

  • Feedback collection.

  • Annotation.

Evaluation Techniques:

• Understand the costs associated with LLM-as-a-judge based evaluations:

  • Latency related.

  • Monetary related.

• Know in which step of the pipeline you should be running evaluations to get most out of it. You will not be able to evaluate every run in production due to cost constraints.

• Learn alternatives to LLM based evaluation:

  • Rule based.

  • Regex based.

  • Regular Statistical measures.

Recently, I wrote a piece on building and evolving your Agentic Systems. The ideas I put out are very tightly connected with being able to Observe and Evaluate your systems as they are being built out. Read more here:

Security.

Guardrails:

• Learn how to guardrail inputs to and outputs from the LLM calls.

• Different strategies:

  • LLM based checks.

  • Deterministic rules (e.g. Regex based).

• Try out tools like GuardrailsAI.

Testing LLM based applications:

• Learn how to test the security of your applications.

• Try to break your own Guardrails and jailbreak from system prompt instructions.

• Performing advanced Red Teaming to test emerging attack strategies and vectors.

Looking Forward.

The future development of Agents will be an interesting area to observe. A lot of successful startups are most likely to succeed due to having one of the following:

• Distribution.

• Good UX.

• Real competitive motes, like physical products. Here is where robotics comes into play.

Voice, Vision and Robotics:

• An interesting blend of capabilities that would allow a physical machine to interact with the world. The areas that I am looking forward to are:

  • On-device Agents.

  • Extreme Quantisation techniques.

  • Foundation Models tuned specifically for robotics purposes.

Automated Prompt Engineering:

• New techniques are emerging that allow you to perform automated Prompt Engineering given that you have good test datasets ready for evaluation purposes.

• Play around with frameworks like DsPy or AdalFlow.

Summary.

The skillset requirements for AI Engineers are becoming larger every month. The truth is that in your day-to-day you will only need a subset of it.

You should always start with your immediate challenges and adapt the roadmap accordingly.

However, don’t forget to look back and learn the fundamental techniques that power more advanced systems. In many cases these fundamentals are hidden behind layers of abstraction.

Happy building!

Hope you enjoyed this article and hope to see you next Wednesday!

Master this roadmap together with me in the End-to-End AI Engineering Bootcamp (𝟭𝟬% 𝗱𝗶𝘀𝗰𝗼𝘂𝗻𝘁 𝗰𝗼𝗱𝗲: Kickoff10 )

AI Engineering Bootcamp

Subscribe now

Partner with SwirlAI

Share

👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in GenAI, MLOps, Data Engineering, Machine Learning and overall Data space.

SwirlAI Newsletter is a reader-supported publication. To receive new posts and support my work, consider becoming a subscriber.

I have been developing Agentic Systems for around two years now. The same patterns keep emerging again and again, regardless of what kind of systems are being built.

I have learned them the hard way and many do so as well. The first project is not a great success, but you learn from the failures and apply the learnings in the next one. Then you iterate.

Today, I am sharing my system of how to approach development of LLM based applications from idea to production. Use it if you want to avoid painful lessons in your own projects.

In the Newsletter episode I will cover:

• The Evaluation driven application development system.

• Moving from Prototype to PoC to MVP.

• Evolving your Agentic Systems.

• Role of LLMOps in development of AI Agents.

Before we move forward, I wanted to give a shoutout to MLOps Community and my friend Demetrios for organising amazing events that help move the AI industry forward. A free event that will be happening in San Francisco and I am looking forward - AI Agent Builders Summit: World Tour Kickoff.

The event brings together builders, researchers, and doers working on AI Agents in Production in the most lo-fi way possible. ​Only engineers. No marketers. A great line-up including Anthropic, Boundary ML, Cleric and more.

Highly encourage to register if you are in San Francisco on May 28th.

Register for Free.

AI Product Development Lifecycle.

Below is a high level diagram that represents the development lifecycle of a modern AI Product.

The goal of today is to go step by step and highlight the key considerations you should have in mind if you choose to implement the same path in your projects.

Let’s start!

Defining The Problem.

The first step of any Agentic System development is defining the problem you want to solve. Agentic System is one where Large Language Models or other GenAI models are used to solve complex, real-world problems, in many cases dealing with automation.

It's vital to ensure the problem is clearly defined, bounded and aligned with business goals. Some questions to consider:

• Is the problem best solved by AI or traditional software?

• Who is the end user?

• What are the edge cases?

• What are the boundaries of acceptable behaviour?

Roles to involve: AI Product Managers, Domain Experts, AI Engineers.

Important: Many AI projects fail not due to bad models, but due to solving the wrong problem.

Building a Prototype.

After you know the problem is a good fit to be solved using AI, start rapid prototyping. In different size organisations this stage can be handled by different people. I am a strong believer of full-stack AI Engineering where AI Engineers prototype and communicate with stakeholders directly, but in large enterprises this is also where AI Product Managers can have a big impact.

Some points to consider:

• Use Notebooks or no-code tools, small datasets and off-the-shelf models.

• This is where you learn rather than focus on performance.

• Document everything so that you don’t repeat the mistakes in the future.

• This is where you will do a lot of prompting and researching market for tools that could potentially help in solving the problem (e.g. Voice to Text platforms).

Roles to involve: AI Product Managers, AI Engineers.

Important: This phase can be treated as a de-risking practice. While your idea might look good on the paper, it might not be technically feasible long term.

Defining Performance Metrics.

You are never just building a cool application, you are solving a real business problem that needs to be grounded in specific metrics you are planning to optimise.

Key points to consider:

• What is that you are trying to optimise for? E.g. reduce headcount, improve user satisfaction, increase development velocity.

• The above is your north star output metric, now you need to split it into input metrics that can actually drive the output metric forward. E.g. reduce an average time to Customer Support ticket resolution.

• Your application should be targeting the input metrics, but the output metric will be what really matters for the business.

Roles to involve: AI Product Managers, AI Engineers, Business Stakeholders.

Important: Without setting this stage up properly you risk the project being deprioritised for not showing enough business value. Remember to align with business before you start implementing, it is not enough to simply have it written down on paper.

Defining Evaluation Rules.

Metrics for LLMs are notoriously tricky. E.g. human alignment, coherence and factuality. However there are also exact rules that are extremely useful in evaluating your applications.

Some points to consider:

• Very likely your system will be composed of multiple LLM calls chained together within an Agentic System topology.

• For any node in your Agentic System topology you should have an evaluation dataset prepared: Inputs → Expected Outputs.

• Define unacceptable responses. E.g. toxicity, hallucinations, unsafe suggestions.

Roles to involve: AI Product Managers, AI Engineers.

I will be teaching how to apply the system described in this blog hands-on and in detail as part of End-to-End AI Engineering Bootcamp (𝟭𝟬% 𝗱𝗶𝘀𝗰𝗼𝘂𝗻𝘁 𝗰𝗼𝗱𝗲: Kickoff10 )

Check it out

Building a PoC.

This is a stage that is often misunderstood. You might be driven to transition from prompts to a functioning interface (CLI, chat UI, API, etc.). However, the goal here is pushing out the system to the users as soon as possible.

Points to consider:

• Use LLM APIs from OpenAI, Google, Anthropic, X etc. to quickly build out the first user facing application.

• Your application could be an Excel Spreadsheet with Input Output pairs rather than a full fledged functioning interface. As long as it helps moving metrics forward, it is good to be exposed.

• The feedback you get from users is key to understand unknown unknowns. In my experience it almost always shifts your perspective of how to improve the application.

Roles to involve: AI Engineers.

Important: A successful LLM PoC may look like an Excel Spreadsheet. If you can’t push it out quick, there is something wrong with the process.

Instrumenting the Application.

This is where we implement best practices of Observability in LLM based systems. We do this by instrumenting the application and logging extensive set of metadata about everything that is happening underneath the surface.

Key considerations:

• Log everything: prompts, completions, embeddings, latency, token counts, and user feedback.

• Add additional metadata like: prompt versions, user inputs, model versions used.

• Make sure that the chains are properly connected and you know the ordering of operations.

• When working with multimodal data log different kinds of data like PDFs, Image, Audio, Video.

• Remember that outputs of one LLM call will often become inputs to the next one.

• Don’t forget the user feedback! Always attach it to the traces that represent the run users were interacting with when the feedback was provided.

Roles to involve: AI Engineers.

Important: This stage is one of the key elements in implementing Evaluation Driven Development.

Integrating with an Observability Platform.

It is not enough to just track the data, you need to be able to efficiently visualise and analyse it. This is where Observability platform come into play they help with efficient search and visualisation as well as prompt versioning and adding automated evaluation capabilities.

Key considerations:

• Store your Evaluation rules as part of the platform as you will later apply them on the traces.

• Use these platforms as Prompt Registries as your application is a chain of prompts, you will want to analyse and group the evaluation results by Prompt Groups.

• Most successful applications reach scale and it becomes too expensive to store all of the traces produced. Observability Platforms have smart sampling algorithms that allow you to store subset of incoming traces.

• Most of the Observability Platforms come with their own tracing SDKs, use them for seamless Instrumentation.

Roles to involve: AI Engineers.

Important: Set this up early as it will bring the visibility to the black box.

Subscribe now

Evaluating Traced Data.

If you have successfully implemented the previous stages, you will have all of the required pieces to successfully measure your application that has been exposed to the users. Here is where you run Evals on top of the trace data.

Key considerations:

• Assumption: You have your Evaluation rules stored in the Observability platform together with the incoming traces via your instrumented application and human feedback attached to corresponding traces.

• Run the Evals automatically on the traces that hit the Observability Platform.

• Filter out all of the traces that have failing evals or negative human feedback. It is up to you to decide what a failing eval or negative feedback is.

• We will focus mostly on this “failing” data moving on.

Roles to involve: AI Engineers.

Important: Running Evals can often be expensive, especially if you are using LLM as a judge tactic in some places. You might want to sample the traces you would be running evals on.

Evolving the Application.

You are now ready to improve your application if you have the above figured out.

Key considerations:

• Focus on Failing Evaluations and human feedback to pinpoint where the improvement is needed.

• If your current topology is not up to the task, make it more complex: Simple Prompts → RAG → Agentic RAG → Agents → Multi-agent systems.

• Make the system more complex only if there is a hard requirement, focus on better prompt engineering, data preprocessing, tool integration.

• You can read more about the evolution of Modern RAG Systems in one of my previous blogs here:

• A powerful concept in Evaluation Driven Development is having a failing eval dataset that you would never solve for 100%. Your goal is to achieve that but by adding more and more failing samples you never get to that 100%.

Roles to involve: AI Engineers, Domain Experts.

Important: Always involve Domain Experts in this stage, they have the insider knowledge about tasks you want to automate, they can even suggest better prompts to solve the problem.

Exposing new version of the Application.

This one is quick - expose new versions of the application as fast as possible. The feedback you will be getting on the updates is invaluable.

Key considerations:

• Deploying new versions fast is important for few reasons:

  • It improves UX as the present problems get fixed.

  • Some fixes will generalise to unknown problems and you will be solving multiple bugs with one shot.

• Be sure to have strict release tests. You should always have evaluation datasets ready so that you know that what is being released is not worse compared to previous version. Integrate these checks into your CI/CD pipelines.

Roles to involve: AI Engineers, Domain Experts.

Share

Continuous Development and Evolution of the Application.

The highlighted loop in the image is the key part of the process when it comes to continuous evolution of your application

Key considerations:

• Remember: Build → Trace, collect feedback → Evaluate → Focus on Failing Evals and Negative Feedback → Improve the application → Iterate.

• As your business requirements become more complex, you might add additional functionality to the application. Very often this would be added as a new route in your Agentic System Topology. E.g. a simple chatbot evolving into a system that can manage your shopping cart automatically on your instruction.

• In order to add additional functionalities you should follow the process of building a prototype, defining performance metrics and new evals.

• This is where multiple AI Engineers can start more easily work on a single project as new independent routes can be developed as a separate functionality.

Roles to involve: AI Engineers, Domain Experts, AI Product Managers.

Monitoring and Alerting.

After you have implemented all of the tracing end evaluation for development purposes, monitoring almost comes out of the box - you have implemented evaluations and traces for development purposes, they can be reused for monitoring. Configure specific alerting thresholds and enjoy the peace of mind.

Key considerations:

• You already have most of the data that is relevant for LLM specific production monitoring if you have properly implemented application instrumentation.

• If you haven’t yet, consider tracing and logging additional advanced metric like TTFT (Time To First Token), inter-token latency etc.

• You will need to figure out the threshold for alerting.

Roles to involve: AI Engineers.

Important: Try to avoid Alert Fatigue by carefully configuring thresholds that would kick off alerts. Avoid False Positives as much as possible.

Wrapping up.

• Development of LLM based Agentic Systems is different from traditional software and techniques like Evaluation Driven Development are key for success.

• Observability and Evaluation is key and should be implemented early in the project lifecycle.

• Having well defined business success metrics up front will help you avoid reprioritisation of your projects long term and keep the business buy in.

• Figure out if what you are about to build is feasible early on and drop the idea if it is not. There are too many low hanging fruits with high impact, use your time wisely.

Hope you enjoyed the writeup and hope to see you in the next one!

I will be teaching how to apply the system described in this blog hands-on and in detail as part of End-to-End AI Engineering Bootcamp (𝟭𝟬% 𝗱𝗶𝘀𝗰𝗼𝘂𝗻𝘁 𝗰𝗼𝗱𝗲: Kickoff10 )

AI Engineering Bootcamp

Subscribe now

Partner with SwirlAI

Share

RAISE Summit is approaching and I am proud to be an official ambassador for the event.

Want to Join?

You can win a ticket by filling out this form:

Fill out the Form

Or secure your place with discount code: amEV73 for 20% off.

👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in GenAI, MLOps, Data Engineering, Machine Learning and overall Data space.

SwirlAI Newsletter is a reader-supported publication. To receive new posts and support my work, consider becoming a subscriber.

After months of planning, conversations with dozens of engineers, and many late nights building and iterating I finally have some big news!

I’m incredibly excited to finally open enrolment for my new cohort-based course: the End-to-End AI Engineering Bootcamp.

You can check it out an enrol here:

End-to-End AI Engineering Bootcamp

• This course is structured around weekly sprints - just like real AI teams work.

• Each week, you’ll scope a feature, build it, test it, and improve it.

• From RAG pipelines to agent orchestration and deployment, we’ll move through the same iterative process used by AI engineers shipping real systems.

Over 8 weeks, you will:

• Build a full-stack AI app (from idea -> deployed).

• Master RAG, agentic and multi-agent systems.

• Implement LLMOps best practices: Evaluation, Observability, continuous feedback loops and more.

• Learn automated prompt engineering.

• Apply emerging technologies like MCP and A2A.

• Leave with a robust project you can actually showcase.
  

If you are:

• Data Analyst, Data Scientist, ML Engineer, Data Engineer or Software Engineer trying to re-skill and break into AI Engineering.

• AI Engineer that has some experience in building agentic systems but wants to up-skill.

• AI Leader that wants to up-skill their teams (encourage them to enrol!).

This bootcamp is for you.

We kick off June 23rd. Hope to see you there!

Free Lightning Lessons.

I will also be hosting a series of free lightning lessons along the way. The first one is happening this Thursday already.

Register here:

Breaking Into AI Engineering

In this session, I’ll walk through:

• What AI Engineering actually is (and what it’s not).

• The end-to-end process of building AI systems - from problem framing to deployment.

• The specific skills needed at each stage of that process.

• How to map your current experience into this evolving role.

If you’re a data scientist, ML engineer, software engineer, or technical team lead - this session is for you.

You’ll walk away with a clear mental model, a structured process, and a practical starting point to grow your impact in AI.

Subscribe now

Share

This recurring question inspired me to write a piece exploring the technical challenges involved in training LLMs with extended context windows.

Transformer-based models have revolutionised the field of deep learning. The ability to model long-range dependencies via self-attention is key in todays AI systems. As these models grow in capability we still face a key limitation: context length.

The problem: self-attention scales quadratically with sequence length. This means doubling the number of tokens more than quadruples the computational and memory load. While this cost may be manageable for short sequences, it quickly becomes unsustainable as models are trained on long-form content like legal documents, books, or extended conversations. And to achieve real (accurate) long context inference we need to train on long sequences or apply advanced techniques that help generalise to long sequences but have shown limitations.

In today’s episode we will explore:

• Memory Bottleneck of the Context Window.

• What is FlashAttention.

• Parallelism techniques in distributed LLM training.

• Combining FlashAttention and distributed training with Kvax.

This newsletter episode was made possible by today’s sponsor, Nebius - a leader in AI Cloud technology.

Recently Nebius has open sourced Kvax, their FlashAttention implementation based on JAX. Designed for efficient training with long sequences, Kvax supports context parallelism and optimised computation of document masks. It outperforms many other Flash Attention implementations in long-context training with dense packing.

In the second part of the article we will explore how Kvax supercharges distributed training with long input sequences. I highly recommend checking out the GitHub repository if you are innovating in this area.

Check out Kvax

Explaining the Memory Bottleneck of the Context Window.

The problem is as old as the Transformer architecture itself. Almost from the very beginning we have tried to stretch the context window of the LLMs to overcome the limitations that make some applications not feasible.

The problem with extending context windows lies within the architecture of Transformer itself, to be more precise - the Self-Attention mechanism that is the core piece of the architecture. Let’s look into the original Transformer description from the famous “Attention is all you need” paper:

The architecture of the network might seem complex from the first glance, but from the implementations side it is just a long sequence of matrix multiplications. To understand where the memory bottleneck of the Context window happens we would need to zoom into the “Scaled Dot-Product Attention” piece of the graph.

We will not go into the details of each computation step but rather focus on the part that is most affected by the length of the sequence which is forward passed through the network.

Let’s say you have a sequence of five tokens “He does not know what” (important: for simplicity sake we are assuming that each of the words result in a single token after the tokenisation step. This is not always the case and a single word can be divided into multiple tokens). The sequence of length 5 will always be transformed into a matrix of size 5x5 when attention operations are performed.

And this is where the scalability challenge of self-attention becomes particularly problematic: the infamous quadratic growth in both memory and computational requirements. In the self-attention mechanism, each token in the input sequence attends to every other token, which requires constructing an attention matrix whose size is proportional to the square of the sequence length. Concretely, if the input sequence has a length of n, the attention matrix is of size n x n. As a result, when we double the length of the input - say from 5 to 10 - the size of the attention matrix increases from 25 to 100, leading to a fourfold increase in both memory consumption and the number of computations needed to compute attention scores. This quadratic scaling makes it computationally prohibitive to process long sequences using standard transformers, especially in environments with limited memory or latency constraints.

Attention Mask.

Before we proceed with analysing the techniques designed to address the memory bottleneck of the context window in transformer models, it's important to first introduce the concept of the causal attention mask, as it plays a critical role in many of those solutions.

A causal attention mask is a type of filtering mechanism applied to the attention matrix during computation. Its purpose is to ensure that each token in the input sequence can only attend to itself and to tokens that come before it - never to future tokens. This is typically implemented by applying a lower triangular matrix (as shown in the image below), where positions representing attention to future tokens are masked out, effectively assigning them zero weight.

Most LLMs we use today - including models in the GPT family - rely on this form of masked attention. It ensures that during both training and inference, the model predicts the next token based only on the preceding context.

Now let’s explore one of the most widely accepted way of mitigating some of the memory bottleneck of the context window - FlashAttention.

Subscribe now

Enter FlashAttention.

There have been multiple iterations of FlashAttention released to date - v1, v2 and v3. However, in this article, instead of focusing on specific implementations we will focusing on the core techniques and architectural ideas of FlashAttention which aim to mitigate the memory and compute bottlenecks in standard self-attention:

• I/O Awareness & Fused Kernels.

• Tiling.

• Dense Packing.

• Skipping Blocks.

Together, these techniques significantly reduce the quadratic memory and compute cost of vanilla self-attention, making it feasible to train and deploy large models on longer sequences and with greater efficiency. Let’s explore how.

Fused Kernel.

A Kernel in accelerated compute programming is a function or operation you launch on the node to perform a specific task in parallel.

Modern accelerated compute nodes have a hierarchy of memory:

• The main memory (DRAM) which is large but relatively slow to access.

• High Bandwidth Memory (HBM).

• Much smaller on-chip memory (like registers, L1 cache, shared memory SRAM) which is very fast.

Remembering the computations involved in attention heads, we have multiple sequential matrix multiplications. With the standard attention implementation without I/O awareness we would load the intermediate results back and forth from HBM to SRAM and vice versa incurring unnecessary reads and writes between different layers of memory.

There are two main problems with this:

• Memory (IO) bottleneck: copying these big matrixes (especially when sequences are long) to and from slow memory is expensive and quickly becomes the bottleneck.

• Kernel launch overhead: each of the attention sub-steps above (matrix multiply, softmax, etc.) would be a separate kernel launch. Launching a GPU kernel has some fixed overhead (like setting up the execution on thousands of threads). If you have to launch multiple kernels one after the other, you pay that overhead every time.

A Fused Kernel is simply combining multiple operations into one kernel. Instead of launching one kernel for the matrix multiplication, another for softmax, and another for the value multiplication, all these steps are fused into a single kernel.

As shown in the below graph, Fused Kernel with other improvements in FlashAttention reduce the runtime significantly.

Subscribe now

Tiling.

Tiling in FlashAttention is an optimisation strategy that partitions large attention matrices into smaller, more manageable sub-blocks which are processed independently. This approach enables efficient use of high-speed memory hierarchies within accelerated compute nodes. By loading each block into fast, local memory and reusing it for multiple operations, tiling minimises the need to access slower, large-scale memory repeatedly.

Clearly, as the sequence length increases, tiling becomes increasingly critical because it restructures the computation of the attention mechanism in a way that significantly reduces its computational and memory overhead. Traditionally, attention scales quadratically with sequence length, making it prohibitively expensive for long sequences. However, by breaking the attention matrix into smaller tiles and computing attention locally within these blocks tiling effectively transforms the quadratic complexity into a form that approaches linear scalability.

Skipping Blocks.

Remember the causal attention masks? This is where they become important in addressing memory bottlenecks of the context window. In causal attention, the lower triangular part of the attention matrix is masked by setting the corresponding attention weights to zero. These masked values effectively have no impact on the final output, making their computation unnecessary.

When applying tiling, this property becomes especially beneficial: if an entire tile or block of the attention matrix falls within the masked region, it can be completely skipped during computation.

Dense Packing.

In traditional implementations, attention computation often involves padding sequences to a uniform length, which leads to wasted computation on these padded tokens.

Imagine two sequences:

• “He does not know what is best for him and his friends.”

• “I am the second sample nice to meet you.”

This is how the attention matrix would look like for the first sequence if we fix the sequence length at 20. Notice all of the padding tokens required.

This is how the second sequence would be represented:

Here is where dense packing comes in. The technique allows you to pack multiple sequences into a single batch. Notice how the masked areas are placed. Dense packing addresses this inefficiency by grouping variable-length sequences together in a way that fills memory blocks as completely as possible, minimising idle compute and reducing memory overhead.

Combined with Tiling, Block Skipping and Kernel Fusion the algorithm achieves maximal efficiency gains.

In practical terms, FlashAttention can handle much longer sequences on a single accelerated compute node before running out of memory, and it can be up to several times faster than the naive attention implementation. However, FlashAttention by itself does not parallelise across multiple nodes - it’s about optimising attention on one device.

Parallelism in Distributed LLM Training.

To use FlashAttention for extremely long contexts that exceed a single node’s capability, or to speed up training by using multiple nodes, we need to parallelise the FlashAttention computation across devices.

Training LLMs often requires splitting work across multiple accelerated compute nodes due to the enormous model sizes and sequence lengths. There are multiple techniques applied to achieve this:

• Data parallelism: the simplest approach - each node gets a different subset of the training data (mini-batch), and a full copy of the model. After each step, gradients are synchronised to keep model replicas in sync. However, data parallelism doesn’t reduce per-node memory or compute load of the model itself, it just uses more data in parallel.

To distribute the model’s workload, model parallelism is used. There are generally two forms of model parallelism:

• Tensor (intra-layer) parallelism: splitting the computations within a layer across devices. For example, splitting a large matrix multiplication or the set of attention heads across nodes. In attention, this could mean each node computes a subset of the attention heads independently, then their outputs are combined. This reduces per-node computation, but it has limitations – e.g. the number of heads might be smaller than the number of available nodes.

• Pipeline (inter-layer) parallelism: splitting the model’s layers among different nodes. Each node holds a consecutive chunk of the network layers. The micro-batch of data flows through the pipeline: first node processes the first few layers, then passes activations to the next node for the next layers, and so on.

In practice, state-of-the-art training combines all types of parallelism. For example, one might use tensor parallelism within each node, pipeline parallelism across nodes, and data parallelism for even higher scale. This is usually done due to minimal network speed considerations required for different parallelism types.

Another strategy relevant for long sequences is sequence parallelism (also called context parallelism). This involves splitting the input sequence length across nodes. Each node then handles a different chunk of the sequence tokens. The challenge is that certain operations (like self-attention) are not independent across sequence chunks and require communication.

Subscribe now

Kvax by Nebius.

Kvax makes FlashAttention “distributed-friendly” through several key techniques:

• Context (Sequence) Parallelism with All-Gather: It splits the sequence across nodes and uses an all-gather of Key/Value tensors so each node obtains the full context before computing attention. This allows each node to run the FlashAttention algorithm on its local queries with all necessary data, just as if it were a single-device run.

• Grouped Query Attention (GQA): By using GQA (grouping heads to share K/V), the size of keys and values to communicate is reduced. This keeps the all-gather overhead very low relative to computation, making scaling efficient. The attention computation remains the dominant cost, so multiple nodes can collaborate with minimal slowdown.

• Fused high-performance kernels: Kvax’s implementation in Triton ensures that on each node the FlashAttention computation (and mask application) is done in a tiled, memory-efficient manner, just like single-node FlashAttention. It also splits and optimises the backward pass, using all-reduce to accumulate gradients, so multi-node training has the same mathematical correctness with minimal overhead.

• Support for complex masks: The approach naturally accommodates causal masks and packed-sequence (document) masks by computing masks on the fly and by virtue of each node having all keys/values (making mask logic simpler). This is crucial for real training scenarios where you concatenate multiple sequences or need attention window constraints.

• Load balancing across sequence chunks: Kvax employs token shuffling across chunks (per LLaMA3’s recipe) to ensure each node does a similar amount of work. This prevents any single shard from becoming a hotspot (for example, the node handling the last part of a long sequence isn’t overwhelmed by having to attend to all preceding tokens).

• Combined parallelism: Kvax doesn’t force you to choose one parallelism over another - it supports using data parallelism simultaneously with model parallelism. For instance, you could use tensor parallelism to split heads across GPUs and use context parallelism to split a very long sequence, in conjunction with data parallel across multiple such groups. This flexibility means Kvax can fit into many multi-dimensional parallel setups for large-scale training.

In essence, Kvax enables large-scale training of transformers with long contexts by efficiently distributing the attention computation.

Kvax demonstrates that FlashAttention can indeed be scaled across multiple devices without losing its efficiency edge. By addressing communication overhead and preserving memory-optimal computation, it allows training of LLMs with extremely long sequences (potentially 10k-100k tokens) on clusters of nodes while also maintaining high throughput.

Don’t forget to check out the open source library:

Check out Kvax

Wrapping up.

I intentionally kept this article high level enough to make it easy to skim through and build understanding of the difficulties of training large context window LLMs. Here are some general conclusions that I came to while writing this piece (some of the supporting facts are not part of the newsletter):

• We have moved quite far in solving the Memory Bottleneck of the Context Window in the past few years after the emergence of LLMs.

• FlashAttention is the standard technique to solve some of the Bottleneck problems and is included as a default in most of the training libraries.

• The distributed training while using FlashAttention is not trivial. While it is being internally solved in big AI Labs, companies like Nebius are doing big favour to the community by open sourcing their implementations that help in distributed training.

• There are much research on the way that is targeting ability of LLMs generalising to longer sequences even when trained on shorter sequences. This in combination with Distributed FlashAttention might allow truly long context windows that actually work.

• The problems of “Lost in the middle” and “Needle in the haystack” have not been solved yet, but it seems it could be in the future.

Hope you enjoyed the writeup and hope to see you in the next one!

Subscribe now

Partner with SwirlAI

Share

The success of MCP (Model Context Protocol) by Anthropic has clearly inspired other players in AI industry to join the race to be the ones to define open protocols that will be used in Agentic Systems integration.

This week Google comes out publicly with their open protocol called A2A (Agent2Agent) that is aiming to normalise how multi-Agent systems communication is implemented. Many are saying (misinterpreting?) that the two protocols are competitive rather than complimentary.

The public stance by Google is that A2A is complimentary to MCP. That is a reasonable statement. However, could there be hidden long term competitive goals? Will we see protocol wars start soon?

I have been asked multiple time how I believe the two protocols might become competitive in the future. Read until the end to get my thoughts on it.

In this Newsletter episode:

• What is A2A?

• What is MCP?

• How is A2A complimentary to MCP and vice versa?

• Could A2A eat up MCP long term?

A2A explained.

Let’s first define A2A as it is the new kid on the block (MCP already had its spotlight).

The problem.

It is becoming clear that Agentic Systems of the future will be multi-Agent. Even more, Agents will be collaborating between each other remotely, each of them potentially implemented using different Agent Frameworks (e.g. LangGraph, CrewAI, Agent Development Kit etc.).

There are few inherent problems with this:

• System State transfer and exchange between systems implemented in different frameworks is not supported.

• Transfer of the System State between remote Agents is also not possible.

• Disconnected Agents do not share Tools, Context and Memory (including System State).

The solution.

A2A is an open protocol that provides a standard way for agents to collaborate with each other, regardless of the underlying framework or vendor.

As per official Google documentation:

A2A protocol facilitates communication between “client” and “remote” agent.

In simple terms, “client” agent creates tasks and communicates them to the “remote” agent expecting some work to be performed or data returned.

Main A2A capabilities facilitating this:

• Capability discovery - all agents implementing A2A expose their capability catalogue via “Agent Card”. It helps other Agents discover potentially useful features implemented by the given Agent.

• Task management - communication protocol that facilitates short and long-running tasks. It helps communicating Agents to stay in-sync until the requested task is completed and the answer returned. This is big because some Agents might take long time to execute their work and there are no standards on how to wait for this to happen.

• Collaboration - Agents can send each other messages to communicate context, replies, artifacts, or user instructions.

• User experience negotiation - this one is pretty interesting. It allows negotiating on the format the data should be returned in order to fit the user UI expectations (e.g. Image, video, text etc.).

Discovery of agents exposed via A2A is a big topic. Google suggests a unified place to store organisations “Agent Cards”. E.g.:

https://<DOMAIN>/<agreed-path>/agent.json

This is not unexpected as Google would then be in the best spot to index all of the available Agents world wide potentially creating a global Agent Catalogue similar to current search index.

There has been a lot of talk about headless browsers and how the future internet of Agents will be implemented. The above is one of the ways it could happen by leveraging already existing technologies.

I love how A2A stresses the need to not reinvent the wheel and is building on existing standards:

• The protocol is built on top of existing, popular standards including HTTP, SSE, JSON-RPC, which means it’s easier to integrate with existing IT stacks businesses already use daily.

• Secure by default - A2A is designed to support enterprise-grade authentication and authorization, with parity to OpenAPI’s authentication schemes.

Subscribe now

MCP Explained.

MCP (Model Context Protocol) as defined by Anthropic is:

An open protocol that standardises how applications provide context to LLMs.

To be more precise it attempts to standardise the protocol on how LLM based applications integrate with other environments.

In Agentic systems the context can be provided in multiple ways:

• External data - this is part of long term memory.

• Tools - the capability of the system to interact with the environment.

• Dynamic Prompts - that can be injected as part of the system prompt.

• …

Why the need to standardise?

Current development flow of Agentic applications is chaotic:

• There are many Agent frameworks with slight differences. While it is encouraging to see the ecosystem flourish, these slight difference rarely add enough value but potentially significantly change the way you write code.

• Integrations with external data sources are usually implemented ad-hoc and using different protocols even within organisations. That is clearly true for different companies as well.

• Tools are defined in code repositories in slightly different ways. How you attach tools to augmented LLMs is different as well.

The goal is to improve the velocity of how fast we can innovate with Agentic applications, how well we can secure them and how easy it is to bring relevant data to the context.

Bellow is the high level architecture of MCP.

1. MCP Host - Programs using LLMs at the core that want to access data through MCP.

2. MCP Client - Clients that maintain 1:1 connections with servers.

3. MCP Server - Lightweight programs that each expose specific capabilities through the standardised Model Context Protocol.

4. Local Data Sources - Your computer’s files, databases, and services that MCP servers can securely access.

5. Remote Data Sources - External systems available over the internet (e.g., through APIs) that MCP servers can connect to.

Splitting control responsibilities through MCP.

MCP Servers expose three main elements that are purposely built in a way that helps implement specific control segregation.

• Prompts are designed to be User-Controlled.

  • Programmer of the server can expose specific prompts (suited for interaction with data exposed by the server) that can be injected into the application using LLMs and exposed to the user of the given application.

• Resources are designed to be Application-Controlled.

  • Resources are any kind of data (text or binary) that can be used by the application built to utilise LLMs. The programmer of the application (usually AI Engineer) is responsible of codifying how this information should be used by the application. Usually, there is not automation in that and LLM does not participate in this choice.

• Tools are designed to be Model-Controlled.

  • If we provide agency to our application of how it should interact with the environment we use tools to do that. MCP Server exposes an endpoint that can list all of the tools available with their descriptions and required arguments, application can pass this list to the LLM so that it can decide which tools are needed for the task at hand and how they should be invoked.

A2A + MCP.

As per official Google stance:

Agentic applications need both A2A and MCP. We recommend MCP for tools and A2A for agents.

What does it mean? Let’s look into an Agentic System architecture that involves multiple Agents.

Moving pieces in MCP:

1. MCP Host - This is where it gets interesting, when combined with A2A, MCP Host is the Agent.

2. MCP Client.

3. MCP Server.

4. Local Data Sources.

5. Remote Data Sources.

A2A:

6. Agents (MCP Hosts) would implement and communicate via A2A protocol, that enables:

   1. Secure Collaboration - MCP lacks authentication. [Update: recently, there has been many improvements to authentication in MCP: link].

   2. Task and State Management.

   3. User Experience Negotiation.

   4. Capability discovery - similar to MCP tools.

The suggestion is that MCP is used mostly for integrating legacy data systems (MCP Resources) and APIs (MCP Tools) with LLM based applications while A2A takes care of inter-Agent communication.

I do believe that as we move forward it will become more common to expose your platforms as Agents rather than MCP Servers so the importance of MCP in point 5. will gradually decrease.

Agent discovery via MCP.

Google goes as far as suggesting exposing A2A Agents via MCP server resources.

1. Each Agent in the mesh would be able to discover other available Agents by connecting to a dedicated MCP Server via a MCP Client and browsing the resource catalogue. The suggestion is to expose Agent Cards through these MCP resources.

2. Once discovered, Agents would continue communication between each other utilising A2A protocol.

Having said this, if we move towards Agent discovery via a global index, the importance of MCP here would also decrease or it could even disappear.

Subscribe now

Could A2A eat up MCP long term?

So is MCP at risk of fading into irrelevance?

For quite some time already there was a clear need for a medium that would connect swarms of Agents unleashed to the world between each other and to other legacy systems. There were talks about headless browsers, but it does seem like these open communication protocols will actually be the way forward. I believe that this is also where the talks about MCP being the new http moment (over-exaggerated?) were coming from.

The following thoughts are based on some assumptions:

• Open communication protocols will integrate the Agents of the new world.

• There is benefit in being behind the leading protocol.

• Both protocols will continue to evolve and potentially expand the boundaries of responsibility.

Similarities in MCP and A2A.

There are some clear similarities in bots protocols and a user could choose how to model their Agentic applications and expose them to the world in multiple ways.

With MCP skyrocketing in popularity it is becoming a norm for companies to ship MCP servers as part of their offerings so that developers can seamlessly integrate context from these platform into their own LLM based applications.

However, MCP is facing some issues in adoption.

• One of the biggest downsides of the protocol is the lack in security and authentication. You do need to tinker around the basic implementation to expose a remote MCP server securely.

• Tools can describe anything, including other Agents. Unfortunately, MCP does not implement any primitives that facilitate proper communication between Agents via tools (state/context exchange, long-running task support etc).

This is where Google might have found a wedge to enter the protocol wars with A2A by fixing the above.

I can not shake off the feeling that Anthropic had bigger plans for MCP than it’s current state. Including the interconnection of multiple Agents together. Now, the door to expand into this are might be closed due to emergence of A2A.

What is really important long term?

If we think long term, how will the Agentic world really be modelled?

• Companies expose their data assets to be used by Agents.

• Companies expose Agents that can return data or perform actions.

• Companies ARE Agents that other Agents can interact with.

I am betting on the movement towards the last option.

If the assumption holds, the real power is in the hands of the protocol that controls remote inter-Agent communication protocol.

Even short term, assuming the second point would be a default option for newly emerging companies, if one chooses to expose their data via an Agent, A2A is a clear winner.

Having said this, will MCP remain as the protocol to stitch new kinds of applications with legacy systems and will fade into irrelevance once Agents take over? Who knows, lets wait and see.

But if that happens, guess which player in the industry I am betting on :)

Wrapping up.

We are living in exciting times. The way how the new type of Agentic application will be connected at scale is being defined before our eyes.

A2A may be the new kid on the block, but it’s quickly shaping up to be the leader in inter-Agent communication. While MCP brought structure to how LLMs integrate context, A2A is solving for what MCP lacks: security, state management, and real-time collaboration. Will A2A eat MCP? Who knows.

While official stance is that the two protocols are solving completely different problems, there are potential overlaps and one could expect the protocols to also be expanded in scope.

If the future is Agentic, and companies begin exposing Agents rather than just tools or data, the protocol that enables seamless inter-Agent interaction might just be the winner. Right now, it seems like A2A might be choosing the right moves.

This is it for today, hope to see you in the next episode!

Subscribe now

Partner with SwirlAI

Share

RAG (Retrieval Augmented Generation) based systems have been, and continue to be, one of the the most useful applications utilising LLMs in enterprises. I remember writing the first post about RAG almost 2 years ago when the term itself was not yet widely adopted.

What I described back then was a RAG system implemented in the most naive way possible. The industry has evolved far since then by introducing different kinds of advanced techniques to the process.

In this Newsletter episode we will go through the evolution of RAG, from Naive to Agentic. After reading through you will understand what challenges have been tackled by each step in evolution.

The emergence of Naive RAG.

Naive RAG has emerged almost at the same time as the LLMs have become mainstream with introduction of ChatGPT at the end of 2022. The Retrieval Augmented Generation technique was brought to life in order to solve for issues that native LLMs faced. In short:

• Hallucinations.

• Limited Context window size.

• Lack of access to non-public data.

• Parametric knowledge limited to the latest data the model was trained on.

The simplest implementation of RAG can be defined in the following steps:

Preprocessing:

1. Split text corpus of the entire knowledge base into chunks - a chunk will represent a single piece of context available to be queried. Data of interest can be from multiple sources, e.g. Documentation in Confluence supplemented by PDF reports.

2. Use the Embedding Model to transform each of the chunks into a vector embedding.

3. Store all vector embeddings in a Vector Database. Save text that represents each of the embeddings separately together with the pointer to the embedding.

Retrieval:

4. Embed a question/query you want to ask using the same Embedding Model that was used to embed the knowledge base itself.

5. Use the resulting Vector Embedding to run a query against the index in the Vector Database. Choose how many vectors you want to retrieve from the Vector Database - it will equal the amount of context you will be retrieving and eventually using for answering the query question.

6. Vector DB performs an Approximate Nearest Neighbour (ANN) search for the provided vector embedding against the index and returns previously chosen amount of context vectors. The procedure returns vectors that are most similar in a given Embedding/Latent space. Map the returned Vector Embeddings to the text chunks that represent them.

7. Pass a question together with the retrieved context text chunks to the LLM via prompt. Instruct the LLM to only use the provided context to answer the given question. This does not mean that no Prompt Engineering will be needed - you will want to ensure that the answers returned by LLM fall into expected boundaries, e.g. if there is no data in the retrieved context that could be used make sure that no made up answer is provided.

Moving pieces in the Naive RAG system.

Even without any advanced techniques, there are many moving pieces to consider when building a production Grade RAG system.

Retrieval:

F) Chunking - how do you chunk the data that you will use for external context.

• Small, Large chunks.

• Sliding or tumbling window for chunking.

• Retrieve parent or linked chunks when searching or just use originally retrieved data.

C) Choosing the embedding model to embed and query and external context to/from the latent space. Considering Contextual embeddings.

D) Vector Database.

• Which Database to choose.

• Where to host.

• What metadata to store together with embeddings. The data to be used for pre-filtering and post-filtering.

• Indexing strategy.

E) Vector Search.

• Choice of similarity measure.

• Choosing the query path - metadata first vs. ANN first.

• Hybrid search.

G) Heuristics - business rules applied to your retrieval procedure.

• Time importance.

• Duplicate context (diversity ranking).

• Source retrieval.

• Conditional document preprocessing.

Generation:

A) LLM - Choosing the right Large Language Model to power your application.

B) Prompt Engineering - having context available for usage in your prompts does not free you from the hard work of engineering the prompts. You will still need to align the system to produce outputs that you desire and prevent jailbreak scenarios.

After all of this work we have ourselves a working RAG system.

The harsh truth - this is rarely good enough to solve real business problems. The accuracy of this kind of system might be low for various reasons.

Subscribe now

Advanced techniques to improve Naive RAG.

Some of the more successfully adopted techniques to continuously improve accuracy of Naive RAG systems are the following:

• Query Alteration - there are few techniques that can be employed:

  • Query rewriting - ask LLM to rewrite original query to better fit the retrieval process. It can be rewritten in multiple ways. E.g. fixing grammar, simplifying the query to keep short succinct statements.

  • Query Expansion - ask LLM to rewrite the query multiple times to create multiple variations of it. Then, run retrieval process multiple times to retrieve more, potentially relevant, context.

• Reranking - rerank the originally retrieved documents using heavier process compared to regular contextual search. Usually, this involves using a larger model and retrieving considerably more documents than needed during the retrieval phase. Reranking also works well with Query Expansion pattern from previous point as it return more data than usual. The overall process is similar to what we are used to seeing in Recommendation Engines. I wrote about such architectures a while ago here (second paragraph):

• Fine-Tuning of the embedding model - retrieval of data in some domains (e.g. medical data) works poorly with base embedding models. This is where you might need to fine-tune your own embedding model.

Let’s move on to some other advanced RAG techniques and architectures.

Contextual Retrieval.

The idea of Contextual Retrieval was suggested by the Anthropic team late last year. It aims to improve accuracy and relevance of data that is retrieved in Retrieval Augmented Generation based AI systems.

I love the intuitiveness and simplicity of Contextual Retrieval. And it does provide good results.

Here are the steps of implementation:

Preprocessing:

1. Split each of your documents into chunks via chosen chunking strategy.

2. For each chunk separately, add it to a prompt together with the whole document.

3. Include instructions to situate the chunk in the document and generate short context for it. Pass the prompt to a chosen LLM.

4. Combine the context that was generated in the previous step and the chunk that the context was generated for.

5. Pass the data through a TF-IDF embedder.

6. Pass the data through a LLM based embedding model.

7. Store the data generated in steps 5. and 6. in databases that support efficient search.

Retrieval:

8. Use user query for relevant context retrieval. ANN search for semantic and TF-IDF index for exact search.

9. Use Rank Fusion techniques to combine and deduplicate the retrieved results and produce top N elements.

10. Rerank the previous results and narrow down to top K elements.

11. Pass the result of step 10. to a LLM together with the user query to produce the final answer.

Some thoughts:

• Step 3. might sound extremely costly and it is, but with Prompt Caching, the costs can be significantly reduced.

• Prompt caching can be implemented in both proprietary and open source model cases (refer to the next paragraph).

The brief emergence of Cache Augmented Generation.

At the end of 2024 a white paper briefly shook the social media. An introduction of a technique that would change RAG forever (or would it?) - Cache Augmented Generation. We already know how regular RAG works, here is a brief description of CAG:

1. Pre-compute all of the external context into a KV Cache of the LLM. Cache it in memory. This only needs to be done once, the following steps can be run multiple times without recomputing the initial cache.

2. Pass the system prompt including user query and the system prompt with instructions on how cached context should be used by the LLM.

3. Return the generated answer to the user. After this, clear any generations from the cache and keep only the initially cached context. This makes the LLM ready for next generations.

Subscribe now

CAG promised a more accurate retrieval by storing all of the context in the KV cache instead of retrieving just a portion of data each time the answer needs to be computed. The reality?

• CAG does not solve the inaccuracies given extremely long context.

• CAG has many limitations when it comes to data security.

• Loading the entire internal knowledge base is close to impossible in large organisations.

• Cache becomes static and adding fresh data is problematic.

Actually, we were using a variant of CAG for some time already after introduction of prompt caching by most of the LLM providers. What we did can be described as a fusion of CAG and RAG and implemented in the following procedure:

Data Preprocessing:

1. We use only rarely changing data sources for Cache Augmented Generation. On top of the requirement of data changing rarely we should also think about which of the sources are often hit by relevant queries. Once we have this information, only then we pre-compute all of this selected data into a KV Cache of the LLM. Cache it in memory. This only needs to be done once, the following steps can be run multiple times without recomputing the initial cache.

2. For RAG, if necessary, precompute and store vector embeddings in a compatible database to be searched later in step 4. Sometimes simpler data types are enough for RAG, a regular database might suffice.

Query Path:

3. Compose a prompt including user query and the system prompt with instructions on how cached context and retrieved external context should be used by the LLM.

4. Embed a user query to be used for semantic search via vector DBs and query the context store to retrieve relevant data. If semantic search is not required, query other sources, like real time databases or web.

5. Enrich the final prompt with external context retrieved in step 4.

6. Return the final answer to the user.

Let’s move to the latest evolution - Agentic RAG.

Agentic RAG.

Agentic RAG has added two additional components that attempted to reduce inconsistencies when answering complex user queries.

• Data Source Routing.

• Reflection.

Let’s explore how it works end-to-end.

1. Analysis of the user query: we pass the original user query to a LLM based Agent for analysis. This is where:

   1. The original query can be rewritten, sometimes multiple times to create either a single or multiple queries to be passed down the pipeline.

   2. The agent decides if additional data sources are required to answer the query. This is where the first step of agency kicks in.

2. If additional data is required, the Retrieval step is triggered. This is where Data Source Routing happens, you could have one or multiple data sets available for the Agentic RAG system and the Agent is given agency to choose which ones should be tapped into to answer this specific query. Few examples:

   1. Real time user data. This is a pretty cool concept as we might have some real time information like current location available for the user.

   2. Internal documents that a user might be interested in.

   3. Data available on the web.

   4. …

3. Once the data is retrieved from potentially multiple data sources, we rerank it similarly like in regular RAG. This is also and important step as multiple different data sources utilising different storage technologies can be integrated into the RAG system here. Complexities of the retrieval process can be hidden behind tools given to the agent.

4. We try to compose the answer (or multiple answers or a set of actions) straight via LLM. It can happen in the first round or after the Reflection happens.

5. The answer gets analysed, summarised and evaluated for correctness and relevance:

   1. If the Agent decides that the answer is good enough, it gets returned to the user.

   2. If the Agent decides that the answer needs improvement, we try to rewrite the user query and repeat the generation loop. This is where the second difference of Regular vs. Agentic RAG is implemented.

Recently open-sourced MCP project by Anthropic can supercharge the development of Agentic RAG applications. Learn how in one of my previous Newsletter episodes:

Wrapping up.

We have reviewed the evolution of Modern RAG architectures. RAG is not dead and is not going anywhere anytime soon. I believe the architectures will continue evolving for some time to come and it is a good investment to learn them, and understand when to use what.

In general, the simpler the better as you will face new challenges while making the system more complex. Some of the emerging challenges include:

• Difficulties in evaluating the end-to-end system.

• Increase in end-to-end latency due to multiple LLM calls.

• Increase in the price of operation.

• …

This is it! Hope to see you in the next issue.

Subscribe now

Partner with SwirlAI

Share

MCP (Model Context Protocol) by Anthropic is all over the news this month. I have been following the project since it’s public release via announcement on 25th of November, 2024. In this article I will share my thoughts on why I believe you should start looking into MCP and why it is important for the future of AI Agents and Agentic systems. Here is the outline of what you will find in the article:

• Refresher on AI Agents and Agentic Systems.

• What is MCP?

• Splitting control responsibilities through MCP.

• Evolving AI Agent architecture with MCP.

• The future roadmap of MCP.

The Only Cloud-Native Kafka Implementation with Jepsen Validation.

Bufstream offers a robust streaming platform built on Protobuf, designed for high-throughput, low-latency data pipelines. Its architecture prioritizes consistency and fault tolerance, as demonstrated by rigorous Jepsen testing - plus it's up to 8x cheaper than self-managed Kafka. Explore the Jepsen Report for detailed analysis of its performance under various failure scenarios. Implement reliable, schema-driven streaming with Bufstream.

Check it out.

Refresher on AI Agents and Agentic Systems.

In it’s simplest high level definition, an AI agent is an application that uses LLM at the core as it’s reasoning engine to decide on the steps it needs to take to solve for users intent. It is usually explained via an image similar to the picture bellow and is composed of multiple building blocks:

• Planning - the capability to plan a sequence of actions that the application needs to perform in order to solve for the provided intent. There are many strategies to this, I have written an article about one of them - Reflection, where we built it from scratch without using any LLM Orchestration frameworks:

• Memory - short-term and long-term memory containing any information that the agent might need to reason about the actions it needs to take. This information is usually passed to LLM via a system prompt as part of the core. You can read more about different types of memories in one of my previous articles:

• Tools - any function that the application can call to enhance it’s reasoning capabilities. One should not be fooled by the simplicity of this definition as a tool can be literally anything:

  • Simple functions defined in code.

  • VectorDBs and other data stores containing context.

  • Regular Machine Learning model APIs.

  • Other Agents!

  • …

  Here is an article where I implement tool use patter from scratch:

Not all Agentic Systems are given full agency over execution in the environment. Anthropic has also described “Augmented LLMs” where application integrating LLMs as reasoning engines are only given control over tools and memory but not planning. The topology on how the interactions happen is defined in code rather than planned out by the LLM.

What is MCP?

MCP (Model Context Protocol) as defined by Anthropic is:

An open protocol that standardizes how applications provide context to LLMs.

To be more precise it attempts to standardise the protocol on how LLM based applications integrate with other environments.

In Agentic systems, AI Agents or chains of augmented LLMs the context can be provided in multiple ways:

• External data - this is part of long term memory.

• Tools - the capability of the system to interact with the environment.

• Dynamic Prompts - that can be injected as part of the system prompt.

• …

Bellow is the high level architecture of MCP.

1. MCP Host - Programs using LLMs at the core that want to access data through MCP.

2. MCP Client - Clients that maintain 1:1 connections with servers.

3. MCP Server - Lightweight programs that each expose specific capabilities through the standardized Model Context Protocol.

4. Local Data Sources - Your computer’s files, databases, and services that MCP servers can securely access.

5. Remote Data Sources - External systems available over the internet (e.g., through APIs) that MCP servers can connect to.

Why the need to standardise?

Current development flow of Agentic applications is chaotic:

• There are many Agent frameworks with slight differences. While it is encouraging to see the ecosystem flourish, these slight difference rarely add enough value but potentially significantly change the way you write code.

• Integrations with external data sources are usually implemented ad-hoc and using different protocols even within organisations. That is clearly true for different companies as well.

• Tools are defined in code repositories in slightly different ways. How you attach tools to augmented LLMs is different as well.

Eventually, the goal is to improve the velocity of how fast we can innovate with Agentic applications, how well we can secure them and how easy it is to bring relevant data to the context.

Subscribe now

Splitting control responsibilities through MCP.

MCP Servers expose three main elements that are purposely built in a way that helps implement specific control segregation.

• Prompts are designed to be User-Controlled.

  • Programmer of the server can expose specific prompts (suited for interaction with data exposed by the server) that can be injected into the application using LLMs and exposed to the user of the given application.

• Resources are designed to be Application-Controlled.

  • Resources are any kind of data (text or binary) that can be used by the application built to utilise LLMs. The programmer of the application (usually AI Engineer) is responsible of codifying how this information should be used by the application. Usually, there is not automation in that and LLM does not participate in this choice.

• Tools are designed to be Model-Controlled.

  • If we provide agency to our application of how it should interact with the environment we use tools to do that. MCP Server exposes an endpoint that can list all of the tools available with their descriptions and required arguments, application can pass this list to the LLM so that it can decide which tools are needed for the task at hand and how they should be invoked.

Evolving AI Agent architecture with MCP.

To explain how the architecture of Agentic application could evolve with MCP, let’s take an example of a very simple Agentic RAG.

Here are the steps the System could involve:

1. Analysis of the user query: we pass the original user query to a LLM based Agent for analysis. This is where:

   1. The original query can be rewritten, sometimes multiple times to create either a single or multiple queries to be passed down the pipeline.

   2. The agent decides if additional data sources are required to answer the query.

2. If additional data is required, the Retrieval step is triggered. In Agentic RAG case, we could have a single or multiple agents responsible for figuring out what data sources should be tapped into, few examples:

   1. Real time user data. This is a pretty cool concept as we might have some real time information like current location available for the user.

   2. Internal documents that a user might be interested in.

   3. Data available on the web.

   4. …

3. If there is no need for additional data, we try to compose the answer (or multiple answers or a set of actions) straight via an LLM.

4. The answer gets analyzed, summarized and evaluated for correctness and relevance:

   1. If the Agent decides that the answer is good enough, it gets returned to the user.

   2. If the Agent decides that the answer needs improvement, we try to rewrite the user query and repeat the generation loop.

Subscribe now

How does the architecture change if we introduce MCP into the picture?

We can introduce MCP servers to face all data sources relevant for the retrieval procedure. The MCP server handles retrieval logic through Tools since the LLM will be “choosing” which data sources will be relevant for the system. Here are some benefits to this approach:

• We decouple retrieval logic from the topology of the Agentic system.

• We can then evolve the retrieval component separately:

  • Introduce additional tools.

  • Introduce additional data sources.

  • Version, evolve and rollback existing tools and data sources.

• We can manage security and access to the data via the MCP server.

• A separate team can be independently working on the data it is responsible for.

Evolving of the architecture in larger enterprises.

As enterprises grow in size, different teams start owning specific data assets. E.g.:

• CRM Data.

• Financial Data.

• Real time Web clickstream data.

• ….

It is hard to effectively manage these disparate data sources and MCP brings a lot of value to the game.

• Each data domain can manage their own MCP Servers.

• All the MCP servers will use the same protocol.

• Because of the above, the integration efforts for LLM based applications will be significantly reduced.

• AI Engineers can continue to focus on the topology of the Agent.

[IMPORTANT]: This is one of the biggest advantages of using MCP in my opinion. It allows the decoupling of systems in large projects. Different teams can work on their own domains while not disturbing development of the main Agentic topology.

The future roadmap of MCP.

The public roadmap for the next 6 months strongly suggests the strengthening of Cloud Native aspect of the project. Improvements in:

• Authentication and authorisation.

• Service Discovery.

As well as expanding support for the future of Agentic Systems with focus on:

• Hierarchical Agent Systems

• Interactive Workflows to improve human in the loop interactions.

• Streaming of Results.

That’s it for today, let’s sum up.

While MCP is still relatively rough around the edges, the roadmap does look promising. The fact that the project has the backing by Anthropic also draws a bright future for it when it comes to adoption.

You should keep your eye on the project and potentially start adopting it!

More hands-on content utilising MCP in the future episodes.

Once again, thank you to the sponsors of this Newsletter - Bufstream.

Subscribe now

Partner with SwirlAI

Share

The big topic of last month was Deep Research Agents that every large player in the LLM industry is building and trying to monetise. The big catalyst for this was the emergence of DeepSeek R1 reasoning model and its open source nature.

In this episode of the Newsletter we are going to build such a Deep Research Agentic system from scratch. It will allow us to strengthen our fundamental knowledge of how these upcoming agentic systems actually function under the hood.

We are also going to run our system utilising one of the previously mentioned LLMs DeepSeek R1.

This brings us to our sponsors of today - SambaNova. The platform will allow us to run and test our system for free. Registering to the platform will give you $5 of credits with no credit card details needed, this will be enough to run the project.

SambaNova provides a selection of models in Llama, Qwen and DeepSeek families through their APIs for applications and a Playground for exploratory purposes. When it comes to DeepSeek models, they provide both distilled and non-distilled 671 Billion parameter versions. In the examples we will be running the 671B version but you can always switch to other versions.

Check it out

Thank you for helping keep SwirlAI Newsletters free for everyone!

What are Deep Research Agents?

To put it simply, these are systems that are capable of running an in depth research on a predefined topic. Usually, this would include at least the following steps:

• Planning of the research - this could mean a creation of an outline of the research report that would eventually become the output of the system.

• Splitting the above into manageable steps.

• Performing deep research on sections of the report. This means reasoning about the data needed to provide comprehensive analysis and utilising web search tools to support the analysis.

• Reflecting on data generated in different steps of the research and improving the results.

• Summarising the retrieved data and coming back with the final Research Report.

Today we will implement all of the above without using any LLM Orchestration framework.

You can find the code (Notebook included for interactive learning) in my “AI Engineers Handbook” GitHub repository:

GitHub Repository

Follow and Star the repository if you like the content!

For a step-by-step explanation, continue reading.

The System Topology.

Below picture represents what we are going to build, here are the steps that the outcome system will perform:

1. A user will provide a query or topic to be researched.

2. A LLM will create an outline of the final report that it will be aiming for. It will be instructed to produce not more than a certain number of paragraphs.

3. Each of the paragraph description will be fed into a research process separately to produce a comprehensive set of information to be used in report construction. Detailed description of the research process will be outlined in the next section.

4. All of the information will be fed into summarisation step that will construct the final report including conclusion.

5. The report will then be delivered to the user in MarkDown form.

The Research Step.

Let’s zoom into the research step defined in the previous paragraph:

1. Once we have the outline of each paragraph, it will be passed to a LLM to construct Web Search queries in an attempt to best enrich the information needed.

2. The LLM will output the search query and the reasoning behind it.

3. We will execute Web search against the query and retrieve top relevant results.

4. The results will be passed to the Reflection step where a LLM will reason about any missed nuances to try and come up with a search query that would enrich the initial results.

5. This process will be repeated for n times in an attempt to get the best set of information possible.

Subscribe now

Implementing the Agent.

Before we go into the implementation stage let’s do some technical hygiene. If you haven’t done that yet, go to SambaNova Cloud console and get your API key, we will use it to explain some important output characteristics of DeepSeek R1 model family.

You can register here.

Go to “APIs” tab, you will get prompted to login, do that. Don’t worry, adding your credit card details is optional, we will be able to run this project without the need to do that.

To run queries against the API we will use OpenAI client. If you don’t have it yet, simply run.

pip install openai

For the project we will use a non-distilled DeepSeek-R1 version with 671B parameters.

If you can’t access it yet by the time of reading the newsletter, join the waitlist and switch the API endpoint and model version to a smaller distilled version.

Makes sure that your SAMBANOVA_API_KEY is exported as an environment variable and run the following in your console or notebook:

import os
import openai

client = openai.OpenAI(
    api_key=os.environ.get("SAMBANOVA_API_KEY"),
    base_url="https://preview.snova.ai/v1",
)

response = client.chat.completions.create(
    model="DeepSeek-R1",
    messages=[{"role":"system","content":"You are a helpful assistant"},
              {"role":"user","content":"Tell me something interesting about human species"}],
    temperature=1
)

print(response.choices[0].message.content)

You should see something similar to:

<think>
Okay, so I'm trying to ... <REDACTED>
</think>

The human species is distinguished by the remarkable cognitive abilities of the brain, which underpin a array of unique traits. Our brain's advanced structure and function enable complex thought, language, and social organization. These capabilities have driven innovation, art, and the creation of intricate societies, setting humans apart in their ability to adapt, innovate, and create beyond any other species. This cognitive prowess is the cornerstone of human achievement and our profound impact on the world.

Reasoning tokens will always be included in the answer. While it is interesting to see the thinking process, what we will need in our systems is the answers only. This is where we can create a hygiene function to remove anything between the <think> tags.

def remove_reasoning_from_output(output):

    return output.split("</think>")[-1].strip()

Simple yet useful.

Great! We have now set up the SambaNova account and understood the output structure of DeepSeek R1 family models, let’s go implement the Deep Research Agent.

Part 1: Defining the State.

First, we will need to define the state of the entire system that will be continuously evolved while the Agent is running in the environment and used by different parts of the system selectively.

Let’s relate the state to the Stages of the Agentic system:

• Stage 1 will be the creation of the outline where report structure will be planned and its state evolved. We will start with an empty state but evolve it to something similar (reasoning is described in stage 2):

  {
      "report_title": "Report Title",
      "paragraphs": [
          {
              "title": "Paragraph Title",
              "content": "Paragraph Content",
              "research": <...>
          },
          {
              "title": "Paragraph Title",
              "content": "Paragraph Content",
              "research": <...>
          }
      ]
  }

  The state can be implemented in a clean way using python dataclasses. The above would look like:

  @dataclass
  class Paragraph:
      title: str = ""
      content: str = ""
      research: Research = field(default_factory=Research)
  
  @dataclass
  class State:
      report_title: str = ""
      paragraphs: List[Paragraph] = field(default_factory=list)

• Stage 2 is be where we will iterate on the state of each paragraph. We will change the “research” of each paragraph. We will use the following structure of the research state per paragraph:

  {
  "search_history": [{"url": "some url", "content": "some content"}],
  "latest_summary": "summary of the combined search history",
  "reflection_iteration": 1
  }

  

  search_history - We will store all of the searches we perform in a list, we will want both the url and the content so that we can deduplicate search results and refer to the links later when forming the final report.

  latest_summary - the summarised version of the paragraph given all of the search results. It will be used in reflection step to figure out if more search is needed and passed to the next step of summarisation and report creation.

  reflection_iteration - this is to track the current number of reflection iteration and force stop if the limit is reached.

  Again, we can implement research state via dataclasses:

  @dataclass
  class Search:
      url: str = ""
      content: str = ""
  
  @dataclass
  class Research:
      search_history: List[Search] = field(default_factory=list)
      latest_summary: str = ""
      reflection_iteration: int = 0

Subscribe now

Part 2: Creating the report outline.

Different versions of the models will have varying consistency with the answers they produce. I experimented with DeepSeek-R1 a bunch and the following prompt seemed to produce consistently well formatted outputs:

output_schema_report_structure = {
        "type": "array",
        "items": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "content": {"type": "string"}
            }
        }
    }

SYSTEM_PROMPT_REPORT_STRUCTURE = f"""
You are a Deep Research assistan. Given a query, plan a structure for a report and the paragraphs to be included.
Make sure that the ordering of paragraphs makes sense.
Once the outline is created, you will be given tools to search the web and reflect for each of the section separately.
Format the output in json with the following json schema definition:

<OUTPUT JSON SCHEMA>
{json.dumps(output_schema_report_structure, indent=2)}
</OUTPUT JSON SCHEMA>

Title and content properties will be used for deeper research.
Make sure that the output is a json object with an output json schema defined above.
Only return the json object, no explanation or additional text.
"""

Let’s run some sample query with the above system prompt:

response = client.chat.completions.create(
    model="DeepSeek-R1",
    messages=[{"role":"system","content":SYSTEM_PROMPT_REPORT_STRUCTURE},
              {"role":"user","content":"Tell me something interesting about human species"}],
    temperature=1
)

print(response.choices[0].message.content)

You will get something similar to:

```json
[
  {
    "title": "Introduction to Human Adaptability",
    "content": "Humans possess a unique capacity for adaptability, which has been crucial in their survival and dominance across various environments. This introduction sets the stage for exploring the different facets of human adaptability."
  },
  ...
  <REDACTED>
  ...
  {
    "title": "Conclusion: The Role of Adaptability in Human Survival",
    "content": "Adaptability has been a cornerstone of human survival and evolution, enabling us to face challenges and explore new frontiers, offering insights into future potential."
  }
]
```

These json tags surrounding the output are not helpful as we will need to transform the output into Python dictionary. Very simple function to remove first and last line of the output:

def clean_json_tags(text):

    return text.replace("```json\n", "").replace("\n```", "")

Here is the properly cleaned output:

json.loads(clean_json_tags(remove_reasoning_from_output(response.choices[0].message.content)))

We can now use above as an input to our global state directly.

STATE = State()

report_structure = json.loads(clean_json_tags(remove_reasoning_from_output(response.choices[0].message.content)))

for paragraph in report_structure:
    STATE.paragraphs.append(Paragraph(title=paragraph["title"], content=paragraph["content"]))

Part 3: The Web Search tool.

We will be using Tavily to perform web search. You can get your token here.

The tool for it is very simple:

def tavily_search(query, include_raw_content=True, max_results=5):

    tavily_client = TavilyClient(api_key=os.getenv("TAVILY_API_KEY"))

    return tavily_client.search(query,
                                include_raw_content=include_raw_content,
                                max_results=max_results)

Each of the function call will return up to max_results of the web search results and for each search result will return:

• Title of the search result.

• URL of the search result.

• Summary of the content.

• Full content of the page if possible. We want this for best results.

Once we get the results from the Web Search tool, we can add all of it to the global state directly without need for calling any LLM but we will need to make sure that we are updating the proper element in the list of paragraphs.

Basically, given the structure we defined:

{
    "report_title": "Report Title",
    "paragraphs": [
        {
            "title": "Paragraph Title",
            "content": "Paragraph Content",
            "research": <...>
        },
        {
            "title": "Paragraph Title",
            "content": "Paragraph Content",
            "research": <...>
        }
    ]
}

And given that we are currently researching the ith paragraph, we will need to be updating the field .paragraphs[i].research.

Remembering the structure of the research field:

{
"search_history": [{"url": "some url", "content": "some content"}],
"latest_summary": "summary of the combined search history",
"reflection_iteration": 1
}

Here is a handy function that will update the state correctly provided Tavily search results, index of the paragraph and the state object.

def update_state_with_search_results(search_results, idx_paragraph, state):
    
    for search_result in search_results["results"]:
        search = Search(url=search_result["url"], content=search_result["raw_content"])
        state.paragraphs[idx_paragraph].research.search_history.append(search)

    return state

We will be appending to the search history. latest_summary and reflection_iteration fields will need some work by LLM and will be discussed in Part 5: Reflecting.

Part 4: Planning the Search.

To plan the first instance of a search I found the following prompt to produce consistently good results:

input_schema_first_search = {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "content": {"type": "string"}
            }
        }

output_schema_first_search = {
            "type": "object",
            "properties": {
                "search_query": {"type": "string"},
                "reasoning": {"type": "string"}
            }
        }

SYSTEM_PROMPT_FIRST_SEARCH = f"""
You are a Deep Research assistant. You will be given a paragraph in a report, it's title and expected content in the following json schema definition:

<INPUT JSON SCHEMA>
{json.dumps(input_schema_first_search, indent=2)}
</INPUT JSON SCHEMA>

You can use a web search tool that takes a 'search_query' as parameter.
Your job is to reflect on the topic and provide the most optimal web search query to enrich your current knowledge.
Format the output in json with the following json schema definition:

<OUTPUT JSON SCHEMA>
{json.dumps(output_schema_first_search, indent=2)}
</OUTPUT JSON SCHEMA>

Make sure that the output is a json object with an output json schema defined above.
Only return the json object, no explanation or additional text.
"""

We ask for reasoning in the output schema just to force more thought around the query. While it is most likely an overkill for a reasoning model it might be a good idea with a regular LLM. While we are using DeepSeek R1 for this, we don’t necessarily need to. Reasoning models are specifically handy in the first step of the Deep Research Agent where planning of the report structure is required.

Given the fact that we now have a list of paragraphs planned with their content and descriptions, we can feed output of part 3 directly to the prompt, this is how it would look like:

response = client.chat.completions.create(
    model="DeepSeek-R1",
    messages=[{"role":"system","content":SYSTEM_PROMPT_FIRST_SEARCH},                {"role":"user","content":json.dumps(STATE.paragraphs[0]])}],
    temperature=1
)

print(response.choices[0].message.content)

STATE.paragraphs[0] points to the first paragraph state where research field is still empty.

I got the following for my first search plan:

{"search_query": "Homo sapiens characteristics basic biological traits cognitive abilities behavioral traits"}

I can directly input the query into my search tool:

tavily_search("Homo sapiens characteristics basic biological traits cognitive abilities behavioral traits")

Subscribe now

Part 5: The first summary.

The first summary is different from the upcoming Reflection step as there is nothing yet to reflect on and this step will be producing exactly that. The following prompt works relatively well:

input_schema_first_summary = {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "content": {"type": "string"},
                "search_query": {"type": "string"},
                "search_results": {
                    "type": "array",
                    "items": {"type": "string"}
                }
            }
        }

output_schema_first_summary = {
            "type": "object",
            "properties": {
                "paragraph_latest_state": {"type": "string"}
            }
        }

SYSTEM_PROMPT_FIRST_SUMMARY = f"""
You are a Deep Research assistan. You will be given a search query, search results and the paragraph a report that you are researching following json schema definition:

<INPUT JSON SCHEMA>
{json.dumps(input_schema_first_summary, indent=2)}
</INPUT JSON SCHEMA>

Your job is to write the paragraph as a researcher using the search results to align with the paragraph topic and structure it properly to be included in the report.
Format the output in json with the following json schema definition:

<OUTPUT JSON SCHEMA>
{json.dumps(output_schema_first_summary, indent=2)}
</OUTPUT JSON SCHEMA>

Make sure that the output is a json object with an output json schema defined above.
Only return the json object, no explanation or additional text.
"""

We now need to provide data to the LLM in the following format:

 {
    "title": "Title",
    "content": "Content",
    "search_query": "Search Query",
    "search_results": []
}

We can construct the json from data we already have.

Given the response from:

search_results = tavily_search("Homo sapiens characteristics basic biological traits cognitive abilities behavioral traits")

The json would look like:

input = {
    "title": "Introduction to Human Adaptability",
    "content": "Humans possess a unique capacity for adaptability, which has been crucial in their survival and dominance across various environments. This introduction sets the stage for exploring the different facets of human adaptability.",
    "search_query": "Homo sapiens characteristics basic biological traits cognitive abilities behavioral traits",
    "search_results": [result["raw_content"][0:20000] for result in search_results["results"] if result["raw_content"]]
}

We can then run:

response = client.chat.completions.create(
    model="DeepSeek-R1",
    messages=[{"role":"system","content": SYSTEM_PROMPT_FIRST_SUMMARY},
              {"role":"user","content":json.dumps(input)}],
    temperature=1
)

print(remove_reasoning_from_output(response.choices[0].message.content))

You will get something similar to:

{
  "paragraph_latest_state": "Homo sapiens, the species to which modern humans belong, represents a unique and fascinating chapter in the evolutionary narrative of life on Earth. As the only living species within the Homo genus, Homo sapiens are distinguished by a combination of biological, cognitive, and behavioral traits that set us apart from other primates and extinct human relatives. Our biological characteristics include a large and structurally advanced brain, with a neocortex that has expanded significantly compared to our evolutionary ancestors. This anatomical development has enabled exceptional cognitive abilities, such as complex problem-solving, abstract thought, and the capacity for language and symbolic communication. Behaviorally, Homo sapiens exhibit sophisticated social structures, cultural practices, and technological innovations, which have been critical in shaping our ability to adapt to diverse environments and thrive as a species. These traits collectively underscore the intricate interplay between biology and behavior that defines the human condition."
}

This is what we will be updating STATE.paragraphs[0].research.latest_summary field with. We will also reflect on the continuously updated latest state of the paragraph summary as we move forward in Part 6.

Part 6: Reflecting.

We now have the latest state of the report paragraph content and will use that to improve the content by prompting our LLM to reflect on the text and look for any angles it might have missed while drafting the piece.

Here is the prompt that does great job:

input_schema_reflection = {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "content": {"type": "string"},
                "paragraph_latest_state": {"type": "string"}
            }
        }

output_schema_reflection = {
            "type": "object",
            "properties": {
                "search_query": {"type": "string"},
                "reasoning": {"type": "string"}
            }
        }

SYSTEM_PROMPT_REFLECTION = f"""
You are a Deep Research assistant. You are responsible for constructing comprehensive paragraphs for a research report. You will be provided paragraph title and planned content summary, also the latest state of the paragraph that you have already created all in the following json schema definition:

<INPUT JSON SCHEMA>
{json.dumps(input_schema_reflection, indent=2)}
</INPUT JSON SCHEMA>

You can use a web search tool that takes a 'search_query' as a parameter.
Your job is to reflect on the current state of the paragraph text and think if you haven't missed some critical aspect of the topic and provide the most optimal web search query to enrich the latest state.
Format the output in json with the following json schema definition:

<OUTPUT JSON SCHEMA>
{json.dumps(output_schema_reflection, indent=2)}
</OUTPUT JSON SCHEMA>

Make sure that the output is a json object with an output json schema defined above.
Only return the json object, no explanation or additional text.
"""

For the run we are currently implementing the input would look like this:

input = {"paragraph_latest_state": "Homo sapiens, the species to which modern humans belong, represents a unique and fascinating chapter in the evolutionary narrative of life on Earth. As the only living species within the Homo genus, Homo sapiens are distinguished by a combination of biological, cognitive, and behavioral traits that set us apart from other primates and extinct human relatives. Our biological characteristics include a large and structurally advanced brain, with a neocortex that has expanded significantly compared to our evolutionary ancestors. This anatomical development has enabled exceptional cognitive abilities, such as complex problem-solving, abstract thought, and the capacity for language and symbolic communication. Behaviorally, Homo sapiens exhibit sophisticated social structures, cultural practices, and technological innovations, which have been critical in shaping our ability to adapt to diverse environments and thrive as a species. These traits collectively underscore the intricate interplay between biology and behavior that defines the human condition.",
            "title": "Introduction",
            "content": "The human species, Homo sapiens, is one of the most unique and fascinating species on Earth. This section will introduce the basic characteristics of humans and set the stage for exploring interesting aspects of the species."}

As before, let’s run the following:

response = client.chat.completions.create(
    model="DeepSeek-R1",
    messages=[{"role":"system","content": SYSTEM_PROMPT_REFLECTION},
              {"role":"user","content":json.dumps(input)}],
    temperature=1
)

print(remove_reasoning_from_output(response.choices[0].message.content))

The output:

{
  "search_query": "Recent research on Homo sapiens evolution, interaction with other human species, and factors contributing to their success",
  "reasoning": "The current paragraph provides a good overview of Homo sapiens' characteristics but lacks depth on evolutionary history and interactions with other species. Including recent research on these topics will enhance the paragraph's comprehensiveness and provide up-to-date information."
}

Now we run the query, append new results to the paragraph state and combine new results with the latest paragraph state.

Part 7: Enriching latest paragraph state with reflection search results.

After running the search query of reflection step:

search_results = tavily_search("Recent research on Homo sapiens evolution, interaction with other human species, and factors contributing to their success")

We can update the search state of the paragraph with:

update_state_with_search_results(search_results, idx_paragraph, state)

We now run steps 6. and 7. in a loop for a specified number of reflection steps.

Part 8: Summarising and producing the report.

We repeat the steps from Parts 4 - 7 for each paragraph that was planned in Part 2. Once we have all of the final states in the paragraphs ready, we can then stitch the whole thing together. We will do that with an LLM and produce a nicely formatted MarkDown document. Here is the prompt:

input_schema_report_formatting = {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "title": {"type": "string"},
                    "paragraph_latest_state": {"type": "string"}
            }
        }
    }

SYSTEM_PROMPT_REPORT_FORMATTING = f"""
You are a Deep Research assistant. You have already performed the research and constructed final versions of all paragraphs in the report.
You will get the data in the following json format:

<INPUT JSON SCHEMA>
{json.dumps(input_schema_report_formatting, indent=2)}
</INPUT JSON SCHEMA>

Your job is to format the Report nicely and return it in MarkDown.
If Conclusion paragraph is not present, add it to the end of the report from the latest state of the other paragraphs.
"""

Run:

report_data = [{"title": paragraph.title, "paragraph_latest_state": paragraph.research.latest_summary} for paragraph in STATE.paragraphs]

response = client.chat.completions.create(
    model="DeepSeek-R1",
    messages=[{"role":"system","content": SYSTEM_PROMPT_REPORT_FORMATTING},
              {"role":"user","content":json.dumps(report_data)}],
    temperature=1
)

print(remove_reasoning_from_output(response.choices[0].message.content))

And that’s it, you have yourself a Deep Researched report on a topic you provided.

Conclusion.

Congratulations! You have successfully implemented a Deep Research Agent from scratch.

If you want to take a look into a more properly implemented code, you can find it in my GitHub repository:

GitHub Repository

Also, there are many nuances to take into account that could make the system more stable:

• It is not easy to make the system produce consistently well formatted json outputs as reasoning models are known to be not the best at structured outputs.

• Knowing the above, it might make sense to use different models for different tasks in the system topology, we really need a reasoning model for the first step mostly.

• There are many improvements that could be implemented in how we search the web and how we rank the retrieved results.

• Number of Reflection steps could be configured to be dynamic where the LLM could choose if more is needed.

• We could return links that were used when searching the web and provide references in the report for each paragraph.

• …

If you want to follow SambaNova work, they have recently released some work on Deep Research agents as well. You can find more information here.

Subscribe now

Partner with SwirlAI

Share

One of the most fulfilling aspects of being part of a community is being able to partner with various organisations with the goal of giving away exciting items.

This time I am partnering with NVIDIA to give away a NVIDIA RTX 4080 SUPER GPU to one of the SwirlAI community members.

In order to participate in the giveaway, you need to:

• Register to the GTC 2025 via the following link:

  Register to GTC 2025

• DM me to get further instructions.

The conference will be running throughout March 17th - 21st. It will be held in a hybrid format: San Jose, CA and virtually.

Virtual attendance is FREE - so there’s no reason to miss out!

The lineup and the list of topics is really extensive.

Personally I am most excited about Robotics, IoT and AI Agents on the edge. These are the four sessions that I am adding to my attendance list:

• An Introduction to NVIDIA Isaac GR00T for Humanoid Developers - excited to learn more about the platform for developing humanoid robots that NVIDIA is building.

• A New Era of Generalist Robotics: The Rise of Humanoids - excited to get the latest insights into how foundation models are used in humanoid robotics and what challenges are still to be overcome.

• Building Edge and Robotics Applications with Generative AI on NVIDIA Jetson - Sharpen my knowledge on latest developments in Agents on the edge and get inspiration for my own projects.

• AI Meets Robotics: European Startup Showcase - looking forward to hear more about where we stand with Robotics advancements in Europe.

Be sure to subscribe to the Newsletter If you don’t want to miss any future giveaways or any other perks of being part of the community!

Subscribe now

Refresher: Simple explanation of Memory in Agentic Systems.

In general, the memory for an agent is something that we provide via context in the prompt passed to LLM that helps the agent to better plan and react given past interactions or data not immediately available.

It is useful to group the memory into four types:

1. Episodic - This type of memory contains past interactions and actions performed by the agent. After an action is taken, the application controlling the agent would store the action in some kind of persistent storage so that it can be retrieved later if needed. A good example would be using a vector Database to store semantic meaning of the interactions.

2. Semantic - Any external information that is available to the agent and any knowledge the agent should have about itself. You can think of this as a context similar to one used in RAG applications. It can be internal knowledge only available to the agent or a grounding context to isolate part of the internet scale data for more accurate answers.

3. Procedural - This is systemic information like the structure of the System Prompt, available tools, guardrails etc. It will usually be stored in Git, Prompt and Tool Registries.

4. Occasionally, the agent application would pull information from long-term memory and store it locally if it is needed for the task at hand.

5. All of the information pulled together from the long-term or stored in local memory is called short-term or working memory. Compiling all of it into a prompt will produce the prompt to be passed to the LLM and it will provide further actions to be taken by the system.

We usually label 1. - 3. as Long-Term memory and 5. as Short-Term memory.

And that is it! The rest is all about how you architect the topology of your Agentic Systems.

Stay safe and hope to see you in the next Newsletter episode. We will continue building Agentic Systems from scratch!

Subscribe now

Share

Partner with SwirlAI

AI and AI Agents are the topic of 2025. However, we often overlook the crucial element that drives these applications - Data.

Data Pipelines in Machine Learning Systems can become complex and for a good reason.

It is critical to ensure Data Quality and Integrity upstream of ML Training and Inference Pipelines, trying to do that in the downstream systems will cause unavoidable failure when working at scale.

Example architecture for a production grade end-to-end data flow:

1. Schema changes are implemented in version control, once approved - they are pushed to the Applications generating the Data, Databases holding the Data and a central Data Contract Registry.

2. Events emitted directly by the Application Services or via CDC to Kafka topics.

3. A Flink Application(s) consumes Data from Raw Data streams and validates it against schemas in the Contract Registry.

4. Data that does not meet the contract is pushed to Dead Letter Topic.

5. Data that meets the contract is pushed to Validated Data Topic.

6. Data from the Validated Data Topic is pushed to object storage for additional Validation.

7. On a schedule Data in the Object Storage is validated against additional SLAs in Data Contracts and is pushed to the Data Warehouse to be Transformed and Modeled for Analytical purposes.

8. Modeled and Curated data is pushed to the Feature Store System for further Feature Engineering.

9. High Quality Data is used in Machine Learning Training Pipelines.

10. The same Data is used for Feature Serving in Inference.

Subscribe now

In this hands-on project Newsletter episode I am putting on my Data Engineer hat and we will implement a simplified real time data ingestion pipeline that represents a pipeline that you would see in production environments in real world:

1. Data producers sending data to the collector service in real time.

2. Real time data ingestion via a collector service.

3. Batching and writing this data in object storage.

4. Raw data staged for further processing.

5. Batch Spark job that cleans, deduplicates and potentially enriches raw data.

6. Scheduled Airflow task that runs the ETL every N minutes.

I will explain why each part of the infrastructure is a good idea as we go through the project.

You can find all of the code to support you in this GitHub repository.

If you do run into any problems while following the project, let me know in the comment section or drop me a PM, we will solve it together.

This newsletter episode was made possible by Nebius - AI Cloud provider that has recently graduated to become one of the 4 AI Cloud giants. Congratulations!

Consider registering if you want to follow the tutorial, all of the pieces will deployed on Nebius platform. We will also use a Managed Service for Apache Spark service provided by Nebius.

Managed Service for Apache Spark is in a preview stage right now which means that you can run the clusters for free!

Register

Lets go build.

We will start by implementing the Real Time Ingestion pipeline:

• The Collector.

  • Write a Python application that uses FastAPI Framework to expose a REST API endpoint.

  • Deploy the application on Kubernetes.

  • Horizontally scale the application for High Availability

• Data Producer.

  • Write a Python application to download data from the internet.

  • Run the application on Kubernetes.

  • Send the downloaded data to a previously deployed REST API endpoint.

Prerequisites.

This tutorial assumes that you know the basics of Python and have skimmed through my latest Guide to Kubernetes which you can find here.

Defining the Collector architecture.

Share

The following is the first version of The Collector that we will be implementing:

• Collector will be a Highly Available REST API application implemented with FastAPI Framework.

• Exposed REST API endpoints will be able to accept events that meet specific schema requirements - we will check for top level field existence. These top level fields will allow us to validate Data Contracts later on in the Downstream Processes.

• Fields that we will be expecting during validation:

  • event_type - this will be needed to identify the source that generated the event.

  • schema_version - this field allows us to identify which version of the schema of the given data source is represented in the event. This data will be used for schema evolution.

  • payload - actual data of interest that is relevant for analytics purposes.

• Collector application will add additional fields on top of the already existing ones:

  • collector_timestamp: timestamp when the collector has processed the event for downstream tasks.

  • root_id - a unique identifier for the event in the system (uuid).

  • collector_id - identifier of the collector process used for debugging purposes when there is the need to know where specifically the event was processed before emitting it to the downstream systems..

• In this tutorial we will not use distributed messaging systems like Kafka and rather write batched events directly to Object Storage from the Collector.

• We will face the Collector applications with a Load Balancer for High Availability purposes.

• We will emulate the Data Producers by downloading batch datasets from the internet and emitting records of these datasets to the Collector applications one by one every n milliseconds/seconds.

  • We will use a popular TLC Trip Record Dataset available for download here.

  • To emulate distributed systems and multiple data sources we will download data for November of 2024 of both Yellow and Green taxi datasets and emit them in parallel using separate applications.

• Collectors are the most sensitive part of your pipeline - you want to do as little processing and data manipulation here as possible. Usually, any errors here would result in permanent Data Loss (naturally less processing on the Data Producer side is also desirable as it would suffer from the same constraints). Once the data is already in the Downstream System you can always retry additional computation.

• Hence, we decouple Collectors from any additional Schema Validation or Enrichment of the Payload and move that down the stream.

How does it all look from Kubernetes' perspective?

As mentioned earlier, our goal in this hands-on tutorial is to deploy everything on Nebius AI Cloud. The first set of applications will be deployed on Kubernetes

Collectors and Producers very naturally map to the Kubernetes resources:

• We will be deploying all applications in a single namespace called swirlai.

• Collector applications will be in a form of deployment called collector.

• We can scale replicas of the collector to as many as we will need.

• We will mount a service named collector on top of the pods created by the collector deployment and map to the ports 80 of the containers managed by pods. The service itself will expose port 80.

• Producer applications will be deployed in a form of separate pods per dataset. We will go for 2 pods named producer-1 and producer-2 that will download datasets for November of Grren Taxi and Yellow Taxi data respectively.

• Since all applications will be deployed in the same namespace, we will be able to access service directly by its name collector.

• We will be exposing the collection endpoint under path /api/v1/collect and accept post requests to it.

• Eventually, we will be sending events to collector applications via post requests to the http://collector/api/v1/collect endpoint.

Share

Implementing the Collector Application.

Reminder - you can find the code used in this Newsletter episode in one of my GitHub repositories here.

Before we can deploy anything, we will need a Kubernetes cluster, Object Storage and service account configured with Object Storage access.

Creating Kubernetes Cluster.

If you haven’t yet, create Nebius account here.

You can easily create Kubernetes clusters via Nebius UI.

Before you can start the deployment, you will need to install Nebius CLI tool. If you already have your account set up it should be straightforward.

Assuming that you are running on MacOS, run:

brew install jq
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo chown root: /usr/local/bin/kubectl
curl -sSL https://storage.ai.nebius.cloud/nebius/install.sh | bash

This will set you up with kubectl tool for communication with Kubernetes cluster and nebius CLI tool for authentication with. After this run:

nebius profile create

It will prompt you for:

• Name - enter any.

• Api endpoint - leave the default api.eu.nebius.cloud.

• Authorisation type - choose federation.

After the above you will be redirected to the browser window where you will be authenticated. You now have your Nebius CLI tool set up.

If anything fails at this point, refer to official Nebius documentation here: Link.

For simplicity reasons, we will perform the deployment via the Nebius Cloud UI, here is how you can do it. It is easy to figure it out, but let’s run step-by-step so that there are no unanswered questions.

• Once you log in to the console, click on the “Managed Kubernetes” tab on the left and click “+ Create cluster” top right

• Once in the cluster creation section:

  • Provide the cluster name.

  • We switch off the Control plane high availability off as there is no need for it in a demo project, be sure to always have it turned on for production use cases.

  • Let’s have the Public endpoint on as it will make configuration of kubectl easier for this example.

  • Press “Create cluster” once configuration is complete.

• You will see a new cluster being provisioned in the “Managed Kubernetes” overview tab. Click on it.

• The above steps have provisioned a control plane for K8s, now we need to add some worker nodes. Click on the “Node groups” tab and click “+ Create new group”.

• Configure the Node Group for our project:

  • Give it a Name.

  • Disable the Public IPv4 address as we will be using LoadBalancer services to expose our apps, no need for the node itself to have a public address.

  • Keep Number of nodes at 3 as we will run cheap CPU based ones for multipple applications.

  • In the Preset field be sure to select 2 CPUs - 8 GiB RAM, we will not need more.

• That’s it, we are ready to connect to our new Kubernetes cluster. Click on the “How to connect” button, copy the third command and run it. We are good to go to use kubectl and communicate with the cluster. Let’s try.

Run the following in the terminal

kubectl get pods

you should see empty list.

Creating Buckets.

For this tutorial, you will need 4 buckets to store the data.

The creation of the buckets is trivial, you simply need to name them, the defaults for other configurations will do just fine. Feel free to replace sai- with your prefix just have in mind that you need to replace the names in two scripts down the line:

• Collector application via deployment configuration.

   - name: RAW_LANDING_BUCKET
     value: sai-raw

• Airflow DAG script.

  RAW_LANDING_BUCKET = 'sai-raw'
  RAW_PROCESSING_BUCKET = 'sai-processing'
  PROCESSED_BUCKET = 'sai-processed'
  ARCHIVE_BUCKET = 'sai-archive'

Creating Service Account.

I will not be able to explain it better than it is done here :) Just follow the instructions step by step.

[IMPORTANT]: We will need the following values from this procedure later, so be sure to save them:

• NB_ACCESS_KEY_AWS_ID.

• NB_SECRET_ACCESS_KEY.

• AWS_DEFAULT_REGION: eu-north1.

• AWS_ENDPOINT_URL: https://storage.eu-north1.nebius.cloud:443

I am assuming that you will be running the tutorial in eu-north1 region.

Subscribe now

The Code.

Below is the screenshot of the structure of the Python code for the Collector application. Let’s dig deeper:

1. FastAPI Application entrypoint.

from fastapi import FastAPI
from schema.schema import TopLevelSchema
from helpers.processor import Processor, Buffer

import config.logger as logger
import config.config as config
import os


logger = logger.setup_logger()
conf = getattr(config, f'{os.environ["APP_ENV"].title()}Config')

app = FastAPI(docs_url=f"/api/{conf.V_API}/docs",
              redoc_url=f"/api/{conf.V_API}/redoc",
              openapi_url=f"/api/{conf.V_API}/openapi.json")

buffer = Buffer()

@app.get(f"/api/{conf.V_API}")
async def root():

    return {"message": f"Welcome to the collector {os.environ['HOSTNAME']}!"}


@app.post(f"/api/{conf.V_API}{conf.COLLECTOR_PATH}")
async def write_data(data: TopLevelSchema):

    try:
        processed_data = Processor(data.dict()).processed_event
        buffer.add(processed_data)
        logger.info(f"Adding event to buffer: {data.dict()}")
    except:
        logger.error(f"Failed to process event: {data}")

    return {"message": f"Event added to buffer"}

Take note that we are versioning the API endpoint as per

/api/{conf.V_API}

It is an important good practice in REST API development as you might want to evolve the contract of it but keep the old version available for compatibility reasons. We store the version in configuration object.

As per

from helpers.processor import Processor

we are implementing all of the processing logic in a separate script located in helpers folder.

2. Any additional helper functions or Classes.

   2.1. We implement processing logic that needs to be applied on the incoming events via Processor class. Buffer class contains logic of buffering the events and flushing them to the object storage after specific thresholds. We go simple for this example, when the buffer reaches 50 events, they get automatically flushed to object storage.

from datetime import datetime
import pytz
import uuid
import os
import json
import boto3


class Processor:

    def __init__(self, event: dict):

        self.event = event

    @staticmethod
    def process_event(event: dict) -> dict:

        def _add_timestamp(record: dict) -> dict:

            record['collector_tstamp'] = datetime.now(tz=pytz.UTC).strftime("%Y-%m-%d %H:%M:%S %z")

            return record

        def _add_id(record: dict) -> dict:

            record['root_id'] = str(uuid.uuid4())

            return record

        def _add_collector_id(record: dict) -> dict:

            record['collector_id'] = os.environ['HOSTNAME']

            return record

        processed_event = _add_timestamp(event)
        processed_event = _add_id(processed_event)
        processed_event = _add_collector_id(processed_event)

        return processed_event

    @property
    def processed_event(self) -> dict:

        return self.process_event(self.event)


class Buffer:

    BUFFER_SIZE = 50
    BUCKET_NAME = os.environ["RAW_LANDING_BUCKET"]

    def __init__(self):

        self.buffer = []
        self.s3 = boto3.client('s3')

    def add(self, event: dict):

        self.buffer.append(event)

        if len(self.buffer) >= self.BUFFER_SIZE:
            self.flush()

    def clear(self):

        self.buffer = []

    def flush(self):

        self.s3.put_object(
            Body=json.dumps(self.buffer),
            Bucket=self.BUCKET_NAME,
            Key=f'{datetime.now(tz=pytz.UTC).strftime("%Y-%m-%d-%H-%M-%S-%f")}.json'
        )

        self.clear()

As discussed while defining the architecture of the applications, the only processing is adding of 3 additional fields: collector_timestamp, root_id and collector_id.

3. Schema definition.

   3.1. We want to decouple schema definition into a separate script from the main entrypoint of the FastAPI application for cleaner structure.

from pydantic import BaseModel


class TopLevelSchema(BaseModel):
    event_type: str
    schema_version: str
    payload: dict

It is really minimal pydantic data model. This is also where one of the main FastAPI advantages comes in. Any field defined in TopLevelSchema including the type is required and validated against any incoming event to the endpoint.

Referring back to app.py:

@app.post(f"/api/{conf.V_API}{conf.COLLECTOR_PATH}")
async def write_data(data: TopLevelSchema):

we are expecting data with schema TopLevelSchema for any data posted to /api/{conf.V_API}{conf.COLLECTOR_PATH} path (this will equal to /api/v1/collect in our example).

4. Additional application level configuration:

   4.1. General application configuration.

class BaseConfig:

    COLLECTOR_PATH = "/collect"
    V_API = "v1"
    TESTING = False
    DEBUG = False


class DevConfig(BaseConfig):

    DEBUG = True


class ProdConfig(BaseConfig):

    ...


class TestConfig(BaseConfig):

    ...

This is also where we define a collection endpoint and current api version.

4.2. Logging configuration.

import logging
import os

LOG_FORMAT = f'[%(asctime)s]: {os.getpid()} %(levelname)s %(message)s'
DATE_FORMAT = '%Y-%m-%d %H:%M:%S'
HANDLERS = [logging.StreamHandler()]
LOG_LEVEL = logging.DEBUG


def setup_logger() -> logging.Logger:

    logging.basicConfig(level=LOG_LEVEL,
                        format=LOG_FORMAT,
                        datefmt=DATE_FORMAT,
                        handlers=HANDLERS)

    return logging.getLogger()

This holds some global logging configuration that you can also parametrise according to your environment.

5. Python library requirements.

fastapi~=0.95.1
pydantic~=1.10.7
uvicorn
pytz
boto3

As you can see, for this example the requirements are extremely minimal as we are not doing any fancy processing inside of the collector and rather treating it as a proxy.

6. Dockerfile for the application container.

FROM python:3.9

WORKDIR /code/src

COPY ./requirements.txt /code/requirements.txt

RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

COPY ./src /code/src

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "80"]

The Dockerfile is also very minimal for now.

Important:

• This is not a production ready Dockerfile as there are many security considerations you should do (we will look into how to build a secure docker container in one of the future Newsletter episodes). For simplicity sake we will not implement them today.

In order to build the container that contains your collector application, go to the folder where Dockerfile resides and run, we will be pushing the container to your docker hub registry, so you will need to change some values in the bellow command:

docker build . --platform linux/amd64 -t <docker_hub_account>/<repository>:<tag>

Once the container is built, push it to your docker registry so that it can be pulled to the K8s cluster:

docker push <docker_hub_account>/<repository>:<tag>

If you want to just follow the tutorial, feel free to use the image I’ve built, the latest one can be found under:

aurimasg/collector:0.2.0

7. Kubernetes resource manifests that will be needed to deploy Collector application.

   7.1. Namespace.

apiVersion: v1
kind: Namespace
metadata:
   name: swirlai

You can create the namespace by running:

kubectl apply -f namespace.yaml

7.2. Secrets.

Remember when we created

• NB_ACCESS_KEY_AWS_ID.

• NB_SECRET_ACCESS_KEY.

these are equivalent to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY respectively. You will need to base64 encode them and add the values to the secrets file.

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: swirlai
type: Opaque
data:
  AWS_ACCESS_KEY_ID: <base64 encoded value of AWS_ACCESS_KEY_ID>
  AWS_SECRET_ACCESS_KEY: <base64 encoded value of AWS_SECRET_ACCESS_KEY>

7.3. Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: collector
  labels:
    app: collector
  namespace: swirlai
spec:
  replicas: 2
  selector:
    matchLabels:
      app: collector
  template:
    metadata:
      labels:
        app: collector
    spec:
      containers:
      - name: collector
        image: aurimasg/collector:0.2.0
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        env:
          - name: APP_ENV
            value: Dev
          - name: RAW_LANDING_BUCKET
            value: sai-raw
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: AWS_SECRET_ACCESS_KEY
          - name: AWS_DEFAULT_REGION
            value: eu-north1
          - name: AWS_ENDPOINT_URL
            value: https://storage.eu-north1.nebius.cloud:443

You can create the deployment by running:

kubectl apply -f deployment.yaml

7.4. Service.

apiVersion: v1
kind: Service
metadata:
  name: collector
  namespace: swirlai
spec:
  selector:
    app: collector
  ports:
  - name: collector
    protocol: TCP
    port: 80
    targetPort: 80

You can create the service by running:

kubectl apply -f service.yaml

That’s it for collector, it should be successfully running in the swirlai namespace:

kubectl get pods -n swirlai

Subscribe now

Implementing the Producer Applications.

We will not spend much time on Producer application as it is really simple, it just downloads data from the internet and pushes it to the collector service.

Below is the screenshot of the structure of the Python project for the Producer application.

1. The entrypoint of the application.

import requests
import pandas as pd
import time
import config.logger as logger
import argparse
import os
import json


SOURCE_URL = os.environ['DATA_URL']
EVENT_TYPE = os.environ['EVENT_TYPE']
SCHEMA_VERSION = os.environ['SCHEMA_VERSION']
COLLECTOR_URL = os.environ['COLLECTOR_URL']

logger = logger.setup_logger()


def main(sleep_time: float, max_record_count: int) -> None:

    df = pd.read_parquet(SOURCE_URL)

    if max_record_count == 0:
        max_record_count = len(df)

    record_counter = 0

    top_level_fields = {'event_type': EVENT_TYPE,
                        'schema_version': SCHEMA_VERSION}

    for index, row in df.iterrows():

        try:
            json_header = {'Content-Type': 'application/json'}
            url = COLLECTOR_URL
            row_data = row.to_dict()
            if EVENT_TYPE == 'YellowTaxiTripRecords':
                row_data['tpep_pickup_datetime'] = row_data['tpep_pickup_datetime'].strftime('%Y-%m-%d %H:%M:%S')
                row_data['tpep_dropoff_datetime'] = row_data['tpep_dropoff_datetime'].strftime('%Y-%m-%d %H:%M:%S')
            else:
                row_data['lpep_pickup_datetime'] = row_data['lpep_pickup_datetime'].strftime('%Y-%m-%d %H:%M:%S')
                row_data['lpep_dropoff_datetime'] = row_data['lpep_dropoff_datetime'].strftime('%Y-%m-%d %H:%M:%S')
            processed_event = {**top_level_fields, "payload": row_data}
            logger.info(requests.post(url, data=json.dumps(processed_event), headers=json_header))
        except:
            logger.error(f'Failed to process payload')

        record_counter += 1
        time.sleep(sleep_time)

        if record_counter >= max_record_count:
            break


if __name__ == "__main__":

    parser = argparse.ArgumentParser(description="Runs producer which writes TLCTripRecordData to collector endpoint")
    parser.add_argument('-s', '--sleep_time', required=False, default='0.5')
    parser.add_argument('-m', '--max_record_count', required=False, default='10000')
    args = parser.parse_args()

    logger.info(f'Starting sample producer (TLCTripRecordData) with {args.sleep_time} s between sending records')

    main(float(args.sleep_time), int(args.max_record_count))

Note the

SOURCE_URL = os.environ['DATA_URL']
EVENT_TYPE = os.environ['EVENT_TYPE']
SCHEMA_VERSION = os.environ['SCHEMA_VERSION']
COLLECTOR_URL = os.environ['COLLECTOR_URL']

<...>

df = pd.read_parquet(SOURCE_URL)

We will be passing location to the NYS cab datasets together with other configurations via environmental variables which we will configure when creating Pods in Kubernetes.

2. Logger options that are the same as in the Collector application.

import logging
import os

LOG_FORMAT = f'[%(asctime)s]: {os.getpid()} %(levelname)s %(message)s'
DATE_FORMAT = '%Y-%m-%d %H:%M:%S'
HANDLERS = [logging.StreamHandler()]
LOG_LEVEL = logging.DEBUG


def setup_logger() -> logging.Logger:

    logging.basicConfig(level=LOG_LEVEL,
                        format=LOG_FORMAT,
                        datefmt=DATE_FORMAT,
                        handlers=HANDLERS)

    return logging.getLogger()

3. Python requirements for the environment.

pandas
pyarrow
requests

4. Dockerfile for building the application container.

FROM python:3.9

WORKDIR /code/src

COPY ./requirements.txt /code/requirements.txt

RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

COPY ./src /code/src

CMD ["python", "app.py"]

5. Kubernetes Pod definition manifest.

apiVersion: v1
kind: Pod
metadata:
  name: producer-1
  namespace: swirlai
spec:
  containers:
  - name: producer
    image: aurimasg/producer-yt:0.2.0
    imagePullPolicy: Always
    env:
    - name: DATA_URL
      value: https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-11.parquet
    - name: EVENT_TYPE
      value: YellowTaxiTripRecords
    - name: SCHEMA_VERSION
      value: 0-1-0
    - name: COLLECTOR_URL
      value: http://collector/api/v1/collect

---

apiVersion: v1
kind: Pod
metadata:
  name: producer-2
  namespace: swirlai
spec:
  containers:
  - name: producer
    image: aurimasg/producer-yt:0.2.0
    imagePullPolicy: Always
    env:
    - name: DATA_URL
      value: https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2024-11.parquet
    - name: EVENT_TYPE
      value: GreenTaxiTripRecords
    - name: SCHEMA_VERSION
      value: 0-1-0
    - name: COLLECTOR_URL
      value: http://collector/api/v1/collect

Note that we pass location to the data to be downloaded via an environment variable. You can create the pods by running:

kubectl apply -f pods.yaml

Subscribe now

Implementing Spark ETL.

We will implement the entire ETL in a single Airflow DAG. The code will also live in a single script for simplicity case.

1. So the data now continuously lands in the sai-raw bucket, each of the file is a json file with 50 batched records.

2. The first thing we will want to do is move all of the accumulated data into the sai-processing bucket. Why?

   1. We will be reading the data with Spark and it needs a static set of files for processing. Remember, sai-raw is continuously updated with new data.

   2. We want to fix the ETL batch and timestamp so that something unexpected happens it is always possible to reprocess.

3. We will read the json files in sai-processing bucket with Spark, expand json payloads into structured data and split all of the data into datasets and their versions accordingly.

4. Once Spark processing is completed, we will move the data that is being processed into an archive bucket.

Creating Spark Cluster.

• Creation of the cluster is straightforward, go to Managed Spark section and click on “+Create cluster”.

• Configure it:

  • Give it a name.

  • Select the beefiest configuration, it’s free ;)

  • Create a password, you will need it in the Airflow section.

• Start a session:

  • Go to Sessions tab in the created cluster.

  • Click “+Create session”

• Most default configurations will be good:

  • Name your session.

  • Give some breathing room to the driver and executors by increasing Disk size a bit.

  • bump the number of executors to 3.

That’s it, we have our Spark cluster ready to take our work.

Running Airflow.

We will be deploying Airflow Application on the K8s cluster we have already deployed. Here is how you can easily do it in Nebius:

• When in the target cluster window select Applications tab. Look for Airflow and select the card under “All applications”.

• You will be dropped to application details window, continue by selecting deploy.

• In the following window edit some important fields:

  • In the Namespace field add airflow - it will help us monitor the health of applications as we are deploying more than one in the cluster.

  • Webserver Secret Key - follow the instructions on the right to generate the value.

  • In the DAGs git repo field add the url to your github repo. You can also use mine if you just want to follow the example - https://github.com/AurimasGr/sai-nebius-spark.git

  • DAGs git sub path - point to the folder where you are storing the DAG files in your repository. Mine are under dags as you can see in the picture below.

  • [Important]: I have emitted one field from the image - Custom Airflow Image. We will need a custom one, because we will be running spark directly in Airflow via attached Spark Session, hence we need nebius-connect library installed. You can add aurimasg/airflow there, it will pull the image from my docker hub registry if you keep Airflow version at 2.9.1 as per default. If you want to build your own image, you can do that by going to airflow folder in my github repo and running:

docker build . --platform linux/amd64 -t <docker_hub_account>/<repository>:2.9.1

docker push <docker_hub_account>/<repository>:2.9.1

• Then you can also add <docker_hub_account>/<repository> in the Custom Airflow Image field.

• Deploy the application and wait for it to successfully spin up. You can check status by running:

kubectl get pods -n airflow

Once fully deployed, you should see something like this:

Expose access to airflow by running:

kubectl -n airflow port-forward services/airflow-webserver 8080:8080

You can now go to localhost:8080 to access Airflow UI. Log in with admin:admin credentials. Congratulations, now you can run your DAGs. If you have already pointed Airflow to the DAG folder that contains the ETL script, you will see some errors. The are all relating to Airflow Variables that need to be configured to run the app. Here is what you need to do:

• Go to Admin → Variables.

• In the picture I have mine already configured, but here is what you will need:

  • AWS_ACCESS_KEY_ID_SECRET - AWS like AWS_ACCESS_KEY_ID you generated earlier in collector section.

  • AWS_SECRET_ACCESS_KEY_SECRET - AWS like AWS_SECRET_ACCESS_KEY you generated earlier in collector section.

  • AWS_DEFAULT_REGION - eu-north1 if you are running in this region.

  • AWS_ENDPOINT_URL - https://storage.eu-north1.nebius.cloud:443

  • NB_SPARK_SESSION_ENDPOINT - Find it as shown below

    

  • NB_SPARK_SESSION_PASSWORD_SECRET - Spark cluster password that you created when bootstrapping the cluster.

Here is the code we will use:

• Definition of the DAG:

with DAG(dag_id="spark_etl", 
         start_date=datetime(2025, 1, 24), 
         max_active_runs=1, catchup=False, 
         schedule="*/20 * * * *") as dag:

    @task()
    def move_data_to_processing():
        move_objects(RAW_LANDING_BUCKET, RAW_PROCESSING_BUCKET)

    @task()
    def run_processing():
        process_data(time_now)

    @task()
    def clean_procesing():
        archive_objects(RAW_PROCESSING_BUCKET, ARCHIVE_BUCKET, time_now)

    move_data_to_processing() >> run_processing() >> clean_procesing()

It will run every 20 minutes and run functions move_objects(), process_data(), archive_objects() in sequence.

Here are the functions themselves:

• move_objects() - simply moves objects from one bucket to another. We will use it to move all data accumulated in sai-raw bucket to sai-processing bucket:

def move_objects(source_bucket, destination_bucket):
    # List all objects in the source bucket
    response = s3.list_objects_v2(Bucket=source_bucket)

    if 'Contents' not in response:
        print("No objects found in the source bucket.")
        return

    for obj in response['Contents']:
        key = obj['Key']
        print(f"Moving: {key}")

        # Copy the object to the destination bucket
        copy_source = {'Bucket': source_bucket, 'Key': key}
        s3.copy_object(CopySource=copy_source, Bucket=destination_bucket, Key=key)

        # Delete the object from the source bucket after copying
        s3.delete_object(Bucket=source_bucket, Key=key)

        print(f"Moved {key} to {destination_bucket}")

    print("All objects have been moved.")

    return "Success"

• process_data() - this is the spark function that reads all the json objects in sai-procesing bucket, groups events by event_type, major schema, adds etl_timestamp and transforms the data to structured format. Then it writes the result to sai-processed bucket:

def process_data(etl_timestamp):

    from pyspark.sql.connect.session import SparkSession
    from nebius.spark.connect import create_channel_builder
    from os.path import expanduser
    from pyspark.sql.functions import from_json, col, lit

    import urllib.request

    print(os.environ["NB_SPARK_SESSION_ENDPOINT"])

    url = "https://storage.eu-north1.nebius.cloud/msp-certs/ca.pem"

    urllib.request.urlretrieve(url, "ca.pem")

    nebius_spark_endpoint = os.environ["NB_SPARK_SESSION_ENDPOINT"] + ':443'
    nebius_spark_cb = create_channel_builder(
        nebius_spark_endpoint,
        password=os.environ["NB_SPARK_CLUSTER_PASSWORD"],
        root_certificates_file=expanduser('ca.pem') 
    )

    print(expanduser('ca.pem'))

    spark = (SparkSession
        .builder
        .channelBuilder(nebius_spark_cb)
        .getOrCreate())

    df_raw = spark.read.schema(schema).json(f"s3a://{RAW_PROCESSING_BUCKET}/")
    distinct_groups = df_raw.select("event_type", "schema_version").distinct().collect()

    for (event_type, schema_version) in [(str(row["event_type"]), str(row["schema_version"])) for row in distinct_groups]:
        (df_raw
        .filter(df_raw.event_type == event_type)
        .filter(df_raw.schema_version == schema_version)
        .withColumn('payload', from_json(col('payload'), schema_map.get(f"{event_type}_{schema_version}")))
        .withColumn('etl_timestamp', lit(etl_timestamp))
        .select("etl_timestamp", "collector_id", "collector_tstamp", "event_type", "root_id", "schema_version", "payload.*")
        .write
        .mode("append")
        .partitionBy("etl_timestamp")
        .parquet(f"s3a://{PROCESSED_BUCKET}/{event_type}/{schema_version.split('-')[0]}"))

    return "Success"

• archive_objects() - this is similar to the first function but it moves the data from sai-processing to sai-archive bucket and also adds etl_timestamp hierarchy. Why? We might need to reprocess the batch of raw data, we can always do it from the archive bucket.

def archive_objects(source_bucket, destination_bucket, etl_timestamp):
    # List all objects in the source bucket
    response = s3.list_objects_v2(Bucket=source_bucket)

    if 'Contents' not in response:
        print("No objects found in the source bucket.")
        return

    for obj in response['Contents']:
        key = obj['Key']
        print(f"Moving: {key}")

        # Copy the object to the destination bucket
        copy_source = {'Bucket': source_bucket, 'Key': f"{key}"}
        s3.copy_object(CopySource=copy_source, Bucket=destination_bucket, Key=f"{etl_timestamp}/{key}")

        # Delete the object from the source bucket after copying
        s3.delete_object(Bucket=source_bucket, Key=key)

        print(f"Moved {key} to {destination_bucket}")

    print("All objects have been moved.")

    return "Success"

Congratulations on reaching the end!

If you have reached this part of the Newsletter - congratulations! You will be able to see your Spark ETL running every 20 minutes, moving data between the buckets and sai-raw bucket being continuously filled with new raw data.

If you want to play around more, find new datasources and send them through the collector service. Then explore how the data changes in the sai-processed bucket.

Let me know if you had any issues while following the project! Hope to see you in the next article :)

Subscribe now

Share

Partner with SwirlAI

👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in GenAI, MLOps, Data Engineering, Machine Learning and overall Data space.

I hope you had a wonderful holiday season! As we move into the year of AI Agents, I also have a present for you, the second part in the series of “Building AI Agents from scratch”. If you missed the previous episode (part 1), you can find it here:

We implemented Tool Use pattern without using any LLM Orchestrator frameworks. Today, we will build on top of the previous project, I will explain how later in the article.

In this article you will learn:

• What Reflection pattern in AI Agent systems is.

• How it relates to short-term memory.

• Pros and Cons of implementing Reflection.

• How to build an Agent class that is able to implement Reflection pattern taking the memory into consideration without using any orchestration frameworks.

  • Just to remind ourselves why we are doing this. If you are using any orchestration frameworks for agentic applications, you might be abstracted away from how Agentic patterns are actually implemented there. Having clarity of how the systems actually work helps you build up systems thinking, enabling you to craft advanced applications more efficiently.

• Fix some of the hallucinations we were producing in the previous project!

You can find the code examples for this and other projects in my GitHub repository here:

AI Engineer's Handbook

Be sure to star the repo if you find the content useful, a lot more to come!

YouTube video where I go through the first part of the series is coming next week. Be sure to subscribe to not miss it here.

As always, if something does not work as expected, feel free to DM me or leave a comment, let’s figure it out together!

Defining Reflection in AI Agents.

As it is with most definitions in AI Agents nowadays, there is no single way to specifically describe Reflection. The high level definition of the pattern is:

The ability of an Agentic System to reflect on it’s outputs and suggest improvements. Optionally, also improve the behaviour of the future actions in the system incorporating the feedback provided.

When explaining Agentic concepts I like to drop the abstraction of an Agent and think in Agentic flows - it is easier to reason when analysing flow diagrams and Agents eventually are just a number of steps interconnected via different topologies. Reflection can be applied in different steps of an Agentic flow. Let’s see how.

The simplest example.

The above Agentic flow includes the following steps:

1. User prompts the LLM with a query.

2. Generated answer is passed to the LLM where it is asked to provide feedback to the previously generated answer and provide instructions for improvement if any.

3. The improved answer is returned to the user.

As simple as it is, even this pipeline will provide significant improvements to the accuracy of the answers in many cases. Ofcourse, similar improvements can be achieved via prompt engineering of the system prompt, but reflection pattern is usually more powerful and flexible.

Reflection Loop.

The above Agentic flow is defined via the following steps:

1. A user query is passed to the LLM.

2. Generated answer is passed to the LLM where it is asked to provide feedback to the previously generated answer and provide instructions for improvement if any.

3. After applying the improvements, the improved answer is passed to the LLM again and is asked to provide feedback and suggestions for improvements once more. This loop is then repeated for a predefined amount of times or until the LLM is not able to generate more suggestions and returns a stop sign, usually a predefined string like “END”.

4. The final answer is returned to a user.

This kind of system is powerful, but requires a very specific use case. In literature it can be most often found described for code generation. It is easy to understand why - generated code can be continuously improved in multiple iterations. The system will always find where to over-engineer ;)

Andrew Ng has defined this pattern for code in one of his articles here.

In most real world use cases I would find it hard to make a business case for the use of reflection loop as it usually increases costs significantly.

Validating execution plans.

I find one of the useful spots to place a Reflection step is to validate an execution plan if planning is part of the Agentic flow. Let’s see how it could look like:

1. A user query with intent and a system prompt is passed to the LLM. The LLM generates an execution plan. This is also where an important point where the execution of the flow might break is. An example of the breakage would be the following:

   1. Agentic flow decides if a direct answer needs to be returned to the user or a predefined tool should be used for additional context generation.

   2. A decision is made that a tool should be used,

   3. An incorrect set of parameters to be passed to the tool are hallucinated.

   4. The tool returns an error.

2. With properly crafted Reflection step we can prompt the Agent to try and fix any hallucinations produced in previous step.

3. The plan can return a direct answer that is provided to the user,

4. Or it can prompt to use a tool which enriches the answer via another LLM call and it then is returned to the user.

We will actually be implementing this kind of flow in our hands-on example in the following sections.

Subscribe now

More Complex Reflection flows.

As mentioned before, there is no predefined place for a reflection step to be invoked. In complex Agentic flows it can be used multiple times to validate intermediary answers, plans or any other part of Agentic topology.

In reality, in order to automate complex processes that are present in organisations, you would build multi-step topologies with multiple probabilistic routers connecting execution nodes. Some routers could be implemented via LLMs, some might be rule based, some might use regular ML models. Some execution nodes will have tools, some will be just LLM calls, some will have non-probabilistic executions. Reflection steps could be implemented all around the place to increase the accuracy of non-deterministic routers and executions.

Connection between Agent Memory and Reflection pattern.

Usually, there is a need for some sort of short-term (working) memory implementation to make Reflection provide best results. Let’s see why by examining the third example of Agentic flows defined above.

The Agentic flow until the Reflection step can be invoked includes:

A) A System Prompt and a user query that will be passed to the LLM in order to generate the initial Execution Plan.

B) The execution plan generated by the initial steps.

C) The Reflection step will need All of this information since the System prompt will most likely have useful information about the available tools and similar, The user Query is important since it invokes the generation with context about user intent, the plan is important because it is what we are trying to improve.

In base generation implementation we are not keeping this information in memory, that is why the Working Memory needs to be implemented.

In the above picture I show the simplest implementation of Working Memory - Each of the interactions with an agent are simply stored in the list and passed as additional context to the system prompt each time the generation is invoked. This is how the simplest type of memory is implemented in the Chat Bots (e.g. ChatGPT).

If we are implementing an Agent with the capability of planning and we want to include the reflection capability, this is how it could look like:

As you can see, it is very similar, the only difference is that the initial response by the agent is the execution plan and the next query by the user is a prompt to reflect on the plan given previous interactions.

Pros and Cons of using Reflection.

As with anything that can bring improvements to the system, there is always upsides and downsides.

Pros.

• Almost guaranteed improvement in the accuracy of final outputs of the system. Sometimes this is the only way to make your application feasible due to extremely high accuracy requirements.

• Flexible compared to editing the initial system prompt. E.g. different Reflection methods or even agents can be utilised for different parts of the pipeline.

• It is possible to achieve more with small models when Reflection is applied. Ofcourse, you should analyse the tradeoff given all of the cons but in general there is always a phase in the project where the goal is to optimise the costs of your AI system.

Cons.

• Adds complexity to the application.

• Adds additional latency to the end-to-end flow since additional LLM calls are invoked.

• Adds additional cost since the LLMs are prompted at least one more additional time (usually more than one) with every Reflection step.

Subscribe now

Building the Reflection Agent.

As mentioned at the beginning of the article, we will be working on the example where Reflection will be used to revise an action plan generated by a LLM. And this is not without a reason!

If you followed me in the first part of the series where we implemented tool usage from scratch, you might remember that we implemented a tool that is capable of converting between two currencies on demand. We prompted the agent to only use the tool if the conversion is actually needed. It worked well, but in the example we asked for Serbian to Japanese currency conversion.

Now, this was not my initial intent. I am from Lithuania and I wanted to showcase the scenario where I would be traveling from Lithuania to Japan so I initially prompted the model with the following query:

I am traveling to Japan from Lithuania, I have 1500 of local currency, how much of Japanese currency will I be able to get?

Imagine my surprise, when the agent returned the following response:

Thought: I need to convert 1500 Lithuanian Litas (LTL) to Japanese Yen (JPY) using the currency conversion tool.
Plan: Use convert_currency tool to convert 1500 LTL to JPY. Return the conversion result
Results: Error: Could not fetch exchange rates

Ofcourse the tool call resulted in an error - Lithuania has Euro (EUR) as an official currency from 2015! The tool was obviously not able to find LTL conversion rate to JPY.

And then I took it personal, in this episode my quest is to try and fix the plan using the Reflection pattern.

You can also find the code in a GitHub repository here.

You can follow the tutorial using the Jupyter notebook here.

Implementing the working memory.

We will implement a single interaction with an agent as a Dataclass:

@dataclass
class Interaction:
    """Record of a single interaction with the agent"""
    timestamp: datetime
    query: str
    plan: Dict[str, Any]

It will have the query and the plan that the agent produced.

It is important to note that we will also need the system prompt to be available for reflection step, but we will implement that separately.

We will be simplifying the Agent Class this time around by stripping any tool related functionality from it to better focus on reflecting on the plan generated by the Agent.

The initial system prompt.

We will keep the system prompt identical to the one we implemented in the part 1 of the series. You can find the explanation in this section of my previous article. The two differences are:

• We are now mocking the available tool instead of actually implementing it. So the tools section will look like this:

"tools": [
    {
        "name": "convert_currency",
        "description": "Converts currency using latest exchange rates.",
        "parameters": {
            "amount": {
                "type": "float",
                "description": "Amount to convert"
            },
            "from_currency": {
                "type": "str",
                "description": "Source currency code (e.g., USD)"
            },
            "to_currency": {
                "type": "str",
                "description": "Target currency code (e.g., EUR)"
            }
        }
    }
]

• We are extending the capabilities and instructions of the agent by including the fourth line in both:

"capabilities": [
    "Using provided tools to help users when necessary",
    "Responding directly without tools for questions that don't require tool usage",
    "Planning efficient tool usage sequences",
    "If asked by the user, reflecting on the plan and suggesting changes if needed"
],
"instructions": [
    "Use tools only when they are necessary for the task",
    "If a query can be answered directly, respond with a simple message instead of using tools",
    "When tools are needed, plan their usage efficiently to minimize tool calls",
    "If asked by the user, reflect on the plan and suggest changes if needed"
]

Implementing the Agent Class.

The agent class will be initialised with an empty Interaction history. Interaction history IS the working memory in this example.

class Agent:
    def __init__(self, model: str = "gpt-4o-mini"):
        """Initialize Agent with empty interaction history."""
        self.client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
        self.interactions: List[Interaction] = []  # Working memory
        self.model = model

Planning.

The plan method prompts the LLM to create the initial execution plan and updates the working memory with the user query and the generated plan.

def plan(self, user_query: str) -> Dict:
    """Use LLM to create a plan and store it in memory."""
    messages = [
        {"role": "system", "content": self.create_system_prompt()},
        {"role": "user", "content": user_query}
    ]
    
    response = self.client.chat.completions.create(
        model=self.model,
        messages=messages,
        temperature=0
    )
    
    try:
        plan = json.loads(response.choices[0].message.content)
        # Store the interaction immediately after planning
        interaction = Interaction(
            timestamp=datetime.now(),
            query=user_query,
            plan=plan
        )
        self.interactions.append(interaction)
        return plan
    except json.JSONDecodeError:
        raise ValueError("Failed to parse LLM response as JSON")

Reflecting on the plan.

The reflect_on_plan method is where the interesting part is.

def reflect_on_plan(self) -> Dict[str, Any]:
    """Reflect on the most recent plan using interaction history."""
    if not self.interactions:
        return {"reflection": "No plan to reflect on", "requires_changes": False}
    
    latest_interaction = self.interactions[-1]
    
    reflection_prompt = {
        "task": "reflection",
        "context": {
            "user_query": latest_interaction.query,
            "generated_plan": latest_interaction.plan
        },
        "instructions": [
            "Review the generated plan for potential improvements",
            "Consider if the chosen tools are appropriate",
            "Verify tool parameters are correct",
            "Check if the plan is efficient",
            "Determine if tools are actually needed"
        ],
        "response_format": {
            "type": "json",
            "schema": {
                "requires_changes": {
                    "type": "boolean",
                    "description": "whether the plan needs modifications"
                },
                "reflection": {
                    "type": "string",
                    "description": "explanation of what changes are needed or why no changes are needed"
                },
                "suggestions": {
                    "type": "array",
                    "items": {"type": "string"},
                    "description": "specific suggestions for improvements",
                    "optional": True
                }
            }
        }
    }
    
    messages = [
        {"role": "system", "content": self.create_system_prompt()},
        {"role": "user", "content": json.dumps(reflection_prompt, indent=2)}
    ]
    
    response = self.client.chat.completions.create(
        model=self.model,
        messages=messages,
        temperature=0
    )
    
    try:
        return json.loads(response.choices[0].message.content)
    except json.JSONDecodeError:
        return {"reflection": response.choices[0].message.content}

We create a new prompt with specific instructions and desired output format. Note that the prompt has both the user query and the initial plan passed as additional context via a context key. Then we pass the prompt together with the initial system prompt to generate reflection on the initially generated plan including any suggestions for improvements.

Subscribe now

Executing the flow and generating Revised Plan.

We now stitch the flow together via the execute method.

def execute(self, user_query: str) -> str:
    """Execute the full pipeline: plan, reflect, and potentially replan."""
    try:
        # Create initial plan (this also stores it in memory)
        initial_plan = self.plan(user_query)
        
        # Reflect on the plan using memory
        reflection = self.reflect_on_plan()
        
        # Check if reflection suggests changes
        if reflection.get("requires_changes", False):
            # Generate new plan based on reflection
            messages = [
                {"role": "system", "content": self.create_system_prompt()},
                {"role": "user", "content": user_query},
                {"role": "assistant", "content": json.dumps(initial_plan)},
                {"role": "user", "content": f"Please revise the plan based on this feedback: {json.dumps(reflection)}"}
            ]
            
            response = self.client.chat.completions.create(
                model=self.model,
                messages=messages,
                temperature=0
            )
            
            try:
                final_plan = json.loads(response.choices[0].message.content)
            except json.JSONDecodeError:
                final_plan = initial_plan  # Fallback to initial plan if parsing fails
        else:
            final_plan = initial_plan
        
        # Update the stored interaction with all information
        self.interactions[-1].plan = {
            "initial_plan": initial_plan,
            "reflection": reflection,
            "final_plan": final_plan
        }
        
        # Return the appropriate response
        if final_plan.get("requires_tools", True):
            return f"""Initial Thought: {initial_plan['thought']}
Initial Plan: {'. '.join(initial_plan['plan'])}
Reflection: {reflection.get('reflection', 'No improvements suggested')}
Final Plan: {'. '.join(final_plan['plan'])}"""
        else:
            return f"""Response: {final_plan['direct_response']}
Reflection: {reflection.get('reflection', 'No improvements suggested')}"""
        
    except Exception as e:
        return f"Error executing plan: {str(e)}"

Note that the method applies reflection improvement on the plan ONLY if requested by the reflection generation. If not, we keep the old plan.

Executing the Agent.

Let’s see if we managed to fix the plan! Execute the agent by running:

query_list = ["I am traveling to Japan from Lithuania, I have 1500 of local currency, how much of Japanese currency will I be able to get?"]

for query in query_list:
    print(f"\nQuery: {query}")
    result = agent.execute(query)
    print(result)

If you used the same models as me, you should be getting something similar to:

Query: I am traveling to Japan from Lithuania, I have 1500 of local currency, how much of Japanese currency will I be able to get?
Initial Thought: I need to convert 1500 Lithuanian Litas (LTL) to Japanese Yen (JPY) using the currency conversion tool.
Initial Plan: Use convert_currency tool to convert 1500 LTL to JPY. Return the conversion result
Reflection: The plan needs modifications because the Lithuanian Litas (LTL) is no longer in use since Lithuania adopted the Euro (EUR) in 2015. Therefore, the conversion should be from EUR to JPY instead of LTL.
Final Plan: Use convert_currency tool to convert 1500 EUR to JPY. Return the conversion result

Great success! we have fixed the plan and would be able to execute the tool successfully.

That’s it for today, we’ve learned:

• How to implement a simple type of Working Memory.

• How to construct a Reflection step to reflect on the execution plan generated by the Agent.

• How to implement the suggestions generated by the reflection step and improve the plan.

  ---

Subscribe now

Partner with SwirlAI

Share

👋 I am Aurimas. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in GenAI, MLOps, Data Engineering, Machine Learning and overall Data space.

First of all, I want to wish you a joyful and peaceful holiday season in advance!

This is the first article in the series where we will build AI Agents from scratch without using any LLM orchestration frameworks. In this one you will learn:

• What are agents?

• How the Tool usage actually works.

• How to build a decorator wrapper that extracts relevant details from a Python function to be passed to the LLM via system prompt.

• How to think about constructing effective system prompts that can be used for Agents.

• How to build an Agent class that is able to plan and execute actions using provided Tools.

You can find the code examples for this and following projects in GitHub repository here:

AI Engineer's Handbook

If something does not work as expected, feel free to DM me or leave a comment, let’s figure it out together!

“The future of AI is Agentic.”

“Year 2025 will be the year of Agents.”

These are the phrases you hear nowadays left and right. And there is a lot of truth to it. In order to bring the most business value out of LLMs, we are turning to complex agentic flows.

What is an AI Agent?

In it’s simplest high level definition, an AI agent is an application that uses LLM at the core as it’s reasoning engine to decide on the steps it needs to take to solve for users intent. It is usually depicted similar to the picture bellow and is composed of multiple building blocks:

• Planning - the capability to plan a sequence of actions that the application needs to perform in order to solve for the provided intent.

• Memory - short-term and long-term memory containing any information that the agent might need to reason about the actions it needs to take. This information is usually passed to LLM via a system prompt as part of the core. You can read more about different types of memories in one of my previous articles:

• Tools - any functions that the application can call to enhance it’s reasoning capabilities. One should not be fooled by the simplicity of this definition as a tool can be literally anything:

  • Simple functions defined in code.

  • VectorDBs and other data stores containing context.

  • Regular Machine Learning model APIs.

  • Other Agents!

  • …

In the following set of articles, I will implement most of the moving parts of an agent from scratch without using any orchestration frameworks. This episode is about Tool use.

If you are using any orchestration frameworks for agentic applications, you might be abstracted away from what using a tool really means. This article will help you understand what providing a tool and using it via an agent involves. I believe that understanding applications from the base building blocks is really important for few reasons:

• Frameworks hide the implementation details of the system prompts used, different approaches might be needed in different use cases.

• You might want to tune the low level details to achieve most optimal performance of the agent.

• Having clarity of how the systems actually work helps build up your systems thinking enabling you to craft advanced applications more efficiently.

Tool use on a high level.

The basic thing one needs to understand when building agentic applications is that LLMs do not run code, they are only used to produce intent via prompting. Why can ChatGPT browse the internet and return more accurate and recent results? Because ChatGPT IS an agent and there are many non LLM building blocks hidden from us behind the API.

Prompt engineering becomes critical when building agentic applications. More specifically, how you craft the system prompt. Simplified prompt structure looks like the following.

The agent will only perform well if you are able to efficiently provide the system prompt with available tool definitions and expected outputs which are in a form of planned actions or raw answers.

Implementing the Agent.

In this part, we will create an AI Agent, that is capable of checking currency conversion rates online and performing the conversion if needed to answer the user query.

You can also find the code in a GitHub repository here.

You can follow the tutorial using the Jupyter notebook here.

I will also create a Youtube video explaining the process in the following weeks. If you don’t want to miss it, you can subscribe to the Youtube channel here.

Subscribe now

Preparing python functions to be used as tools.

The easiest and most convenient way to provide tools to an agent is through functions, in our project we will be using Python for this.

We do not need to provide the function code itself to the system prompt but we need to extract useful information about it so that LLM can decide if and how the function should be invoked.

We’ll define a dataclass that contains desired information including the function runnable.

@dataclass
class Tool:
    name: str
    description: str
    func: Callable[..., str]
    parameters: Dict[str, Dict[str, str]]
    
    def __call__(self, *args, **kwargs) -> str:
        return self.func(*args, **kwargs)

The information we are extracting includes:

• Function name.

• function description (we will extract this from a docstring).

• Function callable so that we can invoke it as part of the agent.

• Parameters that should be used with the function so that the LLM can decide on how to call the function.

Now we will need to extract the above information from the functions we define. One requirement for the functions we will enforce is to have properly formatted docstrings. We will require the following format:

"""Description of what the tool does.

Parameters:
    - param1: Description of first parameter
    - param2: Description of second parameter
"""

The following function extracts information about parameters - parameter names and descriptions.

def parse_docstring_params(docstring: str) -> Dict[str, str]:
    """Extract parameter descriptions from docstring."""
    if not docstring:
        return {}
    
    params = {}
    lines = docstring.split('\n')
    in_params = False
    current_param = None
    
    for line in lines:
        line = line.strip()
        if line.startswith('Parameters:'):
            in_params = True
        elif in_params:
            if line.startswith('-') or line.startswith('*'):
                current_param = line.lstrip('- *').split(':')[0].strip()
                params[current_param] = line.lstrip('- *').split(':')[1].strip()
            elif current_param and line:
                params[current_param] += ' ' + line.strip()
            elif not line:
                in_params = False
    
    return params

We will be extracting function parameter types from typehints provided via function definition. The bellow function will help format them.

def get_type_description(type_hint: Any) -> str:
    """Get a human-readable description of a type hint."""
    if isinstance(type_hint, _GenericAlias):
        if type_hint._name == 'Literal':
            return f"one of {type_hint.__args__}"
    return type_hint.__name__

A very convenient way to turn a function into a tool is to use a decorator. The below code defines a tool decorator that wraps a function if used. It uses either function name for the tool name or a variable provided via decorator.

def tool(name: str = None):
    def decorator(func: Callable[..., str]) -> Tool:
        tool_name = name or func.__name__
        description = inspect.getdoc(func) or "No description available"
        
        type_hints = get_type_hints(func)
        param_docs = parse_docstring_params(description)
        sig = inspect.signature(func)
        
        params = {}
        for param_name, param in sig.parameters.items():
            params[param_name] = {
                "type": get_type_description(type_hints.get(param_name, Any)),
                "description": param_docs.get(param_name, "No description available")
            }
        
        return Tool(
            name=tool_name,
            description=description.split('\n\n')[0],
            func=func,
            parameters=params
        )
    return decorator

Currency exchange tool.

The below creates a tool from a function that takes in the amount of currency to exchange from, the currency code to be converted from and the currency code to convert to. The function searches for the relevant currency exchange rate and performs the calculation of resulting currency amount.

@tool()
def convert_currency(amount: float, from_currency: str, to_currency: str) -> str:
    """Converts currency using latest exchange rates.
    
    Parameters:
        - amount: Amount to convert
        - from_currency: Source currency code (e.g., USD)
        - to_currency: Target currency code (e.g., EUR)
    """
    try:
        url = f"https://open.er-api.com/v6/latest/{from_currency.upper()}"
        with urllib.request.urlopen(url) as response:
            data = json.loads(response.read())
            
        if "rates" not in data:
            return "Error: Could not fetch exchange rates"
            
        rate = data["rates"].get(to_currency.upper())
        if not rate:
            return f"Error: No rate found for {to_currency}"
            
        converted = amount * rate
        return f"{amount} {from_currency.upper()} = {converted:.2f} {to_currency.upper()}"
        
    except Exception as e:
        return f"Error converting currency: {str(e)}"

Let’s just run

convert_currency

It should return something like

Tool(name='convert_currency', description='Converts currency using latest exchange rates.', func=<function convert_currency at 0x106d8fa60>, parameters={'amount': {'type': 'float', 'description': 'Amount to convert'}, 'from_currency': {'type': 'str', 'description': 'Source currency code (e.g., USD)'}, 'to_currency': {'type': 'str', 'description': 'Target currency code (e.g., EUR)'}})

This is great! We have successfully extracted information we will be providing to the LLM as a tool definition.

Subscribe now

Crafting the system prompt.

We will be using gpt-4o-mini as our reasoning engine. It is known that GPT model family performs better when the input prompt is formatted as json. So we will do exactly that. Actually, the system prompt is the most important part of our agent, here is the final one we will be using:

{
    "role": "AI Assistant",
    "capabilities": [
        "Using provided tools to help users when necessary",
        "Responding directly without tools for questions that don't require tool usage",
        "Planning efficient tool usage sequences"
    ],
    "instructions": [
        "Use tools only when they are necessary for the task",
        "If a query can be answered directly, respond with a simple message instead of using tools",
        "When tools are needed, plan their usage efficiently to minimize tool calls"
    ],
    "tools": [
        {
            "name": tool.name,
            "description": tool.description,
            "parameters": {
                name: {
                    "type": info["type"],
                    "description": info["description"]
                }
                for name, info in tool.parameters.items()
            }
        }
        for tool in self.tools.values()
    ],
    "response_format": {
        "type": "json",
        "schema": {
            "requires_tools": {
                "type": "boolean",
                "description": "whether tools are needed for this query"
            },
            "direct_response": {
                "type": "string",
                "description": "response when no tools are needed",
                "optional": True
            },
            "thought": {
                "type": "string", 
                "description": "reasoning about how to solve the task (when tools are needed)",
                "optional": True
            },
            "plan": {
                "type": "array",
                "items": {"type": "string"},
                "description": "steps to solve the task (when tools are needed)",
                "optional": True
            },
            "tool_calls": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "tool": {
                            "type": "string",
                            "description": "name of the tool"
                        },
                        "args": {
                            "type": "object",
                            "description": "parameters for the tool"
                        }
                    }
                },
                "description": "tools to call in sequence (when tools are needed)",
                "optional": True
            }
        },
        "examples": [
            {
                "query": "Convert 100 USD to EUR",
                "response": {
                    "requires_tools": True,
                    "thought": "I need to use the currency conversion tool to convert USD to EUR",
                    "plan": [
                        "Use convert_currency tool to convert 100 USD to EUR",
                        "Return the conversion result"
                    ],
                    "tool_calls": [
                        {
                            "tool": "convert_currency",
                            "args": {
                                "amount": 100,
                                "from_currency": "USD", 
                                "to_currency": "EUR"
                            }
                        }
                    ]
                }
            },
            {
                "query": "What's 500 Japanese Yen in British Pounds?",
                "response": {
                    "requires_tools": True,
                    "thought": "I need to convert JPY to GBP using the currency converter",
                    "plan": [
                        "Use convert_currency tool to convert 500 JPY to GBP",
                        "Return the conversion result"
                    ],
                    "tool_calls": [
                        {
                            "tool": "convert_currency",
                            "args": {
                                "amount": 500,
                                "from_currency": "JPY",
                                "to_currency": "GBP"
                            }
                        }
                    ]
                }
            },
            {
                "query": "What currency does Japan use?",
                "response": {
                    "requires_tools": False,
                    "direct_response": "Japan uses the Japanese Yen (JPY) as its official currency. This is common knowledge that doesn't require using the currency conversion tool."
                }
            }
        ]
    }
}

A lot to unpack, let’s analyse it step by step:

"role": "AI Assistant",
"capabilities": [
    "Using provided tools to help users when necessary",
    "Responding directly without tools for questions that don't require tool usage",
    "Planning efficient tool usage sequences"
],
"instructions": [
    "Use tools only when they are necessary for the task",
    "If a query can be answered directly, respond with a simple message instead of using tools",
    "When tools are needed, plan their usage efficiently to minimize tool calls"
]

This is where we define the qualities of the Agent, in general we are enforcing the behaviour that tools should be used only when necessary.

"tools": [
    {
        "name": tool.name,
        "description": tool.description,
        "parameters": {
            name: {
                "type": info["type"],
                "description": info["description"]
            }
            for name, info in tool.parameters.items()
        }
    }
    for tool in self.tools.values()
]

This is where we unpack the tools into a list. The tool list will be part of Agent class, that is why we loop through self.tools. Remember, each tool is defined by the Dataclass we created in the first part.

"response_format": {
    "type": "json",
    "schema": {
        "requires_tools": {
            "type": "boolean",
            "description": "whether tools are needed for this query"
        },
        "direct_response": {
            "type": "string",
            "description": "response when no tools are needed",
            "optional": True
        },
        "thought": {
            "type": "string", 
            "description": "reasoning about how to solve the task (when tools are needed)",
            "optional": True
        },
        "plan": {
            "type": "array",
            "items": {"type": "string"},
            "description": "steps to solve the task (when tools are needed)",
            "optional": True
        },
        "tool_calls": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "tool": {
                        "type": "string",
                        "description": "name of the tool"
                    },
                    "args": {
                        "type": "object",
                        "description": "parameters for the tool"
                    }
                }
            },
            "description": "tools to call in sequence (when tools are needed)",
            "optional": True
        }
    }
}

Above enforces the LLM output schema. We provide strict instructions here:

• requires_tools: return if tool usage is required.

• direct_response: if above is false return a direct response.

• thought: description on how the task should be solved.

• plan: steps to solve the task.

• tool_calls: tool calls in sequence including functions and parameters to be used. Our example only includes one tool, but it does not necessarily have to.

"examples": [
    {
        "query": "Convert 100 USD to EUR",
        "response": {
            "requires_tools": True,
            "thought": "I need to use the currency conversion tool to convert USD to EUR",
            "plan": [
                "Use convert_currency tool to convert 100 USD to EUR",
                "Return the conversion result"
            ],
            "tool_calls": [
                {
                    "tool": "convert_currency",
                    "args": {
                        "amount": 100,
                        "from_currency": "USD", 
                        "to_currency": "EUR"
                    }
                }
            ]
        }
    },
    {
        "query": "What's 500 Japanese Yen in British Pounds?",
        "response": {
            "requires_tools": True,
            "thought": "I need to convert JPY to GBP using the currency converter",
            "plan": [
                "Use convert_currency tool to convert 500 JPY to GBP",
                "Return the conversion result"
            ],
            "tool_calls": [
                {
                    "tool": "convert_currency",
                    "args": {
                        "amount": 500,
                        "from_currency": "JPY",
                        "to_currency": "GBP"
                    }
                }
            ]
        }
    },
    {
        "query": "What currency does Japan use?",
        "response": {
            "requires_tools": False,
            "direct_response": "Japan uses the Japanese Yen (JPY) as its official currency. This is common knowledge that doesn't require using the currency conversion tool."
        }
    }
]

Finally, we provide some examples of correct reasoning above.

Subscribe now

Implementing the Agent Class

The agent class is quite lengthy due to the long system prompt:

class Agent:
    def __init__(self):
        """Initialize Agent with empty tool registry."""
        self.client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
        self.tools: Dict[str, Tool] = {}
    
    def add_tool(self, tool: Tool) -> None:
        """Register a new tool with the agent."""
        self.tools[tool.name] = tool
    
    def get_available_tools(self) -> List[str]:
        """Get list of available tool descriptions."""
        return [f"{tool.name}: {tool.description}" for tool in self.tools.values()]
    
    def use_tool(self, tool_name: str, **kwargs: Any) -> str:
        """Execute a specific tool with given arguments."""
        if tool_name not in self.tools:
            raise ValueError(f"Tool '{tool_name}' not found. Available tools: {list(self.tools.keys())}")
        
        tool = self.tools[tool_name]
        return tool.func(**kwargs)

    def create_system_prompt(self) -> str:
        """Create the system prompt for the LLM with available tools."""
        tools_json = {
            "role": "AI Assistant",
            "capabilities": [
                "Using provided tools to help users when necessary",
                "Responding directly without tools for questions that don't require tool usage",
                "Planning efficient tool usage sequences"
            ],
            "instructions": [
                "Use tools only when they are necessary for the task",
                "If a query can be answered directly, respond with a simple message instead of using tools",
                "When tools are needed, plan their usage efficiently to minimize tool calls"
            ],
            "tools": [
                {
                    "name": tool.name,
                    "description": tool.description,
                    "parameters": {
                        name: {
                            "type": info["type"],
                            "description": info["description"]
                        }
                        for name, info in tool.parameters.items()
                    }
                }
                for tool in self.tools.values()
            ],
            "response_format": {
                "type": "json",
                "schema": {
                    "requires_tools": {
                        "type": "boolean",
                        "description": "whether tools are needed for this query"
                    },
                    "direct_response": {
                        "type": "string",
                        "description": "response when no tools are needed",
                        "optional": True
                    },
                    "thought": {
                        "type": "string", 
                        "description": "reasoning about how to solve the task (when tools are needed)",
                        "optional": True
                    },
                    "plan": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "steps to solve the task (when tools are needed)",
                        "optional": True
                    },
                    "tool_calls": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "tool": {
                                    "type": "string",
                                    "description": "name of the tool"
                                },
                                "args": {
                                    "type": "object",
                                    "description": "parameters for the tool"
                                }
                            }
                        },
                        "description": "tools to call in sequence (when tools are needed)",
                        "optional": True
                    }
                },
                "examples": [
                    {
                        "query": "Convert 100 USD to EUR",
                        "response": {
                            "requires_tools": True,
                            "thought": "I need to use the currency conversion tool to convert USD to EUR",
                            "plan": [
                                "Use convert_currency tool to convert 100 USD to EUR",
                                "Return the conversion result"
                            ],
                            "tool_calls": [
                                {
                                    "tool": "convert_currency",
                                    "args": {
                                        "amount": 100,
                                        "from_currency": "USD", 
                                        "to_currency": "EUR"
                                    }
                                }
                            ]
                        }
                    },
                    {
                        "query": "What's 500 Japanese Yen in British Pounds?",
                        "response": {
                            "requires_tools": True,
                            "thought": "I need to convert JPY to GBP using the currency converter",
                            "plan": [
                                "Use convert_currency tool to convert 500 JPY to GBP",
                                "Return the conversion result"
                            ],
                            "tool_calls": [
                                {
                                    "tool": "convert_currency",
                                    "args": {
                                        "amount": 500,
                                        "from_currency": "JPY",
                                        "to_currency": "GBP"
                                    }
                                }
                            ]
                        }
                    },
                    {
                        "query": "What currency does Japan use?",
                        "response": {
                            "requires_tools": False,
                            "direct_response": "Japan uses the Japanese Yen (JPY) as its official currency. This is common knowledge that doesn't require using the currency conversion tool."
                        }
                    }
                ]
            }
        }
        
        return f"""You are an AI assistant that helps users by providing direct answers or using tools when necessary.
Configuration, instructions, and available tools are provided in JSON format below:

{json.dumps(tools_json, indent=2)}

Always respond with a JSON object following the response_format schema above. 
Remember to use tools only when they are actually needed for the task."""

    def plan(self, user_query: str) -> Dict:
        """Use LLM to create a plan for tool usage."""
        messages = [
            {"role": "system", "content": self.create_system_prompt()},
            {"role": "user", "content": user_query}
        ]
        
        response = self.client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            temperature=0
        )
        
        try:
            return json.loads(response.choices[0].message.content)
        except json.JSONDecodeError:
            raise ValueError("Failed to parse LLM response as JSON")

    def execute(self, user_query: str) -> str:
        """Execute the full pipeline: plan and execute tools."""
        try:
            plan = self.plan(user_query)
            
            if not plan.get("requires_tools", True):
                return plan["direct_response"]
            
            # Execute each tool in sequence
            results = []
            for tool_call in plan["tool_calls"]:
                tool_name = tool_call["tool"]
                tool_args = tool_call["args"]
                result = self.use_tool(tool_name, **tool_args)
                results.append(result)
            
            # Combine results
            return f"""Thought: {plan['thought']}
Plan: {'. '.join(plan['plan'])}
Results: {'. '.join(results)}"""
            
        except Exception as e:
            return f"Error executing plan: {str(e)}"

Let’s look into it step by step (skipping the create_system_prompt method as we already analysed it in the previous part).

def add_tool(self, tool: Tool) -> None:
    """Register a new tool with the agent."""
    self.tools[tool.name] = tool

def get_available_tools(self) -> List[str]:
    """Get list of available tool descriptions."""
    return [f"{tool.name}: {tool.description}" for tool in self.tools.values()]

def use_tool(self, tool_name: str, **kwargs: Any) -> str:
    """Execute a specific tool with given arguments."""
    if tool_name not in self.tools:
        raise ValueError(f"Tool '{tool_name}' not found. Available tools: {list(self.tools.keys())}")
    
    tool = self.tools[tool_name]
    return tool.func(**kwargs)

Above contain methods to manage tools:

• Attaching tools to the agent.

• List attached tools.

• Invoke execution of a tool.

def plan(self, user_query: str) -> Dict:
    """Use LLM to create a plan for tool usage."""
    messages = [
        {"role": "system", "content": self.create_system_prompt()},
        {"role": "user", "content": user_query}
    ]
    
    response = self.client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        temperature=0
    )
    
    try:
        return json.loads(response.choices[0].message.content)
    except json.JSONDecodeError:
        raise ValueError("Failed to parse LLM response as JSON")

The above simply executes the system prompt, we defined the expected output as part of the system prompt. It exactly provides the actions that the LLM planned or a direct answer if the tool calling is not needed.

def execute(self, user_query: str) -> str:
    """Execute the full pipeline: plan and execute tools."""
    try:
        plan = self.plan(user_query)
        
        if not plan.get("requires_tools", True):
            return plan["direct_response"]
        
        # Execute each tool in sequence
        results = []
        for tool_call in plan["tool_calls"]:
            tool_name = tool_call["tool"]
            tool_args = tool_call["args"]
            result = self.use_tool(tool_name, **tool_args)
            results.append(result)
        
        # Combine results
        return f"""Thought: {plan['thought']}
Plan: {'. '.join(plan['plan'])}
Results: {'. '.join(results)}"""
        
    except Exception as e:
        return f"Error executing plan: {str(e)}"

The above executes the plan method and acts on it. You might remember that the plan can include multiple sequential tool executions, that is why we are looping through planned tool calls.

Running the Agent.

That’s it, we have all of the necessary code to create and use the Agent. in the following code we initialise the agent, attach a convert_currency tool to it and loop through two user queries. First one should require the tool use while the second not.

agent = Agent()
agent.add_tool(convert_currency)

query_list = ["I am traveling to Japan from Serbia, I have 1500 of local currency, how much of Japanese currency will I be able to get?",
                "How are you doing?"]

for query in query_list:
    print(f"\nQuery: {query}")
    result = agent.execute(query)
    print(result)

The output should be similar to:

Query: I am traveling to Japan from Serbia, I have 1500 of local currency, how much of Japanese currency will I be able to get?
Thought: I need to convert 1500 Serbian Dinars (RSD) to Japanese Yen (JPY) using the currency conversion tool.
Plan: Use convert_currency tool to convert 1500 RSD to JPY. Return the conversion result
Results: 1500 RSD = 2087.49 JPY

Query: How are you doing?
I'm just a computer program, so I don't have feelings, but I'm here and ready to help you!

As expected! First query uses the tool, while the second does not.

That’s it for today, we’ve learned:

• How to wrap python functions to be provided as tools to the Agent.

• How to craft a system prompt that uses the tool definitions in planning the execution.

• How to implement the agent that executes on the plan.

  ---

Subscribe now

Partner with SwirlAI

Share

With the rise of AI, a new breed of Cloud that specifically offers GPU optimised resources has emerged. It is no surprise as the demand for GPUs has soared. We call them - AI Clouds. In this article you will find:

• An overview of the role of AI Cloud in the age of AI.

• A hands-on project where you will:

  • Set up a Kubernetes cluster on AI Cloud

  • Deploy an open source LLM (Mistral-7B-Instruct) from HuggingFace using vLLM server on Kubernetes.

  • Build a simple Streamlit based chatbot that uses the previously deployed model endpoint.

  • Expose all of this externally so that you can access the chatbot via your browser.

  • Here is a high level diagram:

If you do run into any problems while following the project, let me know in the comment section or drop me a PM, we will solve it together.

This newsletter episode is sponsored by Nebius - an emerging AI Cloud provider. They are currently running a campaign, offering substantial discounts on the first 1000 GPU hours for NVIDIA® H100 Tensor Core GPU instances with a price point of 1.5 USD/GPU hour.

You should consider registering if you want to take advantage of the offer. Also, you will need an account if you want to follow the hands-on project that follows later in the article.

Create an account

What are AI Clouds and when you might consider using one?

As mentioned at the beginning of the article, AI Clouds are specifically offering GPU optimised resources, usually with advanced performance configurations so that a user can spin up integrated GPU nodes with ease. There has always been a debate if you should simply use a third party LLM API providers like OpenAI/Anthropic or an AI Cloud like Nebius to serve your own open source models for inference purposes. Let’s look into the cons and pros of both.

The LLM application lifecycle has evolved to be complex in the past few years. We went from pre-training of foundation models and then finetuning them for alignment, to building simple RAG (Retrieval Augmented Generation) systems on top of them and eventually evolving them to complex agentic systems capable of more advanced automation.

If you are training your own foundation LLM or aligning it to human preference, there is no escape from needing a GPU cluster, no proprietary model provider will help you here. Unless you are one of the largest companies on the planet that train foundation models, building your own physical GPU clusters is too capital expensive and out of the question, AI Clouds will help you here in getting competitive prices for interconnected and optimised GPU resources. For a startup that is trying to break into frontier model development it might be the only way of getting the opportunity to play in the game.

When it comes to finetuning of the models, some of the largest model providers have finetuning functionality on top of their frontier models, but that increases the cost of inferencing even more and adds additional level of “black box” on the resulting model. For some low traffic/volume use cases this might be just enough. However, if you are serious about the product you are building with LLMs, you might consider finetuning an open source model like Llama 3.1 and serve the resulting model on an AI Cloud.

The interesting part of LLM application lifecycle to analyse when choosing between proprietary model APIs and deploying in AI Clouds is when you are building AI powered systems like RAG or Agents, where inference is key. The choice might not always be obvious there. Let’s delve deeper.

The TCO of LLM inference.

One would argue, that deploying and maintaining your own LLMs on an AI Cloud could be a challenge and brings it’s own additional cost, and that is true. The added cost however is usually fixed and linearly increasing with the amount of GPUs you will need. If you are running distributed workloads, the additional cost of operation (usually human labour) on top of pure GPU costs might be continuously decreasing when divided per GPU operated. Remember, that we are talking about managed GPU clusters and not building your own racks.

Before evaluating costs of using proprietary model APIs vs. hosting your own LLMs on an AI Cloud, you want to consider viability for your application to run on an open source model in the first place. Nowadays it is usually a valid option as open source models are catching up with frontier models quickly, also RAG or agentic applications do not require extremely powerful model in a wide sense as they are using the models as a reasoning/planning engines rather than a system that can answer any question by default. We can usually do well with smaller but more task focused models. The bellow text assumes that you have estimated the viability of using open source models.

TCO (total cost of ownership) is an important measure that eventually should drive your decision of running with proprietary LLM APIs or serving your own model on AI Cloud. When we think of the cost of proprietary LLM APIs it is all about input and output tokens. The good thing about these APIs is that we will always (assuming that the API does not go down and you have a paid plan with high enough quotas) be able to get our inference results and the volume can be easily scaled down to zero. The curve of cost over time fluctuates with the traffic to your application.

On the other hand, if you are deploying your own models on an AI Cloud, you will need a dedicated set of resources that are either always running or are shut down. In the simplest scenario, the cost curve over time is flat. Of course this is a rare case in a production application as you will want to implement some sort of autoscaling to bring your cost per token down and the users of your application happy with the responsiveness of the app.

Throughput, latency and autoscaling in AI Clouds.

The important part to evaluate is what kind of throughput and latency levels your application will require:

• Throughput - how many requests per time interval the system can handle (usually measured in requests per second).

• Latency - how fast the application is able to return the first relevant point of data. It is unique for LLM based applications, especially chatbots, as the first relevant point is usually considered to be the first returned inference token, the following tokens can be generated and rendered faster than an average human can read. The important measure here is TTFT - time to first token. So in order to reduce TTFT, we aim to reduce the time of Prefill, the procedure includes all operations that LLM needs to be performed before starting to generate tokens.

Not surprisingly, throughput can be increased and decreased by throwing more wood to the fire - increasing the size of your GPU cluster which is being used to serve the models. In some cases it is enough to scale the cluster horizontally, where the same model would be deployed on separate machines to serve more traffic, in some cases there is a need to deploy the same model using more GPUs.

In any case, it is worth pre-calculating the amount of GPUs you will need and do some estimation of the traffic that needs to be supported throughout the time - less traffic means less throughput needed. Knowing your approximate traffic needs, you can implement Upscaling and Downscaling strategies that would allow managing the TCO more efficiently.

Additional advantage of running on AI Cloud is that you would be able to control the throughput and latency to a granular level. This also means that in most cases, with enough tuning, you would be able to achieve lower latencies compared to when running with proprietary model API. Also, you can deploy other relevant services closer to models being served, reducing the end-to-end latency even more as the network traffic can be contained within a single region.

An interesting stage in the process of building LLM apps is prototyping, this would mean no need to handle user traffic. While you can easily scale down to zero usage with proprietary APIs, you can also do that in AI Clouds by scaling the cluster size to zero during the periods of non-activity.

TOC - Proprietary model APIs vs. AI Clouds

Let’s see how the cost curves of Proprietary model APIs and AI Clouds could potentially look when overlayed on top of each other.

The math is obvious. You are better off with an AI Cloud in green areas and you are better of with proprietary LLM API providers in the red areas. If you consider using an AI Cloud your goal should be to stay in the green as much as possible (when it comes to optimising TOC).

The truth of the matter is that if you are running a LLM application at scale with high throughput requirements, it is not hard to have your costs lower when using AI Cloud compare to a proprietary API.

To summarise.

• If you are pre-training or finetuning, AI Cloud is your best friend.

• If you are just prototyping a LLM based application it might be a good idea to start off with proprietary model APIs to prove the viability of you application in the first place.

• After proving the case, oftentimes the next step is to cut the costs. It might involve switching to AI Cloud and deploying your own open source models. It also gives you higher control and additional capabilities to tune the system.

• Understanding your TOC is extremely important when choosing where to get your LLM inference tokens from.

Now, let’s go build a chatbot and run it on an Open Source LLM deployed on Nebius.

A hands-on project. Let’s go build!

If you haven’t yet, be sure to register to Nebius Cloud so you can follow along:

Create an account

The end goal of the project is the following:

• A Kubernetes cluster running on Nebius Cloud with a GPU compute worker node. We will use NVIDIA® H100 Tensor Core GPU instance that is now available for 1.5 USD/hour.

• Deploy a vLLM server with Mistral-7b-instruct LLM as a pod. We will use the previously deployed K8s server for this.

• Write a very simple chatbot with Streamlit that will provide an easy interface to test the model.

• Deploy the chatbot as a container in Kubernetes.

• Expose it via a LoadBalancer service so that you can access it through the internet.

Step 1: Deploy the Kubernetes cluster.

Before you can start the deployment, you will need to install Nebius CLI tool. If you already have your account set up it should be straightforward.

Assuming that you are running on MacOS, run:

brew install jq
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo chown root: /usr/local/bin/kubectl
curl -sSL https://storage.ai.nebius.cloud/nebius/install.sh | bash

This will set you up with kubectl tool for communication with Kubernetes cluster and nebius CLI tool for authentication with. After this run:

nebius profile create

It will prompt you for:

• Name - enter any.

• Api endpoint - leave the default api.eu.nebius.cloud.

• Authorisation type - choose federation.

After the above you will be redirected to the browser window where you will be authenticated. You now have your Nebius CLI tool set up.

If anything fails at this point, refer to official Nebius documentation here: Link.

Disclaimer: to follow the following tutorial you will need to have your billing set up and as mentioned before, a single NVIDIA® H100 Tensor Core GPU node would cost 1.5 USD/hour, there is a small charge for the public IP addresses assigned in steps 3 and 4 of the project. Once done, be sure to clean up your resources to not incure additional charges.

For simplicity reasons, we will perform the deployment via the Nebius Cloud UI, here is how you can do it. It is easy to figure it out, but let’s run step-by-step so that there are no unanswered questions.

• Once you log in to the console, click on the “Managed Kubernetes” tab on the left and click “+ Create cluster” top right

• Once in the cluster creation section:

  • Provide the cluster name.

  • We switch off the Control plane high availability off as there is no need for it in a demo project, be sure to always have it turned on for production use cases.

  • Let’s have the Public endpoint on as it will make configuration of kubectl easier for this example.

  • Press “Create cluster” once configuration is complete.

• You will see a new cluster being provisioned in the “Managed Kubernetes” overview tab. Click on it.