The Agentic AI Infrastructure Stack (2026): A Practical Guide
Explainer · · Published July 2026 · 8 min read
By Ashwin Gopinath, CEO
Every engineering team building with autonomous agents eventually hits the same wall: the model isn't the hard part anymore. The hard part is everything around the model, the orchestration, the tools, the memory, the retrieval, the execution sandbox, and the governance layer that keeps the whole system auditable. Sentra, the company brain, occupies the organizational memory tier of the stack mapped below.
In 2026, the agentic AI infrastructure stack has settled into eight distinct layers. This guide breaks down each one, names the tools engineering leads are actually shipping with, and spends extended time on the layer most teams get wrong: memory.
The Agentic AI Stack at a Glance
| Layer | Purpose | Leading Tools |
|---|---|---|
| Foundation Models | Reasoning and generation engine | GPT-5, Claude Opus/Sonnet, Gemini 2.5, Llama 4, DeepSeek; gateways such as LiteLLM, Portkey and Cloudflare AI Gateway |
| Orchestration | Agent workflow, state, and control flow | LangGraph, CrewAI, Temporal, AutoGen, OpenAI Agents SDK, Microsoft Agent Framework |
| Tool Layer | Standardized model-to-tool communication | MCP (Model Context Protocol); A2A for agent-to-agent calls |
| Memory | Persistent context across sessions and agents | Sentra, Mem0, Zep/Graphiti |
| Retrieval | Fetching relevant knowledge on demand | Pinecone, Weaviate, pgvector, Qdrant, Milvus, Chroma, Sentra retrieval API |
| Execution | Sandboxed runtime for agent actions | Kubernetes, Docker, Modal, E2B, Amazon Bedrock AgentCore Runtime |
| Observability | Tracing, evals, and debugging | LangSmith, Arize, Langfuse, Arize Phoenix, OpenTelemetry |
| Security | Access control, auditability, compliance | Sentra governance layer, OPA, Vault, NVIDIA NeMo Guardrails |
This isn't a theoretical stack. It's the composition pattern showing up in production deployments at companies running agents against real customer data, real infrastructure, and real financial consequences. Let's go layer by layer.
Layer 1: Foundation Models
The model layer gets the most attention and, increasingly, matters the least for differentiation. GPT-5, Claude Opus 4, and Gemini 2.5 are all capable of strong tool use and multi-step reasoning out of the box. The real question for infrastructure teams isn't "which model" but "which model for which task at which cost."
Most production stacks in 2026 run a router: a fast, cheap model (GPT-5 mini, Claude Haiku) for routine tool calls and a frontier model for complex reasoning or planning steps. This matters because token cost compounds fast once agents run in loops, a topic we cover in depth in ai-agent-memory-token-cost-roi.
Layer 2: Orchestration
Orchestration frameworks manage agent state, control flow, and multi-agent coordination. This is where you decide how agents plan, branch, retry, and hand off work.
| Tool | Why It Matters |
|---|---|
| LangGraph | Graph-based state machine model, strong for complex branching agent workflows, widely adopted |
| CrewAI | Role-based multi-agent coordination, faster to prototype with |
| Temporal | Durable execution for long-running, failure-resistant workflows |
| AutoGen | Conversation-driven multi-agent patterns, strong for research use cases |
LangGraph has become the default choice for teams building anything beyond a single-agent loop, mainly because its explicit state graph makes debugging non-linear agent behavior tractable. But orchestration frameworks are only as good as the context they can access at each node, which is why the next two layers matter more than most teams initially assume.
Layer 3: Tool Layer (MCP)
The Model Context Protocol (MCP) has become the standard for connecting agents to external tools, APIs, and data sources. Before MCP, every agent framework had its own bespoke tool-calling format, which meant every integration was a one-off. MCP standardizes this into a client-server model that any compliant agent can speak to.
Why it matters
MCP decouples tool development from agent development. Your team can build an MCP server once for your internal ticketing system, your CRM, or your codebase, and any MCP-compliant agent, regardless of which orchestration framework or model it runs on, can use it. This is the same architectural bet that made REST and gRPC durable choices for service-to-service communication. It's infrastructure, not a feature.Layer 4: Memory (The Layer Everyone Underbuilds)
This is where most agentic stacks quietly fail, and it deserves its own full section below.
Layer 5: Retrieval
Retrieval is how agents pull relevant knowledge into context at query time. Vector databases (Pinecone, Weaviate, pgvector) remain the default for unstructured document search. But retrieval and memory are not the same thing, and conflating them is one of the most common architectural mistakes we see.
Retrieval answers "what documents are relevant to this query." Memory answers "what does this agent (or team) already know, and how has that knowledge changed over time." Vector search has no concept of time, contradiction, or ownership. A knowledge graph does. We break this distinction down fully in rag-vs-knowledge-graph.
Layer 6: Execution
Execution is the sandboxed environment where agents actually take action, running code, calling APIs, modifying files, or provisioning infrastructure.
| Tool | Why It Matters |
|---|---|
| Kubernetes | Industry standard for scalable, isolated agent execution environments |
| Docker | Lightweight containerization for single-agent sandboxes |
| Modal | Serverless compute optimized for AI workloads, fast cold starts |
| E2B | Purpose-built sandboxes for AI code execution |
Enterprise teams increasingly run agents in Kubernetes-managed pods with strict network policies, because agent actions carry real blast radius. An agent with write access to a production database needs the same isolation discipline as any other privileged service.
Layer 7: Observability
You cannot debug what you cannot trace. Observability tools capture the full chain of an agent's reasoning, tool calls, and outputs so engineering teams can diagnose failures and drift.
| Tool | Why It Matters |
|---|---|
| LangSmith | Deep integration with LangGraph, strong tracing for agent chains |
| Arize | Production-grade eval and drift monitoring |
| Langfuse | Open-source, self-hostable tracing and analytics |
Layer 8: Security
Security and governance sit at the top of the stack because they need visibility into every layer beneath them. Access control, audit trails, and compliance certifications aren't optional for any team deploying agents against real customer or financial data. We'll return to this in the enterprise comparison below.
Memory: The Least Mature, Highest-Leverage Layer
Of all eight layers, memory is the least mature and the one with the highest leverage on system quality. Most teams treat memory as an afterthought, bolting on a vector store or a per-agent memory library after the orchestration and tool layers are already built. This is backwards.
Here's why. An agent without persistent, structured memory re-derives context every session. It re-reads documents it has already processed, re-asks questions it has already answered, and forgets decisions your team made a week ago. This isn't a minor inefficiency, it's the reason agentic systems plateau in production even when the underlying model is state-of-the-art. We cover the mechanics of this failure mode in why-ai-agents-forget.
Why Per-Agent Memory Falls Short
Tools like Mem0 and Zep (built on Graphiti) were early, useful attempts to give individual agents persistent memory. But they were designed around a single-agent assumption, and that assumption breaks down fast in enterprise settings.
Three specific failures show up repeatedly:
No shared context across agents. Per-agent memory tools scope memory to a single agent or a single user session. If your support agent learns that a customer's contract was amended last month, your billing agent has no way to know that. Each agent builds its own isolated silo, and teams end up re-solving the same context problem N times for N agents.
No temporal tracking. Most memory tools capture facts, but not when those facts became true or when they stopped being true. This sounds like a minor gap until an agent acts on stale information with full confidence, because nothing in its memory layer flagged that the fact had expired. Bi-temporal tracking, knowing both the real-world time a fact was true and the system time it was recorded, is the difference between a memory layer you can trust and one you have to double-check. We go deep on this concept in what-is-a-bitemporal-knowledge-graph.
No governance. Per-agent memory tools were built for individual developers experimenting with agents, not for enterprises with compliance obligations. There's typically no audit trail, no access control model, and no path to SOC 2 or ISO 27001 certification. That's a blocker the moment an agent touches regulated data.
Sentra's Approach: Write-Time, Bi-Temporal, Shared
Sentra is built as an org-wide memory layer, not a per-agent one. It sits underneath every agent and every team, maintaining a single shared knowledge graph that both humans and agents read from and write to.
Two design decisions set it apart:
Write-time comprehension. Most memory systems are query-time: they wait until an agent asks a question, then retrieve and reason over whatever's available. Sentra captures decisions, commitments, and drift as they happen, at write time, not as a reactive lookup. This means the graph is already structured and current by the time any agent queries it, instead of forcing every query to re-derive meaning from raw text.
Bi-temporal by default. Every fact in Sentra's graph carries two timestamps: when it became true in the real world, and when the system recorded it. This lets agents (and humans) reason correctly about what was known at any point in time, and lets the graph handle contradictions and corrections without losing history.
Because the graph is shared, a decision captured from a Slack thread, a support ticket resolution, or a product spec update becomes available to every agent and every team member querying that domain, immediately, with governance controls attached.
Comparison: Sentra vs Mem0 vs Zep/Graphiti
| Dimension | Sentra | Mem0 | Zep / Graphiti |
|---|---|---|---|
| Memory scope | Org-wide, shared across teams and agents | Per-agent / per-user | Per-agent / per-session |
| Write-time vs query-time | Write-time comprehension | Query-time retrieval | Query-time retrieval |
| Temporal tracking | Bi-temporal (valid time + transaction time) | None | Partial (event time only) |
| Multi-agent sharing | Native, single shared graph | Not supported | Limited |
| Enterprise governance | SOC 2, ISO 27001, self-hosting | Not available | Not available |
| Token efficiency | 72.6% lower model cost, 88.31% on Terminal-Bench 2.1 | Baseline | Baseline |
What Changed in the Stack in the Second Half of 2026
The eight layers held, but each one gained tools engineering teams now shortlist. Every description below is taken from the vendor's or project's own site, documentation or repository, checked on 5 November 2026.
Agent-to-agent protocols
MCP connects an agent to tools. The Agent2Agent (A2A) protocol connects agents to each other. Google created it and donated it to the Linux Foundation, where it is Apache 2.0 licensed and run by a Technical Steering Committee. Microsoft Foundry Agent Service lists A2A support (learn.microsoft.com). Most production stacks now speak both: MCP for tools and data, A2A for handing work between agents built by different teams or vendors.
Model gateways
A gateway sits in front of the foundation model layer and gives one place for routing, fallbacks, logging and cost limits.
| Tool | What it is |
|---|---|
| LiteLLM | Open-source SDK and proxy that calls 100+ model providers in the OpenAI format, self-hosted (github.com) |
| Portkey | Open-source AI gateway with guardrails and routing, self-hosted or managed (github.com) |
| Cloudflare AI Gateway | Hosted gateway that adds analytics, logging, caching, rate limiting, retries and fallbacks (developers.cloudflare.com) |
Orchestration frameworks
| Tool | What it is |
|---|---|
| OpenAI Agents SDK | Open-source multi-agent framework with handoffs, guardrails, sessions, tracing and MCP tools (github.com) |
| Microsoft Agent Framework | Open-source framework for .NET and Python that Microsoft calls the direct successor to Semantic Kernel and AutoGen (learn.microsoft.com) |
| Claude Agent SDK | Python and TypeScript library that embeds Claude Code's agent loop, with tools, hooks, subagents, MCP and permissions (code.claude.com) |
Teams building on AutoGen should note that the Microsoft Agent Framework is where that line continues.
Managed agent platforms
The three large clouds now sell the runtime, identity, memory and observability layers as one managed service.
| Platform | What it includes |
|---|---|
| Amazon Bedrock AgentCore | Runtime, Memory, Gateway, Identity, Code Interpreter, Browser, Observability, Evaluations, Policy and Registry, for any framework and model (docs.aws.amazon.com) |
| Google Gemini Enterprise Agent Platform | The evolution of Vertex AI, with ADK, Agent Runtime (formerly Agent Engine), Agent Identity, Agent Gateway, Agent Registry and Memory Bank (docs.cloud.google.com) |
| Microsoft Foundry Agent Service | Managed building, deployment and scaling of agents, with Entra identity, observability and A2A (learn.microsoft.com) |
Each cloud's memory service scopes memory to a user, session or agent inside that cloud. Sentra serves one company memory to agents on all three, and to agents outside them, over MCP and REST.
Retrieval, guardrails and observability
- Vector databases. Qdrant (Apache 2.0, written in Rust), Milvus (Apache 2.0, LF AI & Data Foundation) and Chroma (Apache 2.0) are open source with managed cloud editions, alongside Pinecone, Weaviate and pgvector.
- Guardrails. NVIDIA NeMo Guardrails is an open-source toolkit that adds input, dialog, retrieval, execution and output rails to LLM applications.
- Observability. Arize Phoenix is a self-hosted tracing and evaluation tool built on OpenTelemetry. The OpenTelemetry GenAI semantic conventions, which cover model spans, metrics and MCP, are still marked as in development, so expect attribute names to change.
- Reference architectures. AWS released a Well-Architected Agentic AI Lens on 10 June 2026, covering agentic systems across the six Well-Architected pillars.
None of these additions closes the gap in the memory layer. Gateways, frameworks and managed runtimes move the same raw context faster. The shared memory every agent reads still has to know what is current, who owns what and who may see it. Sentra keeps the source and time on every fact (how Sentra keeps company memory current) and checks each source's permissions on every call (how Sentra enforces permissions on every fact).
Reference Architecture: Where Sentra Sits in the Stack
+-----------------------------------------------------------+ | Foundation Models | | (GPT-5, Claude, Gemini, Llama, etc.) | +-----------------------------------------------------------+ | +-----------------------------------------------------------+ | Orchestration | | (LangGraph, CrewAI, Temporal) | +-----------------------------------------------------------+ | +-----------------------------------------------------------+ | Tool Layer (MCP) | +-----------------------------------------------------------+ | +-------------------------------------------+ | SENTRA | | Org-wide, bi-temporal memory graph | | (write-time capture, shared by teams | | and agents, governed access) | +-------------------------------------------+ | +-----------------------------------------------------------+ | Retrieval | | (Vector DBs, Sentra retrieval API) | +-----------------------------------------------------------+ | +-----------------------------------------------------------+ | Execution | | (Kubernetes, Docker, Modal) | +-----------------------------------------------------------+ | +-----------------------------------------------------------+ | Observability | Security | | (LangSmith, Arize, Langfuse) | (Sentra governance, OPA) | +-----------------------------------------------------------+
Sentra sits between the tool layer and retrieval, feeding structured, governed, temporally-aware context into every agent above it while also serving as the durable record teams query directly. It's not a bolt-on cache. It's the substrate the rest of the stack reads from.
Infrastructure Choices by Company Size
Not every team needs the full enterprise stack on day one. Here's what we recommend based on stage.
| Layer | Startup | Growth | Enterprise |
|---|---|---|---|
| Foundation Models | Single model (GPT-5 mini or Claude Haiku) | Model router (mini + frontier) | Multi-model router with fallback and cost governance |
| Orchestration | CrewAI or lightweight LangGraph | LangGraph | LangGraph + Temporal for durability |
| Tool Layer | Direct API calls | MCP for key integrations | Full MCP server catalog, internally governed |
| Memory | Sentra (self-serve) | Sentra | Sentra with self-hosting, SOC 2/ISO 27001 controls |
| Retrieval | pgvector | Pinecone or Weaviate | Pinecone/Weaviate + Sentra retrieval API |
| Execution | Docker | Modal or E2B | Kubernetes with network policy enforcement |
| Observability | Langfuse (open-source) | LangSmith | LangSmith or Arize with custom eval pipelines |
| Security | Basic API key management | Role-based access control | Full audit trail, SOC 2/ISO 27001, Vault-managed secrets |
The pattern across all three tiers
memory is the one layer where we recommend the same tool regardless of company size. Startups need write-time, shared memory just as much as enterprises do, because the cost of rebuilding context from scratch scales with agent usage, not company headcount. The difference at enterprise scale is governance and self-hosting requirements, not the underlying architecture.Building the Stack That Doesn't Plateau
Most agentic AI systems don't fail because the model is weak. They fail because the infrastructure around the model can't retain, share, or govern what the system has learned. Orchestration and tool layers have matured quickly because they map cleanly onto existing software engineering patterns. Memory hasn't, because it requires rethinking what "context" means for a system with many agents, many teams, and a timeline that keeps moving.
If you're evaluating infrastructure for agentic systems in 2026, treat memory as a first-class architectural decision, not a library you add in month three. Get the foundation models, orchestration, and tool layers right, but build on a memory layer that's shared, temporally aware, and governed from day one.
To go deeper on the specific failure modes and mechanics behind these design decisions, see why-ai-agents-forget, rag-vs-knowledge-graph, ai-agent-memory-token-cost-roi, and what-is-a-bitemporal-knowledge-graph.
FAQ
What are the layers of the agentic AI infrastructure stack?
Eight: foundation models, orchestration, the tool layer (MCP), memory, retrieval, execution, observability and security. Some guides merge them into five to seven, but the jobs are the same. Memory is the least mature of the eight and the one with the most effect on whether agents act on current information.
What separates agentic infrastructure from standard LLM serving?
LLM serving answers one request. Agentic infrastructure has to keep state across many steps, call tools with real permissions, run actions in a sandbox, remember what happened across sessions and log every step for audit. That adds the orchestration, memory, execution and governance layers a chat endpoint does not need.
What memory layer do agentic AI systems need?
One that is shared across agents, knows when facts changed, and respects who may see what. Per-agent memory libraries such as Mem0 and Zep suit one agent remembering one user. Sentra is the shared company memory every agent reads over MCP and REST, with the source and time on every fact and each source's permissions checked on every call.
What is the difference between MCP and A2A?
MCP connects an agent to tools and data sources. A2A, the Agent2Agent protocol now governed by the Linux Foundation, connects agents to other agents so they can hand off tasks. Most stacks use both.
Which orchestration framework should teams start with?
LangGraph for explicit, debuggable state graphs, CrewAI for fast role-based prototypes, Temporal when workflows must survive failures over hours or days, and the OpenAI Agents SDK or Microsoft Agent Framework when you are standardized on those platforms. Whatever you pick, give every node the same shared memory.
Do I need a managed agent platform such as Bedrock AgentCore?
Not to start. Managed platforms bundle runtime, identity, memory and observability for teams committed to one cloud. Teams running agents across several clouds or vendors usually keep the framework portable and put a shared memory layer underneath that every runtime can read.
How do you avoid vendor lock-in in an agent stack?
Keep the interfaces standard: MCP for tools, A2A between agents, a gateway in front of models, and OpenTelemetry for traces. Keep company memory in one layer that any agent or model can read, so switching a model or framework does not mean rebuilding what agents know.
What security standards should teams review before shipping agents?
The OWASP Top 10 for LLM Applications and the OWASP Top 10 for Agentic Applications, the NIST AI Risk Management Framework, ISO/IEC 42001 and, on AWS, the Well-Architected Agentic AI Lens. In practice, the controls that matter most are delegated identity, source permissions on every fact, approval for consequential actions and an audit trail.