Back

The Agentic AI Infrastructure Stack (2026): A Practical Guide

Explainer · · Published July 2026 · 8 min read

By Ashwin Gopinath, CEO

Every engineering team building with autonomous agents eventually hits the same wall: the model isn't the hard part anymore. The hard part is everything around the model, the orchestration, the tools, the memory, the retrieval, the execution sandbox, and the governance layer that keeps the whole system auditable. Sentra, the company brain, occupies the organizational memory tier of the stack mapped below.

In 2026, the agentic AI infrastructure stack has settled into eight distinct layers. This guide breaks down each one, names the tools engineering leads are actually shipping with, and spends extended time on the layer most teams get wrong: memory.

The Agentic AI Stack at a Glance

LayerPurposeLeading Tools
Foundation ModelsReasoning and generation engineGPT-5, Claude Opus/Sonnet, Gemini 2.5, Llama 4, DeepSeek; gateways such as LiteLLM, Portkey and Cloudflare AI Gateway
OrchestrationAgent workflow, state, and control flowLangGraph, CrewAI, Temporal, AutoGen, OpenAI Agents SDK, Microsoft Agent Framework
Tool LayerStandardized model-to-tool communicationMCP (Model Context Protocol); A2A for agent-to-agent calls
MemoryPersistent context across sessions and agentsSentra, Mem0, Zep/Graphiti
RetrievalFetching relevant knowledge on demandPinecone, Weaviate, pgvector, Qdrant, Milvus, Chroma, Sentra retrieval API
ExecutionSandboxed runtime for agent actionsKubernetes, Docker, Modal, E2B, Amazon Bedrock AgentCore Runtime
ObservabilityTracing, evals, and debuggingLangSmith, Arize, Langfuse, Arize Phoenix, OpenTelemetry
SecurityAccess control, auditability, complianceSentra governance layer, OPA, Vault, NVIDIA NeMo Guardrails

This isn't a theoretical stack. It's the composition pattern showing up in production deployments at companies running agents against real customer data, real infrastructure, and real financial consequences. Let's go layer by layer.

Layer 1: Foundation Models

The model layer gets the most attention and, increasingly, matters the least for differentiation. GPT-5, Claude Opus 4, and Gemini 2.5 are all capable of strong tool use and multi-step reasoning out of the box. The real question for infrastructure teams isn't "which model" but "which model for which task at which cost."

Most production stacks in 2026 run a router: a fast, cheap model (GPT-5 mini, Claude Haiku) for routine tool calls and a frontier model for complex reasoning or planning steps. This matters because token cost compounds fast once agents run in loops, a topic we cover in depth in ai-agent-memory-token-cost-roi.

Layer 2: Orchestration

Orchestration frameworks manage agent state, control flow, and multi-agent coordination. This is where you decide how agents plan, branch, retry, and hand off work.

ToolWhy It Matters
LangGraphGraph-based state machine model, strong for complex branching agent workflows, widely adopted
CrewAIRole-based multi-agent coordination, faster to prototype with
TemporalDurable execution for long-running, failure-resistant workflows
AutoGenConversation-driven multi-agent patterns, strong for research use cases

LangGraph has become the default choice for teams building anything beyond a single-agent loop, mainly because its explicit state graph makes debugging non-linear agent behavior tractable. But orchestration frameworks are only as good as the context they can access at each node, which is why the next two layers matter more than most teams initially assume.

Layer 3: Tool Layer (MCP)

The Model Context Protocol (MCP) has become the standard for connecting agents to external tools, APIs, and data sources. Before MCP, every agent framework had its own bespoke tool-calling format, which meant every integration was a one-off. MCP standardizes this into a client-server model that any compliant agent can speak to.

Why it matters

MCP decouples tool development from agent development. Your team can build an MCP server once for your internal ticketing system, your CRM, or your codebase, and any MCP-compliant agent, regardless of which orchestration framework or model it runs on, can use it. This is the same architectural bet that made REST and gRPC durable choices for service-to-service communication. It's infrastructure, not a feature.

Layer 4: Memory (The Layer Everyone Underbuilds)

This is where most agentic stacks quietly fail, and it deserves its own full section below.

Layer 5: Retrieval

Retrieval is how agents pull relevant knowledge into context at query time. Vector databases (Pinecone, Weaviate, pgvector) remain the default for unstructured document search. But retrieval and memory are not the same thing, and conflating them is one of the most common architectural mistakes we see.

Retrieval answers "what documents are relevant to this query." Memory answers "what does this agent (or team) already know, and how has that knowledge changed over time." Vector search has no concept of time, contradiction, or ownership. A knowledge graph does. We break this distinction down fully in rag-vs-knowledge-graph.

Layer 6: Execution

Execution is the sandboxed environment where agents actually take action, running code, calling APIs, modifying files, or provisioning infrastructure.

ToolWhy It Matters
KubernetesIndustry standard for scalable, isolated agent execution environments
DockerLightweight containerization for single-agent sandboxes
ModalServerless compute optimized for AI workloads, fast cold starts
E2BPurpose-built sandboxes for AI code execution

Enterprise teams increasingly run agents in Kubernetes-managed pods with strict network policies, because agent actions carry real blast radius. An agent with write access to a production database needs the same isolation discipline as any other privileged service.

Layer 7: Observability

You cannot debug what you cannot trace. Observability tools capture the full chain of an agent's reasoning, tool calls, and outputs so engineering teams can diagnose failures and drift.

ToolWhy It Matters
LangSmithDeep integration with LangGraph, strong tracing for agent chains
ArizeProduction-grade eval and drift monitoring
LangfuseOpen-source, self-hostable tracing and analytics

Layer 8: Security

Security and governance sit at the top of the stack because they need visibility into every layer beneath them. Access control, audit trails, and compliance certifications aren't optional for any team deploying agents against real customer or financial data. We'll return to this in the enterprise comparison below.

Memory: The Least Mature, Highest-Leverage Layer

Of all eight layers, memory is the least mature and the one with the highest leverage on system quality. Most teams treat memory as an afterthought, bolting on a vector store or a per-agent memory library after the orchestration and tool layers are already built. This is backwards.

Here's why. An agent without persistent, structured memory re-derives context every session. It re-reads documents it has already processed, re-asks questions it has already answered, and forgets decisions your team made a week ago. This isn't a minor inefficiency, it's the reason agentic systems plateau in production even when the underlying model is state-of-the-art. We cover the mechanics of this failure mode in why-ai-agents-forget.

Why Per-Agent Memory Falls Short

Tools like Mem0 and Zep (built on Graphiti) were early, useful attempts to give individual agents persistent memory. But they were designed around a single-agent assumption, and that assumption breaks down fast in enterprise settings.

Three specific failures show up repeatedly:

No shared context across agents. Per-agent memory tools scope memory to a single agent or a single user session. If your support agent learns that a customer's contract was amended last month, your billing agent has no way to know that. Each agent builds its own isolated silo, and teams end up re-solving the same context problem N times for N agents.

No temporal tracking. Most memory tools capture facts, but not when those facts became true or when they stopped being true. This sounds like a minor gap until an agent acts on stale information with full confidence, because nothing in its memory layer flagged that the fact had expired. Bi-temporal tracking, knowing both the real-world time a fact was true and the system time it was recorded, is the difference between a memory layer you can trust and one you have to double-check. We go deep on this concept in what-is-a-bitemporal-knowledge-graph.

No governance. Per-agent memory tools were built for individual developers experimenting with agents, not for enterprises with compliance obligations. There's typically no audit trail, no access control model, and no path to SOC 2 or ISO 27001 certification. That's a blocker the moment an agent touches regulated data.

Sentra's Approach: Write-Time, Bi-Temporal, Shared

Sentra is built as an org-wide memory layer, not a per-agent one. It sits underneath every agent and every team, maintaining a single shared knowledge graph that both humans and agents read from and write to.

Two design decisions set it apart:

Write-time comprehension. Most memory systems are query-time: they wait until an agent asks a question, then retrieve and reason over whatever's available. Sentra captures decisions, commitments, and drift as they happen, at write time, not as a reactive lookup. This means the graph is already structured and current by the time any agent queries it, instead of forcing every query to re-derive meaning from raw text.

Bi-temporal by default. Every fact in Sentra's graph carries two timestamps: when it became true in the real world, and when the system recorded it. This lets agents (and humans) reason correctly about what was known at any point in time, and lets the graph handle contradictions and corrections without losing history.

Because the graph is shared, a decision captured from a Slack thread, a support ticket resolution, or a product spec update becomes available to every agent and every team member querying that domain, immediately, with governance controls attached.

Comparison: Sentra vs Mem0 vs Zep/Graphiti

DimensionSentraMem0Zep / Graphiti
Memory scopeOrg-wide, shared across teams and agentsPer-agent / per-userPer-agent / per-session
Write-time vs query-timeWrite-time comprehensionQuery-time retrievalQuery-time retrieval
Temporal trackingBi-temporal (valid time + transaction time)NonePartial (event time only)
Multi-agent sharingNative, single shared graphNot supportedLimited
Enterprise governanceSOC 2, ISO 27001, self-hostingNot availableNot available
Token efficiency72.6% lower model cost, 88.31% on Terminal-Bench 2.1BaselineBaseline

What Changed in the Stack in the Second Half of 2026

The eight layers held, but each one gained tools engineering teams now shortlist. Every description below is taken from the vendor's or project's own site, documentation or repository, checked on 5 November 2026.

Agent-to-agent protocols

MCP connects an agent to tools. The Agent2Agent (A2A) protocol connects agents to each other. Google created it and donated it to the Linux Foundation, where it is Apache 2.0 licensed and run by a Technical Steering Committee. Microsoft Foundry Agent Service lists A2A support (learn.microsoft.com). Most production stacks now speak both: MCP for tools and data, A2A for handing work between agents built by different teams or vendors.

Model gateways

A gateway sits in front of the foundation model layer and gives one place for routing, fallbacks, logging and cost limits.

ToolWhat it is
LiteLLMOpen-source SDK and proxy that calls 100+ model providers in the OpenAI format, self-hosted (github.com)
PortkeyOpen-source AI gateway with guardrails and routing, self-hosted or managed (github.com)
Cloudflare AI GatewayHosted gateway that adds analytics, logging, caching, rate limiting, retries and fallbacks (developers.cloudflare.com)

Orchestration frameworks

ToolWhat it is
OpenAI Agents SDKOpen-source multi-agent framework with handoffs, guardrails, sessions, tracing and MCP tools (github.com)
Microsoft Agent FrameworkOpen-source framework for .NET and Python that Microsoft calls the direct successor to Semantic Kernel and AutoGen (learn.microsoft.com)
Claude Agent SDKPython and TypeScript library that embeds Claude Code's agent loop, with tools, hooks, subagents, MCP and permissions (code.claude.com)

Teams building on AutoGen should note that the Microsoft Agent Framework is where that line continues.

Managed agent platforms

The three large clouds now sell the runtime, identity, memory and observability layers as one managed service.

PlatformWhat it includes
Amazon Bedrock AgentCoreRuntime, Memory, Gateway, Identity, Code Interpreter, Browser, Observability, Evaluations, Policy and Registry, for any framework and model (docs.aws.amazon.com)
Google Gemini Enterprise Agent PlatformThe evolution of Vertex AI, with ADK, Agent Runtime (formerly Agent Engine), Agent Identity, Agent Gateway, Agent Registry and Memory Bank (docs.cloud.google.com)
Microsoft Foundry Agent ServiceManaged building, deployment and scaling of agents, with Entra identity, observability and A2A (learn.microsoft.com)

Each cloud's memory service scopes memory to a user, session or agent inside that cloud. Sentra serves one company memory to agents on all three, and to agents outside them, over MCP and REST.

Retrieval, guardrails and observability

  • Vector databases. Qdrant (Apache 2.0, written in Rust), Milvus (Apache 2.0, LF AI & Data Foundation) and Chroma (Apache 2.0) are open source with managed cloud editions, alongside Pinecone, Weaviate and pgvector.
  • Guardrails. NVIDIA NeMo Guardrails is an open-source toolkit that adds input, dialog, retrieval, execution and output rails to LLM applications.
  • Observability. Arize Phoenix is a self-hosted tracing and evaluation tool built on OpenTelemetry. The OpenTelemetry GenAI semantic conventions, which cover model spans, metrics and MCP, are still marked as in development, so expect attribute names to change.
  • Reference architectures. AWS released a Well-Architected Agentic AI Lens on 10 June 2026, covering agentic systems across the six Well-Architected pillars.

None of these additions closes the gap in the memory layer. Gateways, frameworks and managed runtimes move the same raw context faster. The shared memory every agent reads still has to know what is current, who owns what and who may see it. Sentra keeps the source and time on every fact (how Sentra keeps company memory current) and checks each source's permissions on every call (how Sentra enforces permissions on every fact).

Reference Architecture: Where Sentra Sits in the Stack

+-----------------------------------------------------------+ | Foundation Models | | (GPT-5, Claude, Gemini, Llama, etc.) | +-----------------------------------------------------------+ | +-----------------------------------------------------------+ | Orchestration | | (LangGraph, CrewAI, Temporal) | +-----------------------------------------------------------+ | +-----------------------------------------------------------+ | Tool Layer (MCP) | +-----------------------------------------------------------+ | +-------------------------------------------+ | SENTRA | | Org-wide, bi-temporal memory graph | | (write-time capture, shared by teams | | and agents, governed access) | +-------------------------------------------+ | +-----------------------------------------------------------+ | Retrieval | | (Vector DBs, Sentra retrieval API) | +-----------------------------------------------------------+ | +-----------------------------------------------------------+ | Execution | | (Kubernetes, Docker, Modal) | +-----------------------------------------------------------+ | +-----------------------------------------------------------+ | Observability | Security | | (LangSmith, Arize, Langfuse) | (Sentra governance, OPA) | +-----------------------------------------------------------+

Sentra sits between the tool layer and retrieval, feeding structured, governed, temporally-aware context into every agent above it while also serving as the durable record teams query directly. It's not a bolt-on cache. It's the substrate the rest of the stack reads from.

Infrastructure Choices by Company Size

Not every team needs the full enterprise stack on day one. Here's what we recommend based on stage.

LayerStartupGrowthEnterprise
Foundation ModelsSingle model (GPT-5 mini or Claude Haiku)Model router (mini + frontier)Multi-model router with fallback and cost governance
OrchestrationCrewAI or lightweight LangGraphLangGraphLangGraph + Temporal for durability
Tool LayerDirect API callsMCP for key integrationsFull MCP server catalog, internally governed
MemorySentra (self-serve)SentraSentra with self-hosting, SOC 2/ISO 27001 controls
RetrievalpgvectorPinecone or WeaviatePinecone/Weaviate + Sentra retrieval API
ExecutionDockerModal or E2BKubernetes with network policy enforcement
ObservabilityLangfuse (open-source)LangSmithLangSmith or Arize with custom eval pipelines
SecurityBasic API key managementRole-based access controlFull audit trail, SOC 2/ISO 27001, Vault-managed secrets

The pattern across all three tiers

memory is the one layer where we recommend the same tool regardless of company size. Startups need write-time, shared memory just as much as enterprises do, because the cost of rebuilding context from scratch scales with agent usage, not company headcount. The difference at enterprise scale is governance and self-hosting requirements, not the underlying architecture.

Building the Stack That Doesn't Plateau

Most agentic AI systems don't fail because the model is weak. They fail because the infrastructure around the model can't retain, share, or govern what the system has learned. Orchestration and tool layers have matured quickly because they map cleanly onto existing software engineering patterns. Memory hasn't, because it requires rethinking what "context" means for a system with many agents, many teams, and a timeline that keeps moving.

If you're evaluating infrastructure for agentic systems in 2026, treat memory as a first-class architectural decision, not a library you add in month three. Get the foundation models, orchestration, and tool layers right, but build on a memory layer that's shared, temporally aware, and governed from day one.

To go deeper on the specific failure modes and mechanics behind these design decisions, see why-ai-agents-forget, rag-vs-knowledge-graph, ai-agent-memory-token-cost-roi, and what-is-a-bitemporal-knowledge-graph.

FAQ

What are the layers of the agentic AI infrastructure stack?

Eight: foundation models, orchestration, the tool layer (MCP), memory, retrieval, execution, observability and security. Some guides merge them into five to seven, but the jobs are the same. Memory is the least mature of the eight and the one with the most effect on whether agents act on current information.

What separates agentic infrastructure from standard LLM serving?

LLM serving answers one request. Agentic infrastructure has to keep state across many steps, call tools with real permissions, run actions in a sandbox, remember what happened across sessions and log every step for audit. That adds the orchestration, memory, execution and governance layers a chat endpoint does not need.

What memory layer do agentic AI systems need?

One that is shared across agents, knows when facts changed, and respects who may see what. Per-agent memory libraries such as Mem0 and Zep suit one agent remembering one user. Sentra is the shared company memory every agent reads over MCP and REST, with the source and time on every fact and each source's permissions checked on every call.

What is the difference between MCP and A2A?

MCP connects an agent to tools and data sources. A2A, the Agent2Agent protocol now governed by the Linux Foundation, connects agents to other agents so they can hand off tasks. Most stacks use both.

Which orchestration framework should teams start with?

LangGraph for explicit, debuggable state graphs, CrewAI for fast role-based prototypes, Temporal when workflows must survive failures over hours or days, and the OpenAI Agents SDK or Microsoft Agent Framework when you are standardized on those platforms. Whatever you pick, give every node the same shared memory.

Do I need a managed agent platform such as Bedrock AgentCore?

Not to start. Managed platforms bundle runtime, identity, memory and observability for teams committed to one cloud. Teams running agents across several clouds or vendors usually keep the framework portable and put a shared memory layer underneath that every runtime can read.

How do you avoid vendor lock-in in an agent stack?

Keep the interfaces standard: MCP for tools, A2A between agents, a gateway in front of models, and OpenTelemetry for traces. Keep company memory in one layer that any agent or model can read, so switching a model or framework does not mean rebuilding what agents know.

What security standards should teams review before shipping agents?

The OWASP Top 10 for LLM Applications and the OWASP Top 10 for Agentic Applications, the NIST AI Risk Management Framework, ISO/IEC 42001 and, on AWS, the Well-Architected Agentic AI Lens. In practice, the controls that matter most are delegated identity, source permissions on every fact, approval for consequential actions and an audit trail.