Search ARuntime.com

Find runtime definitions and implementation guidance

Search page titles, summaries, headings, glossary terms, use cases, and runtime-directory entries.

Enter at least two characters.

Developer

Reference Architecture

Detailed AI runtime reference architecture with request gateway, identity, context providers, model router, inference adapters, tool broker, memory, policy, workflow, evaluation, telemetry, and deployment variants.

Audience: Platform architects; implementation teams; security and governance leaders Reading time: 15 minutes Status: Production guidance Last reviewed:

Key takeaways

  • The request boundary establishes identity, authority, risk, budget, and output contract before model work begins.
  • Control, context, execution, and trust are cross-cutting planes over the hardware-to-product stack.
  • Providers and tools are replaceable adapters behind versioned interfaces.
  • Durable workflow state is separate from model serving and from long-term memory.
  • Policy decisions and trace evidence cross every privileged boundary.
  • The same logical architecture can deploy in one process, distributed services, edge/cloud, or managed-provider combinations.

Runtime boundary

A useful architecture identifies what this layer receives, owns, emits, measures, and refuses to own. That boundary prevents overlapping products from being treated as interchangeable.

Receives

Product request, identity, deployment configuration, component catalogs, policies, model/tool/memory adapters, and operational constraints.

Owns

Logical component boundaries, interface responsibilities, state ownership, enforcement points, and replaceability criteria.

Emits

A component topology, interface contracts, data/control flows, failure domains, telemetry, and deployment mapping.

Does not own

One mandatory vendor implementation or a claim that every deployment needs every component as a service.

Failure modes

Shared mutable state, cross-plane coupling, leaky provider APIs, policy bypass, ambiguous ownership, trace gaps, and broad failure domains.

Evidence and metrics

Interface errors, dependency latency, policy coverage, trace completeness, component availability, recovery, and portability tests.

Concrete reference model

The layered stack describes where execution responsibilities sit. The planes describe how control, request work, data, and evidence move through those layers. They are introduced only where they clarify ownership and failure boundaries.

Control plane

Defines desired state and governs configuration, versioning, routing policy, capacity, deployment, admission, budgets, rollouts, and recovery decisions. It should not carry ordinary request payloads unless required for a bounded control decision.

Execution or runtime plane

Performs request-scoped work: inference, tool execution, workflow transitions, validation, side effects, and runtime state changes under the admitted contract.

Data plane

Carries model inputs and outputs, retrieved context, tensors, token streams, tool payloads, and domain records. Treat it as a separate plane when data movement, tenancy, egress, or locality materially affects the architecture.

Evidence or observability plane

Correlates infrastructure, model, tool, policy, evaluation, audit, and business-outcome records without assuming that raw prompts or sensitive payloads may be retained.

Cross-cutting concerns

  • Identity
  • Tenancy
  • Configuration
  • Secrets
  • Policy
  • Security
  • Observability
  • Evaluation
  • Cost controls
  • Data classification
  • Audit and evidence
  • Versioning
  • Failure recovery

Request lifecycle

  1. Request admission
  2. Identity and authority resolution
  3. Risk classification
  4. Context assembly
  5. Model-route selection
  6. Inference
  7. Tool planning
  8. Authorization
  9. Tool execution
  10. Validation
  11. Human approval where required
  12. Response finalization
  13. Memory decision
  14. Evidence persistence
  15. Evaluation
  16. Incident or recovery handling

Failure and recovery model

Recovery is part of the contract. A retry is safe only when the system can distinguish an absent effect from an unknown or committed effect.

Required handling for representative AI runtime failures
Failure Detection User-visible behavior Retry eligibility Idempotency Rollback or compensation Escalation Evidence
Model timeout Deadline or provider timeout event. Return a bounded retry, fallback, or explicit unavailable result; never imply completion. Retry only within deadline and budget, preferably on a route with known capacity. Reuse the request and correlation identifiers; do not repeat completed side effects. Usually none before tools execute; cancel partial streams and release reservations. Escalate repeated route or provider failures to runtime operations. Route, deployment version, elapsed time, timeout source, retry decision, and final status.
Model refusal Typed finish reason or policy/refusal classification. Expose a safe, scoped refusal or route to authorized human review. Do not blind-retry identical input; retry only after a permitted route or context change. Keep the same task identity and record each model attempt separately. None unless earlier tools produced provisional state. Escalate when the task is required and an approved review path exists. Model route, refusal category, policy basis, redacted context references, and reviewer decision.
Invalid structured output Schema validation, parser failure, or semantic invariant check. Do not execute dependent tools; return a repair attempt or explicit validation failure. Permit a bounded repair/retry with the same schema and stricter decoding constraints. No new side-effect key until output validates. Discard invalid provisional output. Escalate persistent schema drift to model-route and contract owners. Schema version, validation errors, model version, repair attempts, and final disposition.
Context retrieval failure Source timeout, authorization denial, empty required evidence, or provenance failure. State that required context is unavailable; do not answer as though retrieval succeeded. Retry transient sources within the deadline; use fallback sources only when policy allows. Reads should carry request/correlation identifiers and stable pagination tokens. Remove partial untrusted context from the active bundle. Escalate required-source outages or repeated tenant-boundary failures. Source, query class or hash, authorization result, retrieved references, and failure reason.
Tool timeout Tool-contract timeout or missing completion event. Report pending, failed, or unknown outcome according to side-effect semantics. Retry only when the tool contract declares idempotency or a reconciliation check proves no commit. Use a stable idempotency key scoped to tool, actor, tenant, and intended effect. Invoke declared compensation only after the committed outcome is known. Escalate unknown outcomes for irreversible or high-impact tools. Tool/version, arguments hash, idempotency key, timeout, reconciliation result, and side-effect state.
Tool authorization denial Deterministic policy decision at the tool boundary. Do not call the tool; return a denial reason suitable for the actor. Retry only after authority, scope, or approval changes. No side-effect key is consumed when execution never begins. None. Use the defined approval or access-review path; never let the model override the denial. Actor, tenant, resource, action, policy/version, reason codes, obligations, and enforcement point.
Partial side effect Tool result, reconciliation read, or downstream event indicates partial commit. Expose a recoverable incident state rather than success. Do not replay the full action until committed sub-steps are reconciled. Each sub-step needs an idempotency key or transaction reference. Run documented compensation in reverse dependency order when safe. Escalate irreversible or externally visible partial effects. Changed resources, before/after references, completed sub-steps, failed sub-step, and compensation state.
Duplicate execution Repeated idempotency key, duplicate workflow event, or reconciliation mismatch. Return the original result or a duplicate-suppressed status. Do not execute again unless the original attempt is conclusively absent or expired by contract. Persist keys and outcome references for the tool-defined retention window. Reverse an accidental duplicate only with domain-approved logic. Escalate repeated duplicate dispatch to workflow/runtime owners. Idempotency key, original execution reference, duplicate caller, decision, and retained outcome.
Model-route failure No eligible route, incompatible capability, capacity rejection, or unhealthy deployment. Use an approved fallback or return unavailable with the violated constraint. Re-evaluate route state within deadline; never silently relax data, region, quality, or safety constraints. Preserve request identity while recording each routing decision. Release capacity reservations and discard route-specific transient state. Escalate when no compliant route remains. Constraints, candidate routes, exclusion reasons, route-health snapshot, fallback, and final decision.
Policy violation Input, context, output, tool, data-egress, or retention policy decision. Block, redact, transform, or route to review according to policy obligations. Retry only after a policy-permitted change; never prompt the model to bypass enforcement. Record the same attempted action under the same request and a new decision event. Quarantine output and reverse any effect that escaped before detection. Escalate high-severity violations to security/governance owners. Policy/version, input references or hashes, decision, reason codes, obligations, enforcement, and remediation.
Budget exhaustion Token, cost, tool-call, step, time, or resource counter reaches a hard limit. Stop safely and return a partial result only when the output contract permits it. Requires a new authorized budget or a cheaper compliant route. Keep completed effects; do not re-run them when continuation resumes. Release reservations and close unnecessary sessions. Escalate repeated exhaustion to workload and capacity owners. Budget contract, counters, route/tool usage, stop point, partial artifacts, and continuation state.
Human-review timeout Approval deadline expires without an eligible decision. Expire or pause the action; do not infer approval from silence. Create a new review only if the proposed action remains valid and policy allows. Approval tokens bind to one action version and expire explicitly. Release holds or cancel provisional operations. Escalate through the documented review hierarchy. Proposal version, reviewers, notifications, deadline, decisions, expiry, and disposition.
Memory-write failure Storage error, policy denial, conflict, validation failure, or retention rejection. Return the task result with an explicit memory-not-persisted state when safe. Retry transient writes using optimistic concurrency and the same mutation identifier. Memory mutations need stable IDs, expected versions, and deduplication. Remove provisional indexes or pointers that refer to an absent canonical write. Escalate repeated conflicts, poisoning signals, or retention violations. Memory scope, mutation hash, policy, version expectation, storage result, and retry/cleanup outcome.
Trace persistence failure Exporter, durable outbox, audit sink, or correlation validation failure. Continue only when the missing record is non-mandatory; fail closed for explicitly required audit evidence. Buffer and retry with bounded retention and backpressure. Trace/event IDs prevent duplicate records at the sink. No business rollback unless mandatory evidence is part of the transaction. Alert observability/security owners before buffers overflow. Dropped or buffered event classes, exporter state, mandatory-evidence rule, and recovery result.
Dependency outage Health checks, circuit breakers, error rates, or dependency-specific status. Use an approved degraded mode or return unavailable with affected capabilities. Use bounded exponential backoff and circuit state; avoid synchronized retry storms. Preserve request and operation IDs across continuation. Cancel reservations and reconcile in-flight effects when the dependency returns. Escalate according to service criticality and recovery objective. Dependency/version, outage interval, circuit state, degraded path, affected requests, and reconciliation.

Architecture diagrams

Each diagram has an equivalent text description and remains readable in light, dark, print, keyboard, and no-JavaScript contexts.

Control plane and execution plane

Control plane and execution planeTwo large horizontal regions are connected by versioned configuration. The control plane contains model catalog, policies, routes, deployment configuration, evaluation gates, and rollout controls. The execution plane contains admission, context assembly, inference, tool execution, validation, and response handling. An evidence plane receives telemetry and decisions from both and returns evaluation findings to the control plane. A data plane contains model artifacts, retrieved context, caches, and business records, with labeled boundaries rather than unrestricted access.

Control planeCatalogs · policy · routes · deployments · evaluation gatesExecution planeAdmission · context · inference · tools · validation · responseData planeArtifacts · context · caches · recordsEvidence planeTraces · decisions · outcomes · evaluationsversioned desired state

The control plane defines desired state; the execution plane handles live requests under those versioned decisions.
Text description

Two large horizontal regions are connected by versioned configuration. The control plane contains model catalog, policies, routes, deployment configuration, evaluation gates, and rollout controls. The execution plane contains admission, context assembly, inference, tool execution, validation, and response handling. An evidence plane receives telemetry and decisions from both and returns evaluation findings to the control plane. A data plane contains model artifacts, retrieved context, caches, and business records, with labeled boundaries rather than unrestricted access.

Agentic request lifecycle

Agentic request lifecycleA numbered lifecycle contains sixteen steps: request admission; identity and authority resolution; risk classification; context assembly; model-route selection; inference; tool planning; authorization; tool execution; validation; human approval where required; response finalization; memory decision; evidence persistence; evaluation; and incident or recovery handling. A recovery loop can return to a safe retry point without repeating irreversible side effects.

1Admission2Identity3Risk4Context5Route6Inference7Plan8Authorize9Execute10Validate11Approve12Respond13Memory14Evidence15Evaluate16RecoverGoverned requestSafe retry pointsand evidence at every transition

A governed agentic runtime treats planning and tool use as one part of a larger admission-to-evidence lifecycle.
Text description

A numbered lifecycle contains sixteen steps: request admission; identity and authority resolution; risk classification; context assembly; model-route selection; inference; tool planning; authorization; tool execution; validation; human approval where required; response finalization; memory decision; evidence persistence; evaluation; and incident or recovery handling. A recovery loop can return to a safe retry point without repeating irreversible side effects.

Evidence and trace lifecycle

Evidence and trace lifecycleA request creates a correlation identifier and trace root. Infrastructure spans record queues and dependencies. Model spans reference deployment, route, token metrics, and redacted instruction versions. Tool spans record authorization, inputs by hash or reference, results, and side effects. Policy records capture decision and reason codes. Business outcomes and evaluation results are linked asynchronously. A minimization gate controls fields before retention, and retention policies expire or redact records by classification.

CorrelationrootInfra and modelspansTool and policyrecordsOutcome andevaluationMinimize andredactRetain orexpireEvidence and feedback inform later control-plane decisions

Correlation joins infrastructure, model, tool, policy, business-outcome, and evaluation records without requiring raw prompt retention.
Text description

A request creates a correlation identifier and trace root. Infrastructure spans record queues and dependencies. Model spans reference deployment, route, token metrics, and redacted instruction versions. Tool spans record authorization, inputs by hash or reference, results, and side effects. Policy records capture decision and reason codes. Business outcomes and evaluation results are linked asynchronously. A minimization gate controls fields before retention, and retention policies expire or redact records by classification.

Deployment topology comparison

Deployment topology comparisonSix topology columns compare hosted API, managed cloud or data center, local desktop, browser, edge or mobile, and hybrid execution. Rows show model location, runtime ownership, data egress, scaling model, offline behavior, update path, and primary constraints. Hybrid arrows indicate policy-driven routing between local and hosted execution. The diagram does not rank topologies; it highlights responsibility and trust-boundary changes.

Hosted APIClouddata centerLocal desktopBrowserEdgemobileHybridModel locationexternalmanageddevicebrowserdevicepolicy routeRuntime ownermanageddevicebrowserdevicepolicy routeexternalData egressdevicebrowserdevicepolicy routeexternalmanagedScaling modelbrowserdevicepolicy routeexternalmanageddeviceOffline behaviordevicepolicy routeexternalmanageddevicebrowserUpdate pathpolicy routeexternalmanageddevicebrowserdevicePrimary constraintexternalmanageddevicebrowserdevicepolicy route

Hosted, cloud or data-center, local, browser, edge, serverless, and hybrid deployments move different runtime responsibilities across trust boundaries.
Text description

Six topology columns compare hosted API, managed cloud or data center, local desktop, browser, edge or mobile, and hybrid execution. Rows show model location, runtime ownership, data egress, scaling model, offline behavior, update path, and primary constraints. Hybrid arrows indicate policy-driven routing between local and hosted execution. The diagram does not rank topologies; it highlights responsibility and trust-boundary changes.

Request gateway and identity

The gateway authenticates actor/service, resolves tenant, validates the contract, applies rate/deadline/budget, and starts trace context.

Implementation

Keep validation and authority outside model prompts and provider adapters.

Operational implications

The gateway may be in-process for a small deployment but remains a logical boundary.

Measure

Auth/validation, rate-limit, accepted/rejected, contract version, and trace creation.

Runtime coordinator

The coordinator executes the governed state machine and separates transient execution from durable checkpoints.

Implementation

Use explicit step results, cancellation, timeouts, retry classification, and state versions.

Operational implications

Avoid one monolithic function containing provider, tool, policy, and storage details.

Measure

Step duration/status, attempt, checkpoint, cancellation, and task outcome.

Context plane

Context providers expose approved domain data, retrieval, files, memory, and semantic metrics with provenance and policy.

Implementation

Normalize provider results and apply classification/minimization before model assembly.

Operational implications

Context is not equivalent to unrestricted database access.

Measure

Retrieval latency, source/citation, tokens, freshness, and denied content.

Model routing and adapters

The router chooses a compliant candidate; adapters normalize provider/local engine protocols and usage.

Implementation

Use a capability catalog and explicit fallback order. Keep provider details in adapter traces.

Operational implications

Routing changes can affect privacy, cost, quality, and residency.

Measure

Route/fallback, provider latency/error, tokens, cost, quality, and compliance.

Tool broker and execution sandbox

The broker discovers permitted tools, validates proposals, authorizes/approves, executes with idempotency, and validates results.

Implementation

Use narrowly scoped credentials and isolate generated/untrusted code.

Operational implications

The tool subsystem is the primary boundary between probabilistic proposals and deterministic side effects.

Measure

Tool stage timings, policy decisions, approval, result, side effects, and sandbox events.

Memory and systems of record

The memory manager owns runtime memory scopes while domain services own authoritative business records.

Implementation

Use typed read/write commands, provenance, expiry, conflict, and deletion.

Operational implications

Do not make vector stores authoritative systems of record.

Measure

Memory hits/writes/conflicts/deletes and domain-command outcomes.

Policy and trust plane

A policy decision point evaluates versioned policy; enforcement points gate boundary, context, routing, tools, memory, and output.

Implementation

Record decision ID/effect/reason, fail behavior, policy version, and protected input references.

Operational implications

A shared policy library without consistent enforcement can create false confidence.

Measure

Coverage, allow/deny/challenge, latency, unavailable decisions, and bypass attempts.

Telemetry, evaluation, and replay

Tracing correlates components; evaluation assesses output/outcome; replay reconstructs versions, state, and decisions.

Implementation

Use OpenTelemetry-compatible propagation, controlled attributes, evidence references, and workflow links.

Operational implications

Do not store sensitive content merely to make replay convenient.

Measure

Trace completeness, evaluation coverage, evidence availability, and replay success.

Deployment variants

Small systems may deploy components in one process; larger systems separate model serving, workflow, policy, memory, and tools.

Implementation

Preserve logical contracts and trace context across process/network boundaries.

Operational implications

Service decomposition should follow scaling, security, ownership, or failure needs—not diagram aesthetics.

Measure

Network/dependency latency, availability, scaling, failure scope, and operating cost.

Portability tests

Replaceability is proven by contract tests and fixture parity, not by interface names.

Implementation

Maintain test adapters, capability conformance, trace fixtures, failure behavior, and migration/rollback.

Operational implications

Provider-neutral abstractions should not erase capabilities that matter; expose them through versioned extensions.

Measure

Conformance pass, migration effort, output parity, failure parity, and fallback.

Topic

Implementation

Operational implications

Measure

Reference tables

Reference components
Component Owns Does not own
Gateway Identity, boundary validation, budgets Model execution
Coordinator/workflow Task state and transitions Provider-specific API
Context providers Approved data retrieval and provenance Final authorization to act
Router/adapters Model selection and protocol normalization Product business state
Tool broker Authorized side-effect execution Model reasoning
Memory manager Runtime memory lifecycle Authoritative domain records
Policy service Versioned decisions Enforcement without PEPs
Telemetry/evaluation Evidence and outcome assessment Permission to store unrestricted data

Decision checklist

  1. Which component owns every state mutation?
  2. Where are authentication, authorization, and policy enforced?
  3. Which adapters can be replaced independently?
  4. What durable state survives process failure?
  5. What data crosses trust boundaries?
  6. How are model and tool capacity isolated?
  7. What one trace crosses the whole task?
  8. Which failure is contained to a request, worker, or system?
  9. What conformance test proves portability?

Common mistakes

  • Deploying a diagram with no interface or state ownership.
  • Putting durable workflow state inside the model server.
  • Letting context providers return unclassified raw records.
  • Giving tool adapters independent retry policies.
  • Using provider-neutral abstractions that hide privacy/cost/quality differences.
  • Centralizing every component into one failure domain at scale.
  • Splitting every logical component into a service prematurely.

Sources and further reading


  1. OpenTelemetry concepts
    (opens in a new tab)

    OpenTelemetry · Official documentation · reviewed 2026-06-21 UTC

  2. Open Policy Agent
    (opens in a new tab)

    OPA · Official documentation · reviewed 2026-06-21 UTC

  3. Temporal documentation
    (opens in a new tab)

    Temporal · Official documentation · reviewed 2026-06-21 UTC

  4. Model Context Protocol specification
    (opens in a new tab)

    MCP · Protocol specification · reviewed 2026-06-21 UTC

  5. ONNX Runtime architecture
    (opens in a new tab)

    ONNX Runtime · Official documentation · reviewed 2026-06-21 UTC

Last reviewed: 2026-06-23 UTC

Maintenance record

Last materially changed
2026-06-23 UTC
Last reviewed
2026-06-23 UTC

Found an error, outdated capability, or unclear category boundary? Submit a correction with a supporting source.