Home - aRuntime.com

AI runtime is an overloaded term.

The phrase may refer to graph execution, inference, model serving, distributed systems, browser and edge execution, or the agentic layer that governs context, tools, memory, policy, and traces. This site separates those meanings.

Canonical definition: An AI runtime is the execution environment that turns model artifacts or model requests into operational behavior. Depending on the layer, it may compile computational graphs, schedule hardware, execute inference, serve models, coordinate distributed workloads, or govern context, tools, memory, policy, and traces.

AI runtime is an umbrella term, not a single product category. Correct architecture begins by naming the runtime layer under discussion.

Seven-layer reference model

The AI runtime stack

Layers describe responsibilities rather than marketing labels. One product may span several layers, and a production system usually composes several products.

Hardware and system substrate

CPUs, GPUs, NPUs, memory, storage, networking, drivers, operating systems, and isolation provide the physical execution boundary.

Kernels and hardware libraries

Optimized operators, collectives, math libraries, and device runtimes expose efficient hardware primitives.

Compiler and graph runtime

Graph capture, intermediate representations, fusion, partitioning, lowering, scheduling, and memory planning turn models into executable work.

Model and LLM inference engine

Model loading, quantization, prefill, decode, KV-cache management, batching, streaming, and token metrics execute generation.

Serving and distributed runtime

APIs, repositories, versioning, scheduling, autoscaling, traffic management, parallelism, collectives, and failure handling operate models across requests and hosts.

Agentic and application runtime

Identity, context, model routing, tools, memory, approvals, policy, evaluation, evidence, traces, and recovery turn outputs into controlled work.

Product and workflow layer

User experience, domain workflows, business state, review queues, integrations, and outcomes connect runtime behavior to a product.

Seven-layer AI runtime stack

The runtime stack separates hardware, kernels, compilation, inference, serving, agentic control, and product workflow responsibilities.

Text description

A vertical stack of seven numbered layers. Layer 0 is hardware and system substrate. Layer 1 is kernels and hardware libraries. Layer 2 is the compiler and graph runtime. Layer 3 is the model and LLM inference engine. Layer 4 is serving and distributed execution. Layer 5 is the agentic and application runtime. Layer 6 is the product and workflow layer. Cross-cutting identity, security, observability, evaluation, cost, and evidence concerns apply across the stack.

Two paths through the stack

Model execution and request execution intersect, but they are not the same path.

Model execution path

Model artifact or framework graph
Intermediate representation and optimization
Partitioning, lowering, or code generation
Model loading and hardware execution

This path turns model structure and weights into operations that execute on hardware.

Request execution path

Actor, tenant, task, risk, and authority
Context assembly and model routing
Queueing, batching, generation, and tool execution
Validation, policy, evidence, response, and memory updates

This path turns an admitted request into a controlled outcome and a reviewable record.

Where they intersect: The request path selects and invokes model-serving and inference capabilities; the model path performs the compiled execution. Model output then returns to the request path for validation, tools, policy, evidence, response finalization, and memory decisions.

Model execution path

A model artifact becomes executable through import, intermediate representation, optimization, lowering, loading, and hardware execution.

Text description

A left-to-right flow begins with a framework graph or model artifact, then creates an intermediate representation, applies graph rewrites and operator fusion, partitions and lowers work for execution providers, plans memory and generates code, loads and warms the model, and executes kernels on selected hardware. Feedback arrows show profiling data informing optimization and scheduling.

Request execution path

A production request carries identity, authority, risk, context, model-route, tool, policy, evidence, and memory decisions.

Text description

A left-to-right request flow starts with actor and tenant identity, proceeds through admission and risk classification, context assembly and model routing, queueing and inference, optional tool authorization and execution, validation and approval, response finalization, and trace, evidence, and memory decisions. Denied or failed actions branch to recovery and user-visible error handling.

Choose by job

Start with the work the runtime must perform.

Serve an LLM at high concurrency

Run a model locally

Deploy AI in a browser

Deploy to mobile, edge, or TinyML

Build an agentic application

Standardize enterprise model serving

Design a governed tool-execution system

Compare latency, throughput, quality, and cost

Select a runtime architecture

Design replayable evidence and traces

Category boundaries

Different runtime categories own different execution units and controls.

Compact comparison; use the taxonomy for definitions, caveats, and cross-layer products.
Category	Primary execution unit	Typical input	Typical output	Owns model execution	Owns tools	Owns durable state	Enforces policy	Captures traces	Typical boundary
Compiler or graph runtime	Graph or IR	Model graph and shapes	Executable graph or device work	Yes	No	No	Usually execution constraints	Graph and operator events	Process or device execution provider
Inference engine	Model request or token step	Weights, tokens, generation parameters	Logits, tokens, embeddings, or structured output	Yes	No	KV cache and runtime state	Generation constraints	Token and model metrics	Model instance or engine
Model server	Network inference request	API request and model identifier	Network response or stream	Through an engine	No	Repositories and request queues	Admission and traffic policy	Request and serving traces	Service endpoint
AI gateway	Provider request	Prompt or model API request	Routed provider response	Usually no	Usually no	Routing and usage records	Routing, quota, and egress rules	Gateway spans and usage	Provider-facing API boundary
Workflow engine	Workflow step	Events and deterministic state	State transition and scheduled work	No	Via activities	Yes	Workflow rules	Durable execution history	Workflow definition and state store
Agent framework	Agent turn or graph node	Messages, state, tools, model adapters	Next action or response	Via adapter	Plans or invokes	Framework-dependent	Often prompt or callback based	Framework events	Application library
Agentic application runtime	Governed task and side effect	Identity, task, authority, context, policy	Controlled work and evidence	Via route and serving layers	Yes, through a broker	Yes, with scoped policies	Yes	Correlated model, tool, policy, and outcome traces	Application execution boundary
Product application	User or business workflow	User intent and domain state	Product outcome	Via runtime services	Via runtime services	Business state	Business and product rules	Outcome and experience telemetry	Product and organization boundary

Open the detailed category comparison

Developer contract

Make authority, budget, policy, and evidence explicit before execution.

The runtime request contract gives products and runtime services a versioned boundary for identity, tenancy, risk, context, route constraints, tools, memory, approval, output, tracing, retention, idempotency, and deadlines.

Open the full schema Download the example implementation

Valid runtime request

{
  "contractVersion": "aruntime.runtime-request.v1",
  "requestId": "req_01",
  "correlationId": "corr_01",
  "idempotencyKey": "answer:01",
  "timestampUtc": "2026-06-23T14:30:00Z",
  "deadlineUtc": "2026-06-23T14:30:20Z",
  "actor": {
    "type": "user",
    "idRef": "user:42"
  },
  "tenant": {
    "idRef": "tenant:acme"
  },
  "taskType": "answer-with-evidence",
  "riskLevel": "medium",
  "input": {
    "question": "Summarize the approved architecture."
  },
  "permissions": [
    "read:architecture"
  ],
  "contextPolicy": {
    "classificationMax": "internal",
    "citeSources": true,
    "allowExternalRetrieval": false,
    "maxContextTokens": 12000
  },
  "modelRouteConstraints": {
    "allowHosted": true,
    "allowedRoutes": [
      "hosted-general-v3"
    ],
    "maxLatencyMs": 8000
  },
  "allowedTools": [
    "document.search.v2"
  ],
  "toolPolicy": {
    "defaultDecision": "deny",
    "maxCalls": 4,
    "allowSideEffects": false
  },
  "memoryPolicy": {
    "writeScopes": [],
    "writeRequiresApproval": true,
    "ttlSeconds": 3600,
    "readScopes": []
  },
  "budget": {
    "maxInputTokens": 14000,
    "maxOutputTokens": 1200,
    "maxToolCalls": 4,
    "maxCostUsd": 0.25,
    "currency": "USD"
  },
  "approvalPolicy": {
    "mode": "policy-driven",
    "timeoutSeconds": 300,
    "requiredFor": []
  },
  "outputContract": {
    "format": "json-schema",
    "schemaRef": "answer-with-citations.v1.json",
    "maxBytes": 64000,
    "requiresCitations": true
  },
  "traceSettings": {
    "level": "standard",
    "redactInputs": true,
    "captureToolResults": false,
    "samplingDecision": "record-and-sample"
  },
  "dataClassification": "internal",
  "retentionPolicy": {
    "policyId": "retention:standard-30d",
    "days": 30,
    "storeRawInputs": false
  }
}

Editorial trust

Claims are scoped, sourced, dated, and correctable.

Vendor-neutral taxonomy

Primary-source citations

UTC review dates

Scoped comparison methodology

No unsupported leaderboard claims

Public corrections process

Visible changelog

Clear fact, synthesis, and proposal labels

Read the editorial policy and source hierarchy

Start from your role

Move from orientation to the depth your decisions require.

Emerging runtime categories now have boundary pages

The current release keeps the runtime-type expansion, repairs the RSS endpoint, and makes the top navigation easier to scan on desktop and mobile.

Understand the execution stack behind reliable AI systems.

AI runtime is an overloaded term.

The AI runtime stack

Seven-layer AI runtime stack

Model execution and request execution intersect, but they are not the same path.

Model execution path

Request execution path

Model execution path

Request execution path

Start with the work the runtime must perform.

Different runtime categories own different execution units and controls.

Make authority, budget, policy, and evidence explicit before execution.

Valid runtime request

Claims are scoped, sourced, dated, and correctable.

Move from orientation to the depth your decisions require.

Emerging runtime categories now have boundary pages

Continue with a concrete architecture task.