Definitions across the complete AI runtime stack, including hardware execution, graph compilation, inference, serving, distributed systems, edge deployment, agentic controls, governance, and evaluation.
137 terms shown
Agentic
Agentic Execution Environment
Definition: A runtime boundary for safely running autonomous or semi-autonomous agent work with sandboxing, lifecycle control, tool authorization, secret brokering, audit, and recovery.
AEE is deeper than an agent framework because it owns authority, isolation, resources, and traces around side-effecting work.
Cognitive Runtime Environment
Definition: A runtime layer that manages high-level reasoning behavior around a language model, including persistent memory, context paging, planning loops, reasoning budgets, and recovery.
CRE is a software runtime pattern for long-horizon reasoning, not evidence of consciousness or personhood.
Overthinking intervention
Definition: A runtime action that terminates, redirects, verifies, or escalates reasoning when traces show oscillation, repeated reconsideration, or rising cost without progress.
This should be based on structured signals, not hidden chain-of-thought disclosure.
Reasoning budget
Definition: A bound on tokens, time, steps, branches, or tool calls that a reasoning loop may consume before termination or escalation.
Reasoning budgets help prevent overthinking, loops, and cost runaway.
Side-effect classification
Definition: A method for classifying tools or actions by whether they read, write, send, spend, delete, actuate, or otherwise change external state.
This classification drives approval, idempotency, retry, and audit requirements.
Agentic and application runtime
Agent framework
Definition: A programming framework that supplies abstractions for model calls, tools, state, planning, or multi-step flows; it may participate in but does not by itself guarantee production runtime controls.
Agentic runtime
Definition: An application-runtime layer that turns probabilistic model behavior into controlled work by governing request boundaries, identity, context, tools, memory, policy, approvals, evaluation, evidence, and recovery.
Approval gate
Definition: A runtime checkpoint that blocks an action until an authorized human or system supplies a time-bounded decision.
Compensation
Definition: A forward action intended to counteract or mitigate a completed side effect when transactional rollback is unavailable.
Context assembly
Definition: Selection, authorization, transformation, ordering, and attribution of instructions, retrieved data, state, and other inputs supplied to execution.
Human review
Definition: A defined process in which an authorized person evaluates a proposed action, output, exception, or recovery path before execution or release.
Idempotency key
Definition: A stable identifier that lets a system recognize repeated attempts for the same logical operation and avoid unintended duplicate effects.
Long-term memory
Definition: Durable state made available across tasks or sessions under explicit provenance, authorization, update, deletion, and retention policies.
Runtime request contract
Definition: A versioned envelope that states actor, tenant, task, risk, permissions, context, route, tools, memory, budget, approval, output, trace, classification, retention, idempotency, and deadline requirements.
Side effect
Definition: A change outside the model’s transient computation, such as writing a record, sending a message, executing code, moving money, or changing infrastructure.
Tool broker
Definition: A controlled component that resolves tool contracts, authorizes calls, validates arguments, invokes implementations, and records outcomes and side effects.
Tool contract
Definition: A versioned specification for a tool’s identifier, schemas, authentication, permissions, side effects, egress, timeout, retry, idempotency, approvals, compensation, errors, and evidence.
Working memory
Definition: Short-lived state retained for an active request, task, or session and bounded by explicit access and retention rules.
Compiler
Execution provider
Definition: A backend that advertises which neural graph operations it can execute on a target device or library.
Execution providers often coexist with fallback providers when coverage is incomplete.
Neural Execution Engine
Definition: A low-level runtime and compiler-adjacent execution layer that maps neural graphs, weights, activations, and token steps onto heterogeneous hardware.
NEE owns IR lowering, dispatch, memory planning, backend scheduling, and fallbacks, not product policy or agent authority.
Neural Runtime Abstraction Layer
Definition: An abstraction layer that hides hardware-specific neural execution details behind a common runtime interface.
This is closely related to Neural Execution Engine terminology and should be grounded in concrete backend and operator responsibilities.
Compiler and graph
AOT compilation
Definition: Compilation performed before deployment or execution to create target-ready artifacts with controlled versions and compatibility assumptions.
Code generation
Definition: The production of executable machine code, kernels, bytecode, or target artifacts from an intermediate representation.
Delegate
Definition: A component that receives supported operations from a host runtime and executes them on a specialized backend or accelerator.
Dynamic shape
Definition: A tensor dimension whose value may vary between executions rather than being fixed in the compiled artifact.
Fallback
Definition: Execution on a less-specialized path when an operation, shape, device, or feature is unsupported by the preferred backend.
Graph capture
Definition: The process of converting observed or declared model operations into a graph or program representation suitable for analysis and optimization.
Graph partitioning
Definition: The division of a graph into subgraphs assigned to different devices, backends, execution providers, or fallback paths.
Graph rewrite
Definition: A semantics-preserving transformation that replaces part of a computation graph with an equivalent form.
Intermediate representation
Definition: A compiler data model used between source or framework graphs and target code so transformations, analysis, partitioning, and lowering can be performed systematically.
JIT compilation
Definition: Compilation performed during or near execution, commonly specialized to observed shapes, values, devices, or runtime conditions.
Lowering
Definition: The transformation of a high-level operation or IR into a more target-specific representation or implementation.
Memory planning
Definition: The compile-time or runtime assignment and reuse of buffers based on tensor lifetimes, layouts, shapes, and device constraints.
MLIR
Definition: A compiler infrastructure for defining and transforming multiple intermediate-representation dialects across abstraction levels.
ONNX
Definition: An open format for representing machine-learning model graphs and operators so models can be exchanged among compatible tools and runtimes.
Operator fusion
Definition: The combination of adjacent graph operations into a larger operation to reduce intermediate memory traffic or dispatch overhead.
Shape guard
Definition: A condition that determines whether a compiled specialization is valid for a given input shape or value constraint.
StableHLO
Definition: A portable operation set and serialization format in the OpenXLA ecosystem intended to preserve compatibility for machine-learning compiler workflows.
TOSA
Definition: A tensor-operator specification designed as an intermediate layer for machine-learning compilation and deployment.
Discovery and publishing
Answer Engine Optimization
Definition: AEO is the practice of publishing direct, citable, source-backed answers that answer engines can extract and route back to the canonical page.
AEO on ARuntime.com favors definition boxes, stable headings, visible evidence, limitation statements, and correction paths over keyword stuffing.
Generative Engine Optimization
Definition: GEO is the practice of publishing retrievable, grounded, source-resilient content that generative systems can synthesize and cite without losing context or boundaries.
GEO on ARuntime.com requires source-visible context, maturity labels, explicit exclusions, route inventory, and parity across human and machine-readable surfaces.
Route inventory
Definition: A reviewed list of public routes with titles, purposes, review dates, indexability, and support boundaries.
A route inventory helps humans and agents distinguish canonical public pages from retired, private, duplicate, or unsupported routes.
Search Engine Optimization
Definition: SEO is the practice of making public pages crawlable, understandable, canonical, internally linked, performant, accessible, and useful for search users.
SEO on ARuntime.com supports AI-ready publishing but does not replace human-readable definitions, source evidence, or support boundaries.
Edge
Autonomous Inference Runtime
Definition: A local or edge-oriented runtime that executes trained models near the device, sensor, user, robot, vehicle, or industrial process that needs the result.
AIR focuses on latency, privacy, offline operation, power, thermal behavior, local model updates, and device-aware fallback.
Edge, browser, and local
Browser runtime
Definition: A model-execution environment operating within browser security and capability boundaries, commonly using JavaScript, WebAssembly, WebGPU, or platform APIs.
Edge runtime
Definition: A runtime deployed near data sources or users under constrained connectivity, resource, latency, update, and physical-security conditions.
Local runtime
Definition: A runtime executing on the user or operator device rather than a remote hosted service.
Model packaging
Definition: Assembly of weights, tokenizer or preprocessing assets, runtime configuration, licenses, integrity data, and compatibility metadata for deployment.
Offline inference
Definition: Model execution that does not require an active network connection after necessary artifacts and configuration are available locally.
On-device runtime
Definition: A runtime that performs model execution within a phone, desktop, embedded device, appliance, or other endpoint.
Progressive fallback
Definition: A design that selects the best supported local path and falls back to another local or hosted path while preserving explicit privacy and behavior rules.
Thermal throttling
Definition: Automatic reduction of device performance to remain within temperature or power limits.
TinyML
Definition: Deployment of machine-learning inference on highly resource-constrained microcontrollers or embedded systems.
WebAssembly
Definition: A portable binary instruction format and sandboxed execution model used by browsers and other hosts for near-native compiled code.
WebGPU
Definition: A web API exposing modern GPU rendering and general-purpose compute capabilities through a browser-managed security model.
WebNN
Definition: A web API for constructing and executing neural-network graphs using available operating-system and device acceleration capabilities.
Enterprise
Machine Intelligence Runtime
Definition: An enterprise execution-time control layer that coordinates model-backed work across identity, policy, memory, tools, model routing, telemetry, audit, budgets, and governed state changes.
MIR is useful as architecture language for managed machine behavior, not as a claim that machine intelligence is conscious or standardized.
Governance
Dual attribution
Definition: A logging and identity pattern that records both the machine actor and the authenticated human authority behind a workflow.
Dual attribution helps close service-account gaps in model-backed enterprise automation.
Governance and discovery
AI-Ready Web
Definition: A public-content readiness pattern where visible pages, metadata, discovery files, correction paths, and support boundaries agree.
On ARuntime.com, this term is limited to content quality and discovery. It is not certification, endorsement, runtime authority, or a ranking promise.
No-op safety
Definition: A safety posture where unsupported, unauthorized, or unsafe requests stop cleanly instead of guessing, escalating authority, or executing a side effect.
No-op safety is important for public discovery because public files do not authorize scraping outside policy, authentication, posting, mutation, credential validation, certification, or bypassing local controls.
Inference
KV cache orchestration
Definition: Runtime management of transformer key-value cache allocation, reuse, paging, sharing, and offload.
KV cache orchestration often dominates LLM serving throughput and memory capacity.
Inference and memory
Chunked prefill
Definition: Execution of long prompt prefill in bounded chunks so prefill work can be interleaved with decode or other requests.
Context length
Definition: The number of tokens or other units a model accepts within its configured input and generated sequence window.
Continuous batching
Definition: A scheduling method that admits and removes sequences while a batch is running instead of waiting for all batch members to finish.
Decode
Definition: The iterative LLM inference phase that generates subsequent tokens while reading and extending the key-value cache.
ITL
Definition: Inter-token latency: the observed latency between individual streamed output tokens, often analyzed as a distribution rather than a single mean.
KV cache
Definition: Stored attention keys and values from prior tokens that avoid recomputing the full sequence during autoregressive decoding.
Memory pressure
Definition: A condition in which model weights, activations, caches, buffers, or other allocations approach available memory and force eviction, offload, throttling, or failure.
Mixed precision
Definition: Use of multiple numeric precisions within one model or execution path to balance accuracy, speed, and resource use.
PagedAttention
Definition: A KV-cache management technique that maps logical cache blocks to non-contiguous physical memory blocks to improve allocation and sharing.
Post-training quantization
Definition: Quantization applied after model training using calibration, statistics, or weight-only conversion rather than retraining the full model.
Prefill
Definition: The LLM inference phase that processes the input sequence, produces the first-token state, and populates key-value cache entries for subsequent decoding.
Prefix caching
Definition: Reuse of previously computed model state for an identical or compatible input prefix.
Quantization
Definition: Representation and execution of model values at reduced precision or with compressed numeric formats to lower memory, bandwidth, or compute cost.
Speculative decoding
Definition: Generation that uses one or more draft mechanisms to propose tokens that a target model verifies, with the goal of reducing decode latency.
Structured generation
Definition: Generation constrained or validated against a defined structure such as JSON, a grammar, or a schema.
TPOT
Definition: Time per output token: an aggregate interval between generated output tokens, excluding or including specified phases according to the benchmark method.
TTFT
Definition: Time to first token: elapsed time from request admission or arrival to the first generated token, with the measurement boundary stated explicitly.
Memory
Bitemporal memory
Definition: A memory or graph model that records both when a fact was valid in the world and when the system recorded or learned it.
Bitemporal memory supports reproducible time-travel queries and audit reviews.
Dream cycle
Definition: An offline or background consolidation process that deduplicates, compresses, connects, archives, or retires memories.
Dream cycles are an implementation pattern, not a guarantee of human-like cognition.
Observability and evaluation
Benchmark fixture
Definition: A versioned set of models, inputs, expected properties, environment assumptions, and measurement instructions used for repeatable evaluation.
Business outcome
Definition: A product- or domain-level result, such as a resolved case or accepted change, that should be measured separately from model output quality.
Cost per successful task
Definition: Total scoped runtime cost divided by tasks that satisfy defined quality, policy, and completion criteria.
Evaluation result
Definition: A versioned record of a test case, configuration, grader or rule, score or decision, threshold, evidence, and pass/fail interpretation.
Quality regression
Definition: A measurable decline in task quality, validity, safety, or outcome metrics relative to an approved baseline.
Replay
Definition: Re-execution or reconstruction of a prior request using preserved versions, inputs or references, decisions, and controlled substitutes to diagnose or evaluate behavior.
Runtime trace
Definition: A correlated representation of work across runtime components and layers, using spans, events, links, attributes, and status while respecting data-minimization rules.
Span
Definition: A timed unit of work within a trace, identified by trace and span identifiers and annotated with bounded attributes, events, status, and relationships.
Tail latency
Definition: Latency near the high end of a distribution, such as the 95th or 99th percentile, which exposes slow-path and queueing behavior hidden by averages.
Throughput
Definition: The amount of completed work per unit time, such as requests, tokens, or model executions, with workload and success criteria stated.
Trace correlation
Definition: Use of identifiers and links to connect infrastructure, model, tool, policy, product-outcome, and evaluation records without requiring all payloads in one system.
Runtime fundamentals
AI runtime
Definition: The execution environment that turns model artifacts or model requests into operational behavior; depending on the layer, it may compile graphs, schedule hardware, execute inference, serve models, coordinate distributed workloads, or govern context, tools, memory, policy, and traces.
Application runtime
Definition: An execution environment that coordinates application-level state, policies, dependencies, and work around lower-level model execution.
Compiler runtime
Definition: The runtime portion of a compiler-backed system that loads compiled artifacts, binds inputs, allocates memory, dispatches generated code, and handles supported fallbacks.
Execution boundary
Definition: The point at which ownership of inputs, authority, state, failures, telemetry, or side effects transfers between runtime components.
Framework
Definition: A programming environment that supplies model construction, training, export, execution, or application abstractions; a framework may embed a runtime but is not automatically one category.
Graph runtime
Definition: An execution environment that evaluates a model or computation graph, often after graph rewriting, partitioning, scheduling, and memory planning.
Inference engine
Definition: A system optimized to load model weights and perform inference, including device execution, memory management, batching, streaming, and model-specific generation mechanics.
Inference runtime
Definition: A runtime whose primary responsibility is executing trained model artifacts to produce predictions, embeddings, tokens, or other model outputs.
Library
Definition: A reusable collection of functions or components invoked by another program; it does not necessarily own process, request, scheduling, or lifecycle boundaries.
Model lifecycle
Definition: The sequence from model creation and export through optimization, packaging, deployment, loading, serving, version transition, and retirement.
Model server
Definition: A networked service that exposes one or more models through APIs and adds loading, health, scheduling, batching, version, and repository behavior around an inference engine.
Request lifecycle
Definition: The sequence from request admission through identity, context, routing, execution, validation, policy, response, evidence, evaluation, and recovery.
Serving platform
Definition: An operational layer that deploys and manages model servers across environments, including rollout, autoscaling, traffic, multi-model placement, policy, and fleet health.
Security
Opaque secret broker
Definition: A runtime service that gives an agent a handle to a credential rather than placing the raw secret inside the agent environment.
The broker signs or authorizes requests outside the agent context to reduce credential exfiltration risk.
Policy hook
Definition: A deterministic interception point that evaluates a proposed action before or after execution.
Policy hooks keep enforcement outside the model prompt and should emit auditable allow, deny, or approval-required outcomes.
Semantic control plane
Definition: A policy layer that binds access and execution controls to semantic concepts rather than only physical tables, files, or API routes.
This helps prevent restricted meanings from leaking through alternate query paths.
Security, policy, and evidence
Confused deputy
Definition: A security failure in which a component with authority is induced to misuse that authority on behalf of a less-privileged requester.
Data classification
Definition: Assignment of handling categories to data so access, routing, retention, logging, and egress rules can be enforced.
Evidence record
Definition: A minimized, correlated record of request identity, versions, context references, model route, tool actions, policy and approval decisions, side effects, failures, recovery, evaluation, redaction, and retention.
Model artifact integrity
Definition: Verification that model files and associated assets match approved hashes, signatures, provenance, versions, and supply-chain policy.
Policy decision point
Definition: A component that evaluates policy inputs and returns an explicit decision and reason codes.
Policy enforcement point
Definition: A component that applies a policy decision at the boundary where access, routing, data release, or action occurs.
Prompt injection
Definition: Untrusted content crafted to alter model behavior or override intended instructions, especially when the model can access tools, data, or privileged actions.
Provenance
Definition: Information describing the origin, custody, transformation, version, and authority of data, models, instructions, outputs, or evidence.
Redaction
Definition: Removal, masking, tokenization, or transformation of sensitive data before storage, display, telemetry, or downstream transfer.
Retention policy
Definition: Rules governing how long specific data and evidence categories are kept, where they may be stored, and how they are deleted or archived.
Sandbox
Definition: A constrained execution environment that limits code, filesystem, network, process, credential, and device access.
Tenant isolation
Definition: Controls that prevent one tenant’s data, cache, memory, configuration, tools, or evidence from becoming accessible to another tenant.
Semantic
Semantic compiler
Definition: A runtime component that transforms natural-language intent into governed SQL, API calls, code, or structured execution plans.
A semantic compiler must validate schema, permissions, metrics, and provenance before execution.
Semantic Runtime Engine
Definition: A runtime layer that manages meaning as an operational data type, including embeddings, semantic tokens, graph relationships, semantic-layer metrics, and compiled intent.
SRE coordinates semantic memory, retrieval, policy, provenance, and deterministic execution boundaries.
Semantic token
Definition: A discrete representation of meaning used to stabilize or categorize otherwise continuous semantic embeddings or concepts.
Semantic tokens should be governed by schema and provenance rather than invented freely at execution time.
Serving and distributed execution
Admission control
Definition: A decision at the service boundary to accept, queue, shed, defer, or reject work based on capacity, authority, priority, or policy.
Autoscaling
Definition: Adjustment of runtime capacity using demand, queue, utilization, latency, memory, or custom workload signals.
Backpressure
Definition: A mechanism that limits upstream work when downstream capacity, queues, memory, or dependencies are saturated.
Canary rollout
Definition: A deployment strategy that sends a limited, measured share of traffic to a new version before broader promotion.
Cold start
Definition: Latency and resource work incurred when an execution environment, model, cache, or worker is not already initialized.
Collective communication
Definition: A coordinated multi-participant operation such as all-reduce, all-gather, reduce-scatter, or broadcast used in distributed execution.
Disaggregated prefill and decode
Definition: An architecture in which prompt processing and token decoding run on separately scaled workers with model state transferred or remotely accessed.
Dynamic batching
Definition: Server-side combination of requests into batches according to queue, delay, shape, priority, and model constraints.
Expert parallelism
Definition: Distribution and routing of mixture-of-experts components across devices or hosts.
Goodput
Definition: The rate of requests or tokens that meet defined quality and service-level requirements, rather than raw completed throughput alone.
Model repository
Definition: A versioned location and layout from which a serving system discovers and loads model artifacts and configuration.
Multi-model serving
Definition: Operation of several models or versions within a shared serving fleet, often requiring placement, isolation, loading, eviction, and routing policies.
Pipeline parallelism
Definition: Distribution of sequential model stages across devices or workers, with microbatches moving through the pipeline.
Tensor parallelism
Definition: Distribution of tensor operations or parameter dimensions for one model execution across multiple devices.
Warmup
Definition: Controlled execution used to initialize model state, compile paths, allocate memory, populate caches, or stabilize measurements before serving or benchmarking.
No glossary terms match this filter.