Edge or Mobile System

Problem

Edge and mobile deployments operate under resource and lifecycle constraints that are less visible in hosted serving: limited memory, battery, thermal envelopes, intermittent connectivity, heterogeneous accelerators, application-store release cycles, device fragmentation, and restricted telemetry. A model that fits a laboratory device may fail after concurrent application load, OS updates, sustained use, or a delegate fallback. Quantization and backend partitioning can also change quality and numerical behavior.

Why a runtime layer is needed

The edge runtime packages the model and operators, selects CPU/GPU/NPU backends, manages memory and scheduling, and exposes health and fallback. The product application runtime adds permissions, local data policy, update rollout, telemetry minimization, and remote-service boundaries. Both layers are needed: efficient inference alone does not define update safety, privacy, or degraded behavior.

Reference architecture

An export and packaging pipeline produces a versioned model artifact, tokenizer or pre/post-processing assets, operator manifest, quantization metadata, test vectors, license reference, and integrity signature.
A device capability layer maps hardware, OS, runtime, delegate, memory, and feature support to an approved execution plan.
The inference runtime preallocates bounded memory where possible, schedules work around foreground responsiveness, and supports cancellation and thermal/power adaptation.
Backend partitioning and delegate fallback are observable; the application does not assume that requested accelerator execution actually occurred.
Model updates use staged rollout, compatibility checks, signed manifests, health criteria, and automatic rollback to a known-good package.
Local inputs, outputs, caches, and telemetry follow a device-appropriate data policy. Remote fallback and synchronization are explicit and resilient to intermittent connectivity.
Fleet evidence aggregates version, backend, latency, error, battery, thermal, and quality signals without requiring sensitive raw data.

Request flow

Resolve task, device capability, application state, privacy mode, latency target, and power/thermal policy.
Select an approved model, quantization, backend, and execution plan for the device.
Acquire local sensor or user data through platform permissions and apply on-device preprocessing.
Execute with bounded resources, cancellation, and backend-fallback telemetry.
Validate output against task-specific confidence, safety, or schema rules before product action.
Use remote fallback only under the declared connectivity and privacy policy.
Record minimized local/fleet evidence and update health signals for staged rollout decisions.

Contracts

Model package: version, format, operator set, quantization, expected accuracy range, memory estimate, backend compatibility, test vectors, signature, and rollback target.
Device capability: hardware/OS/runtime versions, supported delegates, memory class, power state, thermal state, and fallback reasons.
Execution policy: latency deadline, foreground/background behavior, thread limits, accelerator priority, quality fallback, and cancellation.
Fleet update policy: cohort, minimum compatibility, health thresholds, staged percentage, rollback trigger, telemetry fields, and retention.

Failure modes and recovery

Detection, behavior, retry, idempotency, recovery, and evidence
Failure	Detection	User-visible behavior	Retry and idempotency	Recovery and evidence
Delegate or NPU initialization fails	Backend returns unsupported operator, driver, allocation, or initialization error.	Use a tested CPU/GPU fallback or disable the feature with a clear message.	Retry only after releasing resources or an OS/runtime change; avoid loops. Inference itself is side-effect free.	Switch to an approved execution plan and report the actual backend. Evidence: Device/runtime class, failed backend, operator partition, and fallback.
Thermal or battery budget exceeded	Platform thermal state, power policy, or sustained-latency signal crosses threshold.	Reduce frequency/model quality, defer background work, or stop gracefully.	Resume when the platform reports a safe state and the task is still relevant. Requests have deadlines so stale work is not replayed.	Release accelerator resources and select a lower-power plan. Evidence: Thermal/power category, adaptation, canceled work, and user impact.
Quantized model quality regression	Canary evaluation or production quality proxy falls outside approved bounds.	Roll back the affected package or backend; do not mask with a generated explanation.	Retry only with a known-good package or execution plan. Package activation is versioned and atomic.	Stop rollout, rollback cohort, and investigate representative failure samples under privacy controls. Evidence: Package/quantization/backend versions, evaluation results, cohort, and rollback.
Update interrupted or incompatible	Download, signature, storage, application-version, or runtime compatibility check fails.	Continue using the known-good model and avoid partial activation.	Resume verified chunks or wait for a compatible app/runtime update. Artifacts are content-addressed and activation is atomic.	Delete invalid staging data and preserve rollback package. Evidence: Manifest, compatibility checks, downloaded hashes, and retained version.
Telemetry unavailable	Device offline, consent disabled, queue full, or collector unavailable.	Core inference continues under offline policy; do not block unless safety evidence is mandatory.	Use bounded, expiry-aware local queueing only for approved fields. Telemetry events have deterministic IDs and deduplication.	Drop expired low-value events and retain mandatory safety events according to policy. Evidence: Queue state, consent mode, dropped categories, and eventual delivery.

Security considerations

Sign model and runtime packages, verify before activation, and protect rollback artifacts from downgrade abuse.
Use mobile/embedded platform permission and sandbox boundaries for sensors, files, microphone, camera, and interprocess communication.
Do not expose raw device identifiers, sensor data, embeddings, or model inputs through fleet telemetry by default.
Treat remote configuration and model rollout as privileged control-plane operations with authenticated, integrity-protected messages.
Validate pre/post-processing and generated output before it controls physical, financial, or safety-relevant behavior.
Document which operators or subgraphs may fall back to CPU or a remote service and how that affects privacy and performance.

Observability

Record actual backend, partition/fallback, model and runtime versions, latency percentiles, peak memory, errors, cancellation, power/thermal category, and update state.
Use local aggregation and sampling to reduce battery, network, and privacy cost.
Separate fleet health metrics from user-content diagnostics; detailed captures require an explicit support or test mode.
Alert on rollout-correlated crashes, backend fallback shifts, quality regression, signature failures, and rollback loops.

Evaluation

Test representative device tiers, OS/runtime versions, sustained workloads, background/foreground transitions, low battery, and thermal throttling.
Compare quantized and partitioned execution against approved accuracy, numerical, and safety tolerances.
Exercise update interruption, insufficient storage, delegate failure, rollback, offline operation, and remote-fallback denial.
Evaluate telemetry minimization and fleet observability under real intermittent-connectivity conditions.

Metrics

Task quality by model, quantization, device tier, and backend
P50/P95/P99 latency, peak memory, startup time, and package size
Battery and thermal impact under sustained representative use
Delegate success, partition, and fallback rates
Crash-free and error-free session rate by rollout cohort
Update success, rollback, and time-to-healthy metrics
Offline completion and remote-fallback rates

Implementation checklist

Define representative device tiers and test each approved model/backend plan.
Package model, operators, pre/post-processing, test vectors, signatures, and rollback metadata together.
Observe actual delegate execution and fallback rather than requested backend alone.
Set memory, battery, thermal, background, and deadline policies.
Stage signed updates with health gates and automatic rollback.
Minimize fleet telemetry and test prolonged offline operation.

Sources

ExecuTorch Documentation

PyTorch Foundation · Official project documentation · 2026-06-23T00:00:00Z

Edge deployment concepts, model export, hardware backends, and constrained-device execution.
LiteRT Overview

Google · Official project documentation · 2026-06-23T00:00:00Z

On-device model execution, delegates, acceleration, and mobile deployment considerations.
Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

National Institute of Standards and Technology · Government framework · 2026-06-23T00:00:00Z

Risk identification, measurement, governance, and lifecycle controls for generative AI systems.
OpenTelemetry Traces

OpenTelemetry · Official project documentation · 2026-06-23T00:00:00Z

Trace and span semantics used to correlate runtime, model, tool, and policy operations.

Find runtime definitions and implementation guidance