Keep context, tools, policy, and durable project state under user control while using local or explicitly governed hosted inference.
Key takeaways
- Primary risk: Unexpected cloud egress, weak device security, stale local models, loss of encrypted state, and misleading privacy claims.
- Keep authoritative domain state outside model memory.
- Measure task outcome, safe failure, and evidence—not output fluency alone.
Problem
Keep context, tools, policy, and durable project state under user control while using local or explicitly governed hosted inference.
Principal risk: Unexpected cloud egress, weak device security, stale local models, loss of encrypted state, and misleading privacy claims.
Why runtime layers are needed
A single model invocation cannot reliably own identity, authorization, durable state, external side effects, recovery, or evidence. The runtime composes the necessary compiler/inference/serving path with application controls appropriate to this use case.
Reference architecture
- Local identity and user-controlled policy
- Encrypted local project and memory store
- Local model engine plus explicit hosted fallback router
- Local file/application tools with per-resource permissions
- Offline-capable task state and evidence export
- Update, backup, and restore mechanism
- Clear UI showing local, hosted, and unavailable modes
Request flow
- Resolve active user, project, device state, and network mode.
- Select local context and memory under project scope.
- Choose a local model when capability and resource limits allow.
- Request explicit policy/consent before hosted fallback and minimize the payload.
- Invoke local tools using OS permission boundaries.
- Record local/hosted route, external destinations, and effects.
- Persist only approved memory and encrypted evidence.
- Support offline continuation or a clear unavailable state.
Contracts
- Request contract includes local-only/hosted-allowed route policy, permitted destinations, local tools, memory scopes, and retention.
- Model-route contract states what data classes may leave the device and which providers/regions are allowed.
- Local tool contracts map runtime authority to OS files, applications, and accounts.
Use the runtime request, tool, policy and approval, evidence, and trace schemas as versioned reference boundaries.
Failure modes
- Local model cannot satisfy output contract
- Device memory or thermal limit
- Hosted fallback occurs without informed consent
- Encrypted store is unavailable or corrupt
- Local tool sees a broader filesystem than intended
- Backup includes prohibited memory
- Offline task resumes against stale external state
Security considerations
- Use operating-system user boundaries, encrypted storage, and secure credential stores.
- Default to no external egress and make fallback visible.
- Sign and verify model and policy updates.
- Separate local analytics from sensitive task content.
- Provide export and deletion controls.
Observability
Correlate request, model route, context sources, tool operations, policy decisions, approvals, artifacts, failures, recovery, and domain outcome. Apply redaction and retention before exporting traces.
Evaluation and metrics
- Local completion rate
- Hosted fallback rate by reason
- Unexpected egress incidents
- Offline continuity
- Resource/thermal failure
- Memory correction/deletion success
- User-visible route accuracy
- Evidence export success
Implementation checklist
- Publish a data-placement matrix.
- Test with network disabled.
- Show the active model route and external destination.
- Define signed update and rollback.
- Back up keys and encrypted data safely.
- Use a local deterministic application for tasks that do not need generation.