Scientific or Analytical Workflow

Problem

Scientific and analytical systems can produce plausible narratives from data without preserving enough information to reproduce the computation. Dataset versions, filters, code, package environments, random seeds, model routes, generated queries, intermediate artifacts, and human interventions may all affect the result. A chat transcript alone is not an adequate research record, and unrestricted code execution can expose sensitive data or create irreproducible dependencies.

Why a runtime layer is needed

The runtime binds each analytical claim to an admitted dataset scope, an executable plan, a controlled environment, versioned tools, intermediate artifacts, validation checks, and provenance. It separates exploratory model behavior from deterministic computation and records which statements are calculated, sourced, inferred, or proposed. This supports reproduction and review even when raw datasets or prompts cannot be broadly retained.

Reference architecture

A study or task contract identifies the research question, dataset versions, permitted transformations, sensitivity, expected outputs, acceptance tests, and retention requirements.
A provenance-aware data broker resolves immutable dataset snapshots or records query text, source versions, access policies, and extraction timestamps.
Code and query generation occurs in a constrained workspace. Execution uses a pinned image or environment lockfile, bounded resources, controlled network access, and deterministic seeds where applicable.
Intermediate datasets, notebooks, figures, tables, and model outputs receive content hashes and lineage links instead of being treated as disposable chat attachments.
Validation combines schema checks, unit tests, statistical diagnostics, domain constraints, and independent calculations where practical.
A citation and provenance assembler maps final claims to source records, computations, artifacts, and human approvals.
The evidence record stores enough metadata to reproduce or audit the run while honoring access, minimization, redaction, and retention policy.

Request flow

Admit the research question, data scope, authority, sensitivity, validation requirements, and reproducibility target.
Resolve immutable data snapshots or record authoritative query and source-version metadata.
Generate a plan that distinguishes retrieval, transformation, statistical computation, model interpretation, and human judgment.
Execute code and queries in a pinned, isolated environment and hash intermediate artifacts.
Run declared validations and compare results against acceptance criteria or known invariants.
Assemble claims with citations to datasets, code, computations, and external sources; mark unresolved uncertainty.
Publish the result with an environment manifest, provenance graph, limitations, and evidence reference.

Contracts

Runtime request: research question, dataset scope and versions, permitted tools, sensitivity, budget, deadline, validation plan, and output contract.
Dataset tool: dataset identifier, snapshot or query semantics, authorization, row/column limits, classification, freshness, and provenance fields.
Code-execution tool: environment digest, package policy, resource and network limits, input artifacts, expected outputs, and deterministic-seed policy.
Evidence schema: artifact hashes, lineage edges, code and environment references, validation results, human interventions, citations, and retention.

Failure modes and recovery

Detection, behavior, retry, idempotency, recovery, and evidence
Failure	Detection	User-visible behavior	Retry and idempotency	Recovery and evidence
Dataset version unavailable	Snapshot identifier cannot be resolved or source changed without a retained version.	Stop or explicitly mark the run non-reproducible; do not silently use current data.	Retry only after resolving an equivalent authoritative snapshot. Data extraction records a snapshot or exact query/source version.	Restore from governed archive or revise the task contract with disclosed limitations. Evidence: Requested and resolved versions, source status, and decision.
Environment cannot be reproduced	Image digest, lockfile dependency, or runtime package is unavailable.	Return the environment failure before presenting results as reproducible.	Retry with a verified replacement only after recording the change. Execution attempts have immutable environment identifiers.	Rebuild from trusted artifacts, re-run validations, and compare outputs. Evidence: Environment digest, missing dependency, replacement, and output comparison.
Validation fails	Schema, unit, statistical, domain, or acceptance test fails.	Present no accepted conclusion; show the failed check and bounded diagnostics.	Retry only after a reviewed change to data, code, or method. Artifacts and attempts are versioned rather than overwritten.	Correct the method or mark the hypothesis unsupported. Evidence: Failed test, inputs, code/environment version, and remediation.
Intermediate artifact write fails	Object store or metadata catalog rejects an artifact or lineage edge.	Pause finalization because the result cannot be fully reproduced.	Retry with the same content hash; avoid creating divergent artifact identities. Content-addressed artifact identifiers prevent duplicates.	Persist to approved durable fallback or fail closed. Evidence: Artifact hash, attempted location, storage error, and final status.
Citation or provenance gap	A final claim lacks an external source or computational lineage path.	Mark the claim unsupported or remove it from the accepted result.	Regenerate explanation without rerunning completed computation when possible. Claim IDs remain stable across explanation revisions.	Add the missing evidence or route for human methodological review. Evidence: Claim ID, missing edge type, reviewer decision, and final disposition.

Security considerations

Apply data-access policy before extraction and again before model context, artifact publication, and result sharing.
Run generated code without ambient credentials and with explicit filesystem, process, package, and network policy.
Treat datasets and documents as untrusted inputs that may contain executable content or prompt-injection text.
Prevent sensitive rows, raw identifiers, secrets, and proprietary code from entering general-purpose traces or model-provider logs.
Sign or otherwise integrity-protect environment manifests, artifact hashes, and evidence records when auditability requires it.
Define when results may be aggregated or de-identified and validate that policy independently of model output.

Observability

Correlate data access, code generation, execution, artifact creation, validation, model interpretation, and publication.
Capture environment digests, dataset snapshot IDs, query hashes, resource use, artifact hashes, validation results, and model routes.
Use provenance records for lineage and traces for execution timing; do not collapse the two concepts into one log stream.
Alert on unpinned dependencies, mutable dataset references, failed validation, missing artifacts, and unauthorized egress attempts.

Evaluation

Re-run fixed studies from retained contracts, data versions, code, and environments and compare artifact hashes or bounded tolerances.
Evaluate factual interpretation against source and computation evidence, not fluency.
Use negative controls, synthetic known-answer datasets, perturbation tests, and independent implementations for high-impact analyses.
Record human methodological review separately from automated validation and identify unresolved limitations.

Metrics

Reproducible-run rate
Dataset, code, environment, and artifact provenance completeness
Validation pass rate and failure-resolution time
Claim-to-evidence coverage
Environment drift and mutable-source incidence
Compute, storage, and review cost per accepted result
Correction or retraction rate after publication

Implementation checklist

Pin or precisely identify datasets, code, package environments, and model deployments.
Separate deterministic computation from model interpretation in the plan and evidence.
Hash and retain intermediate artifacts according to sensitivity and retention policy.
Specify validation tests and acceptance criteria before accepting the result.
Publish limitations, unresolved uncertainty, and human interventions.
Verify that a second authorized operator can reproduce the run.

Sources

PROV-Overview: An Overview of the PROV Family of Documents

World Wide Web Consortium · W3C recommendation · 2026-06-23T00:00:00Z

A formal vocabulary for provenance relationships among entities, activities, and agents.
OpenTelemetry Traces

OpenTelemetry · Official project documentation · 2026-06-23T00:00:00Z

Trace and span semantics used to correlate runtime, model, tool, and policy operations.
Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

National Institute of Standards and Technology · Government framework · 2026-06-23T00:00:00Z

Risk identification, measurement, governance, and lifecycle controls for generative AI systems.
OWASP Top 10 for LLM Applications 2025

OWASP Foundation · Security guidance · 2026-06-23T00:00:00Z

Threat categories for prompt injection, sensitive information disclosure, excessive agency, and insecure output handling.

Find runtime definitions and implementation guidance