A compiler and graph runtime turns a model program into executable work for one or more targets. It captures or imports a graph, represents it in intermediate forms, rewrites and partitions it, lowers operations, generates code or dispatch plans, and plans memory.
Key takeaways
- The compiler is responsible for transformation; the graph runtime is responsible for loading and executing the transformed program.
- Intermediate representations preserve enough semantics for optimization while progressively exposing hardware constraints.
- Shape dynamics, unsupported operators, data movement, and memory planning often determine portability.
Definition and scope
A framework produces operations and parameters. A compiler makes those operations executable efficiently on a target. A graph runtime coordinates executable partitions and execution providers. Some products combine ahead-of-time compilation, just-in-time specialization, graph execution, and hardware libraries in one package.
The boundary ends before network request management, tenant identity, tool authority, and durable application state. Those upper-layer responsibilities may consume compiler outputs but are not compiler features.
Compilation pipeline
- Import or capture: receive a framework graph, exported model, or dialect.
- Canonicalize: normalize equivalent forms, resolve constants, and expose dataflow.
- Infer and specialize: propagate shapes, types, layouts, and target capabilities.
- Rewrite and fuse: combine operations and remove redundant work.
- Partition: assign supported subgraphs to execution providers or devices.
- Lower: convert high-level operations to progressively lower-level representations.
- Schedule and tile: choose order, parallelism, memory locality, and vectorization.
- Generate executable artifacts: kernels, binaries, dispatch sequences, and metadata.
- Load and execute: bind inputs, allocate memory, dispatch, synchronize, and return outputs.
Intermediate representations
An IR is more than a file format. It defines which program properties remain visible to optimization passes. High-level IRs retain tensor and model semantics; mid-level IRs expose loops, layouts, and dataflow; low-level IRs express target instructions and memory. Multi-level systems such as MLIR use dialects and conversion passes so one program can be optimized at several abstraction levels. [ar_cite id=”mlir” label=”MLIR”]
Portability requires more than importing a model. The target must support required operators, dynamic shapes, data types, control flow, and memory. A runtime may fall back to a CPU provider or split the graph, which can preserve correctness while creating expensive transfers.
Partitioning and placement
Partitioning assigns graph regions to backends or devices. A good partition reduces transfers, keeps compatible operations together, and respects memory and capability constraints. Heterogeneous systems may combine CPU preprocessing, GPU attention, NPU vision, and custom accelerators. The compiler needs cost models and explicit affinities rather than assuming one preferred device.
Dynamic partitioning at request time is possible, but it complicates reproducibility and capacity planning. Record the target configuration, partition plan, compiler version, and fallback behavior with the deployment.
Memory planning
Static shapes allow lifetimes to be analyzed and buffers reused. Dynamic shapes require bounds, allocation strategies, or runtime specialization. Memory planning covers parameters, activations, temporary workspaces, constants, and transfers. It can determine whether a model fits at all and whether operator fusion actually improves performance.
Runtime execution
The graph runtime loads compiled artifacts, binds inputs and outputs, selects execution providers, allocates buffers, dispatches work, and synchronizes dependencies. It should expose errors that distinguish invalid model, unsupported target, allocation failure, backend failure, and numerical issue. These signals allow the serving layer to reject an incompatible deployment rather than discovering it during user traffic.
Failure modes
- Unsupported operator, data type, dynamic shape, or control flow
- Incorrect graph rewrite or numerical regression
- Partition that creates excessive device transfer
- Compilation time or artifact-size explosion
- Memory plan exceeding target capacity
- Runtime/backend version incompatibility
- Fallback path violating latency or privacy requirements
Selection guidance
Evaluate model coverage, target support, dynamic-shape behavior, quantization path, debugging, artifact reproducibility, licensing, and deployment footprint. Benchmark the complete graph with representative shapes and fallback disabled or explicitly measured. For heterogeneous deployment, verify each partition and transfer rather than relying on a general “supports GPU/NPU” claim.