Post-training infrastructure components

August 23, 2026

Post-training is usually drawn as a pipeline: mid-training, supervised fine-tuning, preference optimization, reinforcement learning, evaluation, serving.

The order is correct. The shapes are not. These stages have three different execution shapes, and the shape decides what you must build. This article lists the twelve components, the weight-synchronization decision, the load calculation, and the build order.

Three execution shapes

Loop 1 · offline training

closes once per run

Loop 2 · the RL hot loop

closes every optimizer step

Loop 3 · promotion

closes once per release

The three loops of post-training, drawn at their true cadence. The moving marker is one unit of work. Loop 2 closes thousands of times faster than the loops around it, and it is the only one that couples an inference server, a sandboxed executor and a trainer inside a single step.

Loop 1 is offline training. It contains continued pre-training, supervised fine-tuning, and offline preference methods such as direct preference optimization. The trainer reads a fixed dataset, computes gradients, and writes a checkpoint. There is no feedback inside the run. The job is long, schedulable, and restartable.

Loop 2 is the RL hot loop. Reinforcement learning with verifiable rewards runs here. At each optimizer step, the rollout engine samples completions from the current policy. The verifier scores each completion. The trainer computes one gradient step. The weights change, and the loop repeats. Three workloads are coupled at step granularity: an inference server, a sandboxed executor, and a trainer.

Loop 3 is promotion. It runs at release cadence. It converts a checkpoint into a model that callers can reach. It returns usage data to the next set of tasks and evaluations.

Loop 2 is not a larger loop 1. It is a different class of system, with a latency budget for each step. Most of the difficult problems below are in loop 2.

The twelve components

Artifacts

Base weights registry. Pin the base model to one revision. Verify it with a checksum. Keep it in storage that does not change. Upstream model repositories change and disappear. A base model that changes without a record makes the run unreproducible and the numbers incomparable.

Difficulty: size and staging latency. Hundreds of gigabytes must reach the training node before the first step.

Dataset store. Version each dataset and hash its content. Each version carries a mixture specification, a deduplication index, decontamination against the evaluation sets, and license and provenance records.

Difficulty: continuous integration must enforce decontamination. One leaked evaluation example increases every subsequent number. Detection comes months later, when an external benchmark disagrees with the internal one.

Checkpoint and artifact store. A checkpoint contains the model weights, the optimizer state, the RNG state, and the data-loader position. The last three are frequently omitted. They decide whether a resumed run repeats the trajectory or only produces a similar model.

Difficulty: checkpoint frequency competes with throughput. Retention competes with cost. The correct frequency depends on the mean time between node failures, which is unknown until the cluster has run for some time.

Model registry. The registry holds each promotable model and its lineage: base model, dataset version, code revision, configuration, seed, evaluation report, stage.

Difficulty: the registry works only as the sole path to serving. A registry that can be bypassed becomes a spreadsheet, and then no one can state which model serves traffic or how it was built.

Compute

Trainer. The trainer computes gradients. It controls the parallelism strategy, the precision, activation checkpointing, and optimizer-state sharding.

Full-parameter training with mixed-precision Adam holds bf16 weights, bf16 gradients, an fp32 master copy, and two fp32 moments:

M_{\text{state}} \;\approx\; \underbrace{2N}_{\text{weights}} + \underbrace{2N}_{\text{grads}} + \underbrace{12N}_{\text{fp32 master} + m + v} \;=\; 16N \text{ bytes}

An 8B model therefore holds 128 GB of persistent state before any activations. ZeRO-style sharding and FSDP are consequently required from the start, not added later.

Rollout engine. The rollout engine is an inference server that generates samples from the current policy. It does not exist in loop 1. In loop 2 it consumes most of the wall-clock time.

A FLOP count predicts the opposite. One training step costs approximately $6N$ FLOPs per token for the forward and backward passes. Generation costs approximately $2N$ FLOPs per token.

Generation is nevertheless the slower half. It is autoregressive and memory-bandwidth-bound. Each token reads the full weight matrix and the KV cache, and the KV cache grows. Generation cannot be parallelized across the sequence; a training forward pass can. PagedAttention and continuous batching reduce the gap by increasing batch occupancy. The rollout engine is therefore a production serving deployment with a throughput target.

Difficulty: weight synchronization. See the next section.

Verifier. The verifier converts one completion into a scalar reward. Three families, with different economics:

Family	Cost	Signal	Weakness
Executable tests (code)	low, seconds	high	coverage is limited; a wrong solution can pass the tests
Answer checkers (math)	very low	high	the work is parsing and symbolic equivalence
Model critics	one inference call	rich, general	a learned proxy, so the policy can exploit it

Difficulty: the verifier executes untrusted adversarial code at rollout throughput, and it must be a true isolation boundary because the policy optimizes against it. Throughput and isolation pull in opposite directions.

Orchestrator. One command requests accelerators, syncs the code and a pinned dataset version, sets the environment, starts the stage, streams the logs, writes the artifacts back, and handles retry and resume.

Difficulty: the same command must run on a workstation, a rented node, and a reserved block, so no platform change is needed between tiers. Without it, an operator drives every run by hand, which limits the experiment count.

Control

Experiment tracking and training metrics. These are not service telemetry. Record loss, gradient norm, tokens per second, and model FLOPs utilization. For reinforcement learning, also record reward mean and variance, KL divergence to the reference policy, policy entropy, and mean response length.

Three of these report failure while the loss curve still looks correct:

KL divergence to the reference increases. The policy is moving off the reference distribution and will collapse.
Entropy falls to near zero. The policy stopped exploring and emits one memorized form.
Response length increases, reward does not. The policy found that length correlates with reward and adds unnecessary content.

Evaluation harness and gate. Use held-out suites with contamination control and confidence intervals, plus the rule that permits promotion. Suites cover capability, agentic and trajectory behavior, safety, regression guards, and reward-hacking probes.

Difficulty: small-sample noise. On 50 cases, four points is usually not a real difference. A gate that promotes on it promotes noise indefinitely. The confidence interval is the gate.

Cost and budget control. Attribute GPU-hours and currency to each run. Compare each run to a budget for its stage. Block a launch whose projection exceeds the budget. A self-improvement loop starts its own training runs with no person in the path.

Serving path. This component publishes weights, deploys the engine, reports health and readiness, registers the model with the gateway, autoscales, and rolls back. It is the largest recurring cost and the only component that runs continuously.

Weight synchronization

The trainer holds the authoritative weights. The rollout engine holds a copy. After each optimizer step the copy is out of date. The solution decides the physical layout of the system.

1. Colocated2. Disaggregated3. Asynchronous

One GPU pool. The trainer and the rollout engine take turns holding memory.

trainer

rollout engine

one optimizer stepnext step →

The two lanes are never busy at the same time. Cost: the phases serialize, and each offload and reload adds idle time. Utilization is still best at small scale, because no weights cross the network.

Two GPU pools. The trainer broadcasts the weights across the fabric at each step.

trainer

rollout engine

one optimizer stepnext step →

Both lanes run at once, and the rollout side can be scaled on its own. Cost: the broadcast — 16 GB of bf16 weights per step needs a fast interconnect.

Two GPU pools. The rollout engine runs ahead on weights that are a few steps old.

trainer

rollout engine

one optimizer stepnext step →

No gap and no bubble: utilization is highest. Cost: the samples are off-policy, so the algorithm needs an importance correction, and more failures present as slow convergence rather than as crashes.

computingidleweights moving

Device occupancy across one optimizer step. Select a layout to compare. The layout decides where the GPUs go, so it is an architecture decision, not an implementation detail.

Colocated. The trainer and the rollout engine share the GPUs and exchange which one holds memory. No weights cross the network. Utilization is best at small scale and operation is simplest. The HybridFlow design in verl is the reference implementation. Cost: the two phases serialize, and each offload and reload adds idle time.

Disaggregated. The two components use separate GPU pools. The trainer broadcasts the weights at each step. Each side scales independently, and the rollout side is the side that needs capacity. Cost: a broadcast of 16 GB of bf16 weights per step requires a fast interconnect.

Asynchronous. The rollout engine runs ahead on weights that are slightly out of date. Utilization is highest and there is no bubble. Cost: the samples are off-policy, so the algorithm needs an importance correction, and more failures present as slow convergence rather than as crashes.

Start colocated. Move to disaggregated after measuring that generation dominates and by how much. Treat asynchronous as a later optimization with an algorithmic cost.

Load calculation

Take a moderate configuration. Each step uses $P = 256$ prompts and $G = 8$ samples per prompt. GRPO normalizes its advantage against this group; RLOO uses the same group for its leave-one-out baseline. Each sample generates $L = 2048$ tokens.

T_{\text{step}} = P \cdot G \cdot L = 4.2 \times 10^6 \text{ tokens} \qquad V_{\text{step}} = P \cdot G = 2048 \text{ verifications}

At a 60-second step, the system sustains 34 verifications per second. Each verification isolates and executes untrusted code under a timeout. A 1,000-step run needs 2 million sandboxed executions and 4.2 billion generated tokens, for one experiment.

Two results follow.

RLVR is bound by rollout and by reward, not by gradients. The engineering is in the inference server and the sandbox. A larger trainer does not move the bottleneck, and neither does a better optimizer.

The verifier is a throughput system, not a test runner. This is the most frequent sizing error. A sandbox that executes a few tasks concurrently is not a small version of loop 2. Its shape is wrong by three orders of magnitude, and configuration does not close the gap. Capacity architecture closes it: pooled workers, sub-second cold start, and a scheduler that treats a verification as a request rather than a deployment.

Build order

The build order is not the pipeline order.

Checkpoint store and training metrics, before buying GPU time. They make a run resumable and legible. Accelerator time bought first pays for runs that cannot be trusted, restarted, or compared.
Verifier capacity, before rollout capacity. The verifier sets the throughput floor. A rollout engine feeding a slower verifier is an idle GPU.
Rollout engine and synchronization layout, together. They are one decision, and it determines whether to buy one GPU pool or two.
Orchestrator last in the compute plane. Built after the run shape is known, it encodes facts. Built first, it encodes an assumption.

Serving and cost control follow a separate track, driven by demand. A serving deployment that runs continuously costs more per month than an early training program.

Build or buy

Trainers, sharding strategies, inference engines, and RL frameworks are mature and improve faster than an in-house equivalent. Adapt them.

The verifier is the exception, for a structural reason. A learned reward model is a proxy, and the policy finds the maximum of the proxy rather than of the objective. A verifier removes that path: the policy cannot argue its way to a passing test suite. The property holds under two conditions. The verifier is independent of the output it scores. The verifier runs at rollout throughput inside a real isolation boundary, while hostile code attempts to escape it.

The same two conditions apply to runtime verification one layer up, which is where our existing components sit: sandboxed execution as a metered service in Cella, and metered model access that retains custody of credentials in Lux.