Agent harness design: the model-access fabric

June 4, 2026

This is a companion to Agent harness design: trade-off analysis, which frames the agent platform as a small set of separable primitives connected by contracts. That overview names four primitives the ecosystem commonly agrees on, sandbox, persistence, identity, and the harness itself, and argues that two more deserve to be separated out. This post makes the case for the first of them: model access.

The claim is narrow and falsifiable. Model access is not a configuration line on the harness. It is a distinct ownership boundary with its own contract, its own failure modes, and, increasingly, its own dedicated services. The monolith treats it as scattered concerns, model capability is a harness assumption, the LLM gateway is a touch point, model lifecycle is a training pipeline, cost is an operational afterthought. That scattering is the tell. When one concern is smeared across four sections, it is usually because nobody has named the primitive that owns it.

The contract

A model-access fabric resolves exactly one request and a handful of administrative verbs.

Complete model X (or an alias), under this budget, attributed to this (tenant, agent, session, product), without handing me the provider key.

The admin surface around that request: rotate provider keys, set the approved-model list, cap spend, and kill a run. Everything else, where code runs, who the caller is, what the harness decides to send, belongs to a different primitive.

This is a clean contract in the sense the overview uses: it has one consumption shape, it has vendor-neutral implementations (LiteLLM [1], Portkey [2], Cloudflare AI Gateway [3], Kong AI Gateway [4], Helicone [5], and the hyperscaler routers AWS Bedrock and Google Vertex), and the production failures that matter cluster at its edges. Latere's Lux is one implementation of this contract; it is an illustration here, not the argument. The argument is that the contract exists and is separable.

Why it is not the identity fabric, and not the harness

Two boundaries have to be drawn precisely, because model access borrows from both.

Identity answers who is calling: it authenticates the principal, resolves the (user, tenant, agent, workload) tuple, and decides resource access. Model access consumes that answer (every usage record and every per-tenant budget is attributed through it) but it does not produce it. What identity never holds is a provider credential, a routing table, or a spend ceiling.

The harness decides what to send: it assembles the prompt, picks the tool set, manages the context window. Model access decides which model answers, on whose key, against which budget. A harness that embeds a provider key in its config has not absorbed the model-access fabric; it has skipped it, and inherited the failure modes below.

The cleanest way to see the seam is the question each primitive answers about a single LLM call. Identity: "is this caller allowed?" Harness: "what is the best thing to send?" Model access: "which model, on whose credential, within what spend, attributed to whom?" Three different owners, three different teams in a mature org, one call.

Credential custody is the secret-reachability problem at the model boundary

The overview's first axis is harness placement, and its load-bearing security property is secret reachability: generated code must not be able to read high-value credentials. Provider keys are exactly such credentials, and they are the ones an agent is most tempted to reach, because the agent's whole purpose is to call models.

So the model-access fabric inherits the placement argument rather than re-deriving it. The provider key lives in the fabric, never in the agent's environment, config, logs, or command history. The agent receives a scoped, short-lived handle, a Lux-style local connector inside the sandbox, or a gateway endpoint outside it, that the fabric exchanges for the real key at call time. The same reasoning that keeps OAuth tokens out of the sandbox keeps provider keys out of it.

This matters more for model keys than for most secrets because of blast radius. A leaked provider key is not just spend; it is unattributed spend, on someone else's models, with no per-tenant ceiling, discovered on the invoice. The placement discipline that the overview applies to the sandbox boundary applies with full force here.

Routing, aliasing, and version pinning

The fabric's second job is choosing the target. Three sub-decisions hide here, and each has a failure mode.

Provider routing and fallback. One endpoint, many providers, with automatic failover when a provider degrades. The benefit is obvious; the trap is subtle. Providers do not share tool-call syntax, token accounting, or stop-sequence behavior, so a fallback that silently swaps providers mid-session can change the model's behavior, not just its latency. The fabric has to treat a fallback as a capability change (see below), not a transparent reroute.

Model aliases. Application code should call default-coding-model, not claude-opus-4-8 or gpt-5.2. The alias is resolved by the fabric, so a model migration is a configuration change in one place instead of a code change in many. This is the same indirection the overview's agent-lifecycle axis argues for at the prompt layer, applied to the model binding.

Version pinning. "The model is swappable" is true at the protocol level and false at the behavior level. Chen, Zaharia, and Zou [6] documented GPT-4 dropping from 84% to 51% on a prime-identification task between two dated API versions, same endpoint, same prompts. A model-access fabric that does not pin versions is exposing every downstream agent to silent drift it cannot see. Pinning, and deliberately promoting pins behind an eval gate, is the fabric's answer.

Cost-aware routing. Easy requests to a cheap model, hard ones escalated. RouteLLM [7] reports an 85% cost reduction on MT-Bench while retaining most of GPT-4-level quality; a unified routing-and-cascading formulation [8] reports quality gains at materially lower relative cost. The catch is the classifier: a router that misjudges a hard task as easy produces worse output and sometimes a retry loop that costs more than the direct expensive call would have. Routing is leverage with a sharp edge.

Capability is consumed here, not owned here

There is one thing the model-access fabric deliberately does not own: model behavior. The overview keeps "model capability as a load-bearing assumption" inside the harness, and that is the right call. The fabric routes a request to a target; how that target behaves, tool-calling reliability, prompt sensitivity, multi-turn coherence, is the harness's dependency, surfaced across the model-access seam.

This distinction is what keeps the primitive clean. The fabric's responsibility is to make capability legible: to pin versions, to label which model actually served a call, and to refuse to swap a model silently. The harness's responsibility is to cope with whatever behavior it gets. Concretely, the literature the harness must plan around: tool-call accuracy degrades as the catalog grows (BFCL [9] and long-context function-calling studies); prompt phrasing alone can shift accuracy by up to roughly 13 points (Errica et al. [10]) and formatting by up to 76 points in few-shot settings (Sclar et al. [11]); top models lose an average 39% between single-turn and multi-turn (Laban et al. [12]). None of this is the fabric's to fix. All of it is the fabric's to make observable, by emitting, on every call, exactly which model and version answered.

The practical contract across the seam: a fabric may reroute or upgrade a model, but it must tell the harness it did, because the harness's prompts and tool schemas are tuned per model.

Attribution: the call that cannot be billed

Every LLM call should carry a verified (tenant, agent, session, product) tag from the moment it leaves the harness to the moment it lands in the cost pipeline. This is where model access consumes the identity fabric most heavily: without a trustworthy tenant on the call, the fabric can neither enforce a per-tenant cap nor bill the spend.

The failure is quiet and expensive. A platform that tags calls inconsistently ends up with one large undifferentiated model bill and no way to argue cost back to the tenants who produced it. Attribution is not a reporting nicety bolted on later; it is a property of the call itself, and it has to be present at emission, because it cannot be reconstructed afterward.

Budget enforcement has to be synchronous

Budgets enforced by the billing system are alerts, not enforcement: they fire after the spend has happened. Real enforcement is synchronous with the call, which means it lives in the fabric or at the gateway, the only points with both the per-call context and the authority to refuse.

The cost of getting this wrong is documented. A LangChain A2A pipeline looped between two agents for eleven days and produced a roughly $47,000 bill because neither agent had a budget ceiling [13]. A Claude Code recursion consumed about 1.67 billion tokens in five hours [14]. In both cases budgets existed; they just fired after the money was gone. The minimum the fabric owes: a hard per-session and per-tenant token ceiling that terminates the call with a structured error the agent can observe, plus admission control that rejects new sessions under cost pressure rather than accepting and timing out.

Prompt caching is a cost lever with a tenancy hazard

Identical prompt prefixes can be cached at the provider, and the savings are large: Anthropic prices cache reads at 10% of input cost (with a 1.25x write premium for the 5-minute tier and 2x for the 1-hour tier) [15]; an evaluation across long-horizon agentic tasks over 500+ sessions and three providers reports 41-80% cost reductions [16]; ProjectDiscovery reports 59-70% in production [17].

Two hazards make this the fabric's concern rather than the harness's. First, prefix drift: a system prompt that embeds a timestamp, a session ID, or a git SHA invalidates the cache on every call, silently turning a cost win into full price. Second, cross-tenant cache poisoning: a cached prefix that contains one tenant's data must never be served to another, which means cache partitioning is a tenancy decision, and tenancy is sourced from the identity fabric. Caching is where cost optimization and isolation meet, and the fabric is where that meeting is mediated.

Model lifecycle: the artifact is routable, the pipeline is not owned

Fine-tuning, distillation, and adapters produce new models. The overview's model-lifecycle axis is genuinely cross-cutting, the training pipeline touches persistence (the training data), identity (lawful basis for using it), and evaluation (the gate before promotion), so the fabric does not own it.

What the fabric does own is the result. A fine-tuned model is just another routable target with a version pin; a distilled model is a cheaper target the cost router can prefer; a per-agent LoRA adapter is a target selected by agent identity. The clean division: the pipeline that produces a model is cross-cutting and lives outside this primitive; the addressability and routing of the produced artifact is the fabric's, through the same contract as any other model. Preference-tuning results are contested enough (PPO has been shown to beat DPO on code and reasoning in at least one careful study [18], and domain-specific agent distillation reports gains that are often domain-bound [19]) that the fabric's job is not to have an opinion on the training method, only to route the resulting version behind an eval gate.

The gateway is a single point of failure

One fabric in front of all model inference is, by construction, a single point of failure for the entire platform. When it is down, every active session stalls at once. This is the cost of the primitive, and it has to be designed for from day one: a fallback route, a circuit breaker, and graceful degradation rather than a synchronous stall. The same centralization that makes attribution and budget enforcement possible makes availability a first-order concern. A fabric without a fallback plan has concentrated risk without earning resilience.

Where this leaves the human

The thread running through why Latere exists is that the most important intelligence in an autonomous system is the one you cannot see: the person who set the direction and drew the boundaries. The model-access fabric is where several of those boundaries live. Which models an agent may call, how much it may spend, whose credential pays, and which tenant is accountable are not decisions an agent should make for itself, and they are not decisions that should be buried in a key sitting in a config file where the trust decision was made once and never revisited.

Moving the control point back to model access keeps those decisions visible and revocable. A runaway run is stopped by narrowing its access, not by waiting for the invoice. A provider key never travels with the process that might leak it. The agent runs at full speed; the authority over what it may spend and which intelligence it may reach stays with a person. That is the same bet the rest of the platform makes, applied to the one resource every agent reaches for.

References

[1] LiteLLM. "LLM Gateway / Proxy Server." litellm.ai

[2] Portkey. "AI Gateway." portkey.ai

[3] Cloudflare. "AI Gateway." Cloudflare Docs

[4] Kong. "Kong AI Gateway." konghq.com

[5] Helicone. "LLM Observability and Gateway." helicone.ai

[6] L. Chen, M. Zaharia, J. Zou. "How is ChatGPT's Behavior Changing over Time?" Harvard Data Science Review, 2023. arXiv:2307.09009

[7] "RouteLLM: Learning to Route LLMs with Preference Data." 2024. arXiv:2406.18665

[8] J. Dekoninck et al. "A Unified Approach to Routing and Cascading for LLMs." ICML 2025. arXiv:2410.10347

[9] Berkeley Function Calling Leaderboard (BFCL). gorilla.cs.berkeley.edu

[10] F. Errica et al. "What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering." 2024. arXiv:2406.12334

[11] M. Sclar et al. "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design." 2023. arXiv:2310.11324

[12] P. Laban et al. "LLMs Get Lost In Multi-Turn Conversation." 2025. arXiv:2505.06120

[13] "The $47,000 Agent Loop: Why Token Budget Alerts Aren't Budget Enforcement." 2026. dev.to

[14] anthropics/claude-code. "Massive token consumption: 1.67B tokens in 5 hours." Issue #4095, 2025. GitHub

[15] Anthropic. "Prompt Caching." Anthropic Docs

[16] "Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks." January 2026. arXiv:2601.06007

[17] ProjectDiscovery. "Reducing LLM Costs with Prompt Caching." projectdiscovery.io

[18] S. Xu et al. "Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study." ICML 2024. arXiv:2404.10719

[19] "Agent Fine-tuning through Distillation for Domain-specific LLMs in Microdomains." October 2025. arXiv:2510.00482