Agent harness design: the cross-cutting contracts

June 7, 2026

This is a companion to Agent harness design: trade-off analysis, which frames an agent platform as a small set of separable primitives connected by contracts. The hub names six primitives, sandbox, persistence, identity, model access, the harness runtime, and governance. This post is about the seventh thing that the hub keeps insisting is not a primitive: the contracts between them.

The claim is that the failures which actually take production platforms down rarely live inside a primitive. They live at the seams. A primitive can be individually correct, well-tested, and well-isolated, and the platform still fails because two of them disagree about ordering, identity, or who owns a side effect. These seam failures share a signature: no single team owns them, each owning team assumes the other handled it, and the gap is invisible until an incident makes it visible. This post takes five of those seams and goes deep on the mechanism of each: concurrency coordination, the operational failure modes that are chronically under-scoped, transport adaptation, observability, and evaluation.

None of these is a primitive. Each is a contract that crosses primitive boundaries, and each is where the boundary-drawing has to be explicit or it will be wrong.

Concurrency: the lock you have is not the lock you need

The first seam is the gap between two kinds of coordination that look identical and are not.

A harness almost always has session-level locking: it prevents two concurrent Run() calls from mutating the same session at once. This is necessary, and it is also the wrong scope for the conflict that matters. Session-level locking says nothing about two different sessions of the same agent touching the same external resource at the same time.

graph TB
    classDef locked fill:#d5f5e3,stroke:#27ae60
    classDef unlocked fill:#f9d6d6,stroke:#c0392b

    A[Agent: acme-api] --> S1[Session 1<br/>sandbox A]
    A --> S2[Session 2<br/>sandbox B]
    A --> S3[Scheduled tick<br/>sandbox C]

    S1 -->|git push main| GIT[git repo: acme/api]
    S2 -->|git push main| GIT
    S3 -->|create ticket| JIRA[Jira: ACME project]
    S1 -->|create ticket| JIRA

    class S1,S2,S3 locked
    class GIT,JIRA unlocked

    style GIT fill:#f9d6d6,stroke:#c0392b
    style JIRA fill:#f9d6d6,stroke:#c0392b

The diagram makes the seam visible. Every session in green is individually locked. Every shared resource in red is unlocked. Two coding sessions for the same agent push conflicting commits to the same branch. Two conversational sessions post duplicate responses to the same thread. A user-initiated session and a scheduled tick for the same hybrid agent target the same external state, and neither knows the other exists. The lock is in the harness; the conflict is at the resource, and the harness's lock does not reach it.

This is not a hypothetical conflict surface. Ogenrwot and Businge's AgenticFlict [1] filtered the 932,791-PR AIDev dataset down to 142,652 agentic pull requests and measured a 27.67% textual merge-conflict rate. Roughly one in four AI-generated pull requests produces a conflict, with an average of 540 conflicting lines per PR. The reason this rate is so much higher than human baselines is structural rather than incidental: agents working in parallel have no shared model of what the others are doing, and the harness's session lock gives them none, so they collide on shared state exactly because each one believes it is alone.

The options, and what each actually buys.

• Worktree-per-session. Give each session a dedicated branch or worktree and merge through a pull request. The conflict still exists, but it is moved to a place that has machinery for resolving it (the merge, the review) instead of happening as a silent overwrite at push time. Geng and Neubig (CMU) [2] measured this directly: worktree isolation outperformed instruction-level (soft) isolation on coding agents, 59.1% against 56.1% on Commit0-Lite. The same study showed why soft isolation is not a substitute. Telling an agent in its prompt to stay in its lane can beat a single-agent baseline on some benchmarks (53.1% to 56.1% on Commit0-Lite) and fall below it on others (55.5% against 57.2% on PaperBench). Instruction-level constraints are advisory; the model may or may not honor them, and you find out which after the conflict. Real isolation does not depend on the model's compliance.

• Resource-level distributed locks. Before a session performs a well-typed operation on a shared resource, for example a git push to a named branch, it acquires a lock keyed on that resource. This works precisely when the conflict surface is sharp and nameable. It does not help when the conflict is fuzzy, two sessions editing overlapping but non-identical regions of the same files, because there is no clean key to lock on.

• Optimistic concurrency with reconciliation. Let the collision happen and resolve it on the external system: merge commits in git, deduplication on the ticketing system. This is the cheapest to implement and the most expensive to operate, because it shifts all the complexity onto the recovery path, where it is hardest to test and most likely to be exercised first in production.

• Agent-level coordination. The conflicts above are not accidents of timing; they are structural properties of running multiple agents against shared state. MAST [3] formalizes inter-agent misalignment and coordination failure as first-class failure categories, and MultiAgentBench [4] formalizes coordination-protocol evaluation across star, chain, tree, and graph topologies. The takeaway for the seam is that coordination has to be designed at the agent level, not patched at the session level, because the session lock is the wrong altitude for a multi-agent conflict.

The right choice depends on how strongly the workload's external effects conflict, and on whether the downstream system already has its own concurrency control that can be leaned on. Git's ref updates and most databases' row locks are real coordination primitives; where they exist, the harness can defer to them. Where they do not, the harness owns a coordination problem it cannot see from inside a single session.

Operational failure modes that are usually under-scoped

The architectural choices get the attention. The failure modes in this section get the post-mortems. Each is documented-in-principle and under-addressed in practice, and each has public incident evidence. They are seam failures because each one falls between two owners: the harness assumes the platform handles it, the platform assumes the harness does.

graph TB
    classDef risk fill:#f9d6d6,stroke:#c0392b

    GW[LLM gateway down] -->|all sessions stall| STALL[No fallback or circuit breaker]
    K8S[K8s capacity exhausted] -->|session creation hangs| HANG[No admission control or timeout]
    LOOP[Runaway agent loop] -->|unbounded LLM spend| COST[No hard timeout for request-driven sessions]
    GROWTH[Event log growth] -->|Postgres bloat| PERF[No storage compaction or partitioning]

    class GW,K8S,LOOP,GROWTH risk

The gateway-as-single-point-of-failure case in the top-left of that diagram is real, but it is owned by the model-access seam and treated in depth from the hub. In short: one fabric in front of all inference stalls every active session at once when it goes down, and the answer is a fallback route, a circuit breaker, and graceful degradation designed in from day one rather than retrofitted after the first outage. The trade-off hub carries the incident evidence. The other three failure modes are the focus here.

Sandbox scheduling exhaustion. Session creation that calls the Kubernetes API does not actually create a sandbox; it requests one. Between the request and a running pod sits the cluster's scheduler, and the scheduler can refuse: resource quota exhausted, no node with capacity, a scale-up pending that has not landed yet. The dangerous failure here is not the refusal, it is the shape of the refusal. The common implementation hangs. The pod sits Pending, the harness waits for it, and the SSE stream to the client never starts. There is no error to surface because nothing errored; admission simply never happened. The fix is admission control at the harness layer, where the harness checks capacity signals before it promises the client a session and returns an explicit rejection when the cluster cannot honor the request. The distinction that matters is between a fast, legible "no" and a slow, silent nothing. The second is worse, because the client cannot retry intelligently against a hang.

Runaway request-driven sessions. There is an asymmetry in how harnesses bound execution. Scheduled agents almost always carry a timeout_per_tick, because a recurring job that never ends is an obvious design smell that gets caught early. Request-driven sessions frequently carry no equivalent wall-clock bound, because the mental model is "a user asked, the agent answers, it finishes." A coding agent stuck in a test-fix loop violates that model: it does not finish, it iterates, and each iteration is another model call. The public evidence for how far this goes is stark. A LangChain A2A pipeline [5] looped between two agents for eleven days and produced roughly a $47,000 bill, with the post-mortem noting neither agent had a budget ceiling. A Claude Code recursion incident [6] consumed about 1.67 billion tokens in five hours. The detailed treatment of budget enforcement, why a per-tenant ceiling has to be synchronous with the call rather than reconstructed from billing, belongs to the model-access seam and is reachable from the hub. The harness-side control is narrower and complementary: a hard wall-clock timeout on every session regardless of trigger, token budgets enforced at the harness layer rather than discovered in the invoice, and admission control that can reject new sessions when the platform is already under cost pressure. The reasoning behind the wall-clock bound specifically is that it is the one limit that holds even when the token accounting is wrong or lagging, because wall-clock time is the one quantity the harness can measure locally without trusting any downstream meter.

Scheduler retry is not exactly-once side effects. This is the subtlest of the four because the obvious defense looks sufficient and is not. An agent-level distributed lock stops two replicas from firing the same scheduled tick concurrently. That solves overlap. It does not solve replay. Consider the sequence inside a single tick: the agent posts to Slack, opens a Jira ticket, writes a row to a database, and only then persists the fact that the tick completed. If the replica crashes in the window between emitting a side effect and recording that the tick finished, recovery sees an unfinished tick and runs it again. The lock was never violated, exactly one replica ran the tick each time, and yet the Slack message is posted twice. The mechanism is that the lock coordinates who runs, while exactly-once requires coordinating what already happened to the outside world, and those are different problems. Locking does not make a side effect idempotent. The only durable fix lives at the side-effect call sites: idempotency keys carried into Slack, Jira, and the database so a replayed tick is recognized and absorbed rather than duplicated. That is work the external-tool seam owns, not the scheduler, which is exactly why it falls through.

Session-store growth. An append-only event log is the cleanest way to record session history and the easiest way to bloat a database, because "append-only" means it has no compaction at the storage layer. There is a naming collision worth being precise about: compaction in the context-management sense is an LLM feature that shrinks the prompt, and it does nothing for the storage table. A high-throughput agent that emits many events per turn grows its backing table without bound. Azguards [7] measured LangGraph's append-only Postgres checkpointing at modest scale, 100 concurrent agents, and found roughly 150 MB per second of write-ahead-log generation with 3 to 5 second replication lag, against under 100 ms with optimization. Their Pointer State Pattern cut checkpoint size by 99.8%. The reason this is a seam and not just a persistence detail is that the harness decides how much to write per turn while persistence owns where it lands, and neither side alone sees the steady-state growth curve. Partitioning, archival, or storage-level TTL is load-bearing, not optional, and it has to be designed before the table is the problem, because retrofitting a retention policy onto a table that is already the bottleneck is its own incident.

Transport adaptation: where the outside world's identity meets the harness

The harness speaks one internal language: sessions, events, tool calls. The outside world speaks many, HTTP, Slack events, email, webhooks, A2A [8], MCP, message queues. Something maps between them, usually called the transport or adapter layer, and that mapping is a contract with several primitives at once, not a transport-internal detail.

The three shapes of the adapter layer.

• Single-transport harness. The harness exposes exactly one inbound protocol, almost always HTTP, and everything else is the caller's problem. It is the simplest thing to deploy and the right default for an internal-only tool. The cost is that every integration partner must build an HTTP client, and there is no Slack, email, or A2A path without an external translator that someone else now owns.

• Pluggable transport adapters. The harness core is transport-agnostic, and adapters translate external events into a stable set of session operations. A Slack event becomes a resume on the session mapped from thread_ts; an A2A task becomes a create with the A2A task_id as the external correlation key. This is strong for heterogeneous integrations and the abstraction pays for itself the moment a second transport exists. Before that, it is overhead. The requirement it imposes is a stable core session-operations contract that all adapters target, which is precisely the contract that, if it leaks transport-specific assumptions, stops being reusable.

• Transport-native harnesses. A separate harness binary per transport, each with its own session model. It works for a small surface and duplicates the hard parts, durability, recovery, policy, once per transport. It is usually a sign of organizational divergence more than a deliberate design choice, because the duplicated parts drift independently and the platform ends up with N slightly different notions of a session.

Client observation is a separate axis from event arrival. However events come in, clients have to watch a session progress, and the four common patterns trade off differently:

SSE [9] is the usual default for browser clients, webhooks for server-to-server integrations, and polling the fallback when SSE-hostile proxies sit in the path. The non-obvious requirement is multiplexing: the same session visible simultaneously over SSE to a UI and over a webhook to a back-office system. That is common, and it is non-trivial, because the two observers can be at different cursor positions and the harness has to serve both from one authoritative event stream.

The failure modes are all seam failures.

• ID-mapping loss. The transport's native identifier, Slack thread_ts, A2A task_id, email Message-ID, must map deterministically to the harness session_id. When that mapping is lost or ambiguous, the harness either spawns a duplicate session for an existing conversation or, worse, bleeds one thread's state into another. The mapping is the join key between the transport and persistence, and a non-deterministic join key corrupts both sides.

• At-least-once versus exactly-once. Webhooks retry by design. If the adapter writes to the session on every retry, the same logical event lands multiple times unless the adapter or the session store deduplicates. This is the same exactly-once problem the scheduler section described, arriving from a different direction: the transport guarantees at-least-once delivery, the session store wants each event once, and the gap between those two guarantees is the adapter's to close.

• Out-of-order events. Slack and email can deliver events out of order. The adapter either enforces ordering, paying a latency cost to buffer and sort, or the harness accepts unordered events and pays a state-reconstruction cost to make sense of them. There is no free option; the choice is which cost to pay.

• Transport auth resolved to a canonical user. Each transport carries its own identity model: a Slack user ID, an email sender domain, an A2A agent card. None of those is the harness's notion of who the user is. Every adapter must perform a bounded step, "trust this transport's assertion, then ask the identity fabric to resolve it to (user_id, tenant_id, scopes)," before any session operation runs. This is a contract with the identity primitive, not a transport-internal concern, and getting it wrong is a privilege-escalation bug, not a usability bug: if the contract is implicit or unverified, a leaked Slack token or a spoofed email header grants harness authority directly. The corollary is that the Slack-to-canonical and email-to-canonical mappings must live in the identity fabric, not in adapter-local tables, so that revocation and right-to-erasure cascade correctly. A mapping stored in the adapter is a mapping the identity fabric cannot revoke.

Observability: two regimes, and the AI-specific one is the hard one

Agent observability splits into two regimes that get conflated and should not be. Infrastructure observability, CPU, memory, pod health, request latency, is the same problem as any Kubernetes service and is well-served by existing tooling. Agent-behavior observability, what the model decided, why, and whether the decision was good, is AI-specific and needs dedicated infrastructure. It is a seam because every primitive emits telemetry and audit, and the value only appears when those streams correlate.

The infrastructure layer is Prometheus and Grafana for metrics, OpenTelemetry for traces, Loki or Elastic for logs. The one AI-specific twist at this layer is cost attribution: token spend per agent, per session, per user, per tenant. That is derivable from the model gateway's logs, but only if the gateway emits the identity tags, which is itself a contract with the identity primitive. Without the tags at emission time, the attribution cannot be reconstructed afterward.

The agent-behavior layer is where the standards are still settling. OpenTelemetry has published GenAI semantic conventions [10], still marked experimental as of 2026, with span names of the form gen_ai.{operation}, for example chat and execute_tool [11]. A GenAI agent-and-framework sub-spec [12] was added in 2025 specifically to cover multi-agent trajectories, which the base spec did not. Vendor implementations exist across Langfuse [13] (with an OTLP endpoint since February 2025 [14]), LangSmith [15], Arize Phoenix [16], Helicone [17], and Braintrust [18], and they converge on capturing the full conversation (user turns, model responses, tool calls, tool results), per-span latency and cost, trace correlation across multi-agent and multi-tool flows, and replay, re-running a historical session against a different prompt or model.

The trade-offs are where the seam bites.

• Sampling versus full capture. Full capture of every prompt and response runs to hundreds of gigabytes per day at modest scale. Sampling at 1 to 10% is far cheaper and loses precisely the rare failure the operator most wants to inspect, because rare failures are, by definition, the ones a uniform sampler drops. The standard compromise inverts the sampling against severity: full capture of errors and flagged sessions, sampled capture of the successful ones. The reasoning is that the value of a trace is not uniform, so the sampling rate should not be either.

• PII in traces. Traces contain prompts, prompts contain user data, and so trace storage silently becomes a data-residency and right-to-erasure surface. Automated redaction with tools such as Presidio exists, and it is imperfect and adds latency, so it is a mitigation rather than a guarantee. The seam here is that observability stores data the audit and compliance owners are accountable for, and the hub treats that data-governance side in its own right.

• Cardinality. Tagging spans with session_id, user_id, tool_name, and model_version is exactly what makes a trace useful and exactly what produces high-cardinality metrics. Prometheus chokes on this; dedicated trace stores absorb it at higher cost. The tags you need for behavior analysis are the tags that break time-series storage, which is why behavior observability cannot just be bolted onto the infrastructure metrics stack.

• Trace lag. Exporting traces synchronously blocks the harness on the trace backend; exporting asynchronously risks losing the buffer on a crash. Most vendors default to buffered async with a bounded buffer, which is the pragmatic middle and an explicit decision to lose some traces under failure rather than slow every request.

What operators actually need out of all this is a short list of behavior signals, not raw spans: tool-call success rate per agent and per tool trended over time (the primary early signal for model drift), turn count per task as an efficiency proxy, cost per resolved task rather than cost per call, compaction events and the fraction of session preserved, and a failure-mode taxonomy, tool errors, context overflow, permission denied, user abandonment, where a per-type trend line is more actionable than an aggregate SLO. The cost-per-resolved-task metric is the one most often missing, because cost per call is trivially available from the gateway and cost per outcome requires joining the gateway's spend to the harness's notion of task completion, which is, again, a seam.

Evaluation: the gate that the rest of the platform leans on

An eval suite is to an agent what a test suite is to a library. Without it, every prompt change and every model swap is a wager. With it, regressions are caught before users see them. Evaluation is a seam because the suite gates rollout for the harness, consumes traces from observability, enables canaries for the agent lifecycle, and validates swaps for model access; it sits in the middle of all of them.

The eval types form a ladder from cheap and shallow to expensive and real.

• Unit-level, tool-call correctness. Does the agent call the right tool with the right arguments for a known task? BFCL [19] is the standard public benchmark and the methodology most in-house suites mirror for custom tools.

• Trajectory-level, multi-turn. Can the agent complete a multi-step task end to end? The benchmark landscape moves fast enough that the headline numbers are the point: SWE-Bench [20] Verified now exceeds 93% on top models, which means it has stopped discriminating, so the harder SWE-Bench Pro [21] (which drops top models to roughly 23%) is the current signal-bearing coding benchmark. OSWorld [22] covers computer use, WebArena [23] covers web tasks, and METR's [24] time-horizons framing now has concrete numbers, GPT-5 at roughly a 2h17m 50%-horizon, doubling about every four months across 2024 and 2025. The operational lesson is not any single score; it is that a benchmark stops being useful the moment top models saturate it, so the eval suite needs a refresh policy.

• Quality judgment, LLM-as-judge. An LLM scores outputs against a rubric: cheap, scalable, and biased. It favors verbose outputs and struggles with correctness judgments [25]. Position-bias work [26] finds the bias is strongly driven by the quality gap between candidate answers, and the CALM framework [27] catalogs a dozen distinct judge biases. The practical consequence is to use LLM-judge for style and tone, where its biases are tolerable, and automated metrics for correctness, where they are not.

• A/B testing in production. Route a fraction of traffic to a variant and compare quality metrics. This is ground truth for real user behavior and it is slow and expensive, and it needs a statistical framework to avoid false positives. It is the only method on this list that sees the signal users actually generate.

• Shadow mode. Run a new prompt or model alongside production, log both outputs, compare offline. Risk-free for the user and expensive, it doubles inference cost, and it cannot see user-facing signal like click-through or task abandonment because no user ever sees the shadow output.

The CI/CD integration is what turns evals from a research artifact into a gate.

graph LR
    classDef gate fill:#fdebd0,stroke:#e67e22

    DEV[Prompt / tool change] --> OFFLINE[Offline eval suite]
    OFFLINE --> GATE1{Passes?}
    GATE1 -->|yes| SHADOW[Shadow deploy]
    SHADOW --> COMPARE[Compare vs. baseline]
    COMPARE --> GATE2{No regression?}
    GATE2 -->|yes| CANARY[Canary: 1-5%]
    CANARY --> GATE3{Quality holds?}
    GATE3 -->|yes| FULL[Full rollout]

    class GATE1,GATE2,GATE3 gate

The vendor-neutral pattern is consistent across Promptfoo [28], Inspect AI [29], Patronus [30], and Ragas [31] for RAG specifically: a golden dataset in git, an eval runner in CI, results published to a dashboard, and the merge gated on a threshold. The pattern is simple; the failure modes are where the difficulty lives.

• Eval overfitting. Prompts tuned to pass the suite stop generalizing to real users, for the same reason a model that memorizes the test set scores well and learns nothing. The mitigation is a held-out set that is rotated, so that passing the visible evals cannot be the whole of the optimization target.

• Static evals for dynamic tasks. A trajectory eval run against a frozen environment, a snapshot repository, a recorded API, cannot catch changes in the live dependencies the agent actually faces. A green eval suite against a frozen world is a statement about the frozen world, not the live one.

• LLM-judge bias. The judge's own failures, verbosity bias, style bias, self-preference, leak directly into the scores. The defense is periodic human cross-validation, because a biased judge that is never checked against humans will confidently certify its own preferences as quality.

• No trajectory replay on real sessions. The single most useful eval is replaying last week's production sessions against a proposed change, and it is the one many systems cannot run, because it depends on trace infrastructure actually capturing the trajectories, which the observability seam either built or skipped. Record-and-replay of agent traces is an active research direction [32], usually framed around reusing past experience rather than evaluation replay, but the infrastructure dependency is the same. This is the cleanest illustration of why these are contracts and not primitives: the eval suite cannot replay what observability did not capture, and observability has no reason to capture trajectories unless eval consumes them. Each is useless to this purpose without the other, and the value is in the contract between them.

• Benchmark contamination via harness exploitation. Berkeley RDI [33] showed that top agent benchmarks can be gamed by exploiting the benchmark harness itself, reading answers from git log, reward-hacking the grader, with several benchmarks hitting near-100% without the agent solving the tasks at all. The unsettling part is that the exploit lives in the harness, not the task, so the defense is hardening the eval harness and using third-party evaluation, not just keeping a private held-out set. A held-out set protects against memorizing answers; it does nothing against an agent that reads the grader.

Where this leaves the human

The thread running through why Latere exists is that the most important intelligence in an autonomous system is the one you cannot see: the person who set the direction and drew the boundaries. The seams in this post are where those boundaries are easiest to lose, precisely because no single primitive owns them. A coordination conflict between two sessions, a tick replayed after a crash, a Slack token quietly resolved to more authority than it should carry, a trace that captured a user's data nobody scheduled for erasure, an eval that certified a change against a frozen world: each is a place where authority and visibility leak out through a gap that every individual team assumed someone else was watching.

Making these contracts explicit is the same bet the rest of the platform makes, applied to the connective tissue. A side effect that carries an idempotency key stays attributable when it replays. A transport assertion resolved through the identity fabric stays revocable. A trajectory captured by observability stays replayable by evaluation. The agent runs at full speed across all of them; the authority over what crosses each seam, and the visibility into what already did, stays with a person. The failures cluster at the seams because that is where ownership is ambiguous. Naming the contract is how you make it un-ambiguous, and how you keep the human in the one position from which all of it is still legible.

The evaluation seam in this post is where Latere's Adversarial Review lives, adversarial verification as a first-class step rather than an afterthought; the hub maps where the other seams, several of them still unowned, stand.

References

[1] D. Ogenrwot, J. Businge. "AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub." 2026. arXiv:2604.03551

[2] J. Geng, G. Neubig. "Effective Strategies for Asynchronous Software Engineering Agents." CMU, 2026. arXiv:2603.21489

[3] M. Cemri et al. "Why Do Multi-Agent LLM Systems Fail?" NeurIPS 2025 Datasets and Benchmarks Track. OpenReview

[4] "MultiAgentBench: Evaluating Coordination Protocols in Multi-Agent LLM Systems." ACL 2025. ACL Anthology, arXiv:2503.01935

[5] DEV Community. "The $47,000 Agent Loop." 2025. dev.to

[6] anthropics/claude-code. "Massive token consumption: 1.67B tokens in 5 hours." Issue #4095, 2025. GitHub

[7] Azguards. "The Checkpoint Bloat: Mitigating Write-Amplification in LangGraph Postgres Savers." 2025. Azguards

[8] Google, Microsoft et al. "Agent-to-Agent (A2A) Protocol." 2025. a2aprotocol.org

[9] WHATWG. "Server-sent events." HTML Living Standard. html.spec.whatwg.org

[10] OpenTelemetry. "Semantic Conventions for Generative AI." OpenTelemetry

[11] OpenTelemetry. "Generative AI Spans." opentelemetry.io

[12] OpenTelemetry. "Generative AI Agent and Framework Spans." 2025. opentelemetry.io

[13] Langfuse. "Open-source LLM Engineering Platform." langfuse.com

[14] Langfuse. "OpenTelemetry OTLP Endpoint." February 2025. langfuse.com

[15] LangSmith. "LLM Application Development, Monitoring, and Evaluation." smith.langchain.com

[16] Arize. "Phoenix: Open-source ML Observability." arize.com/phoenix

[17] Helicone. "LLM Observability and Gateway." helicone.ai

[18] Braintrust. "LLM Evaluation and Observability." braintrust.dev

[19] F. Yan et al. "The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models." ICML, 2025. OpenReview

[20] C. Jimenez et al. "SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024. arXiv:2310.06770

[21] Scale. "SWE-Bench Pro: A Harder Benchmark for Coding Agents." 2025. scale.com

[22] T. Xie et al. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments." NeurIPS 2024. arXiv:2404.07972

[23] S. Zhou et al. "WebArena: A Realistic Web Environment for Building Autonomous Agents." ICLR 2024. arXiv:2307.13854

[24] METR. "Measuring AI Ability to Complete Long Tasks." 2025. metr.org

[25] A. Panickssery et al. "LLM Evaluators Recognize and Favor Their Own Generations." 2024. arXiv:2404.13076

[26] "Position Bias in LLM Judges: An Empirical Study." IJCNLP 2025. ACL Anthology

[27] "CALM: Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge." 2024. arXiv:2410.02736

[28] Promptfoo. "Open-source LLM Testing and Evaluation." promptfoo.dev

[29] UK AI Safety Institute. "Inspect: A Framework for Large Language Model Evaluations." inspect.aisi.org.uk

[30] Patronus AI. "Automated Evaluation for LLMs." patronus.ai

[31] Ragas. "Evaluation Framework for Retrieval-Augmented Generation." docs.ragas.io

[32] "Get Experience from Practice: LLM Agents with Record & Replay." 2025. arXiv:2505.17716

[33] Berkeley RDI. "Trustworthy Benchmarks for Agents." 2025. rdi.berkeley.edu