Circuit Breakers

Wallfacer uses circuit breakers to automatically pause automation when something goes wrong, preventing cascading failures. They self-heal once the problem resolves — no manual intervention required.

There are two independent systems: watcher breakers (per-automation) and the container breaker (runtime-level).

Watcher Breakers

Each automation watcher (Promote, Retry, Test, Submit, Catch Up, Refine) has its own circuit breaker. When a watcher encounters repeated errors, its breaker opens and suppresses that watcher while leaving all others running.

What you see

The header shows a Circuit Breakers section. When all watchers are healthy, it displays a green dot with "All healthy." When a breaker trips, a red indicator appears showing:

• The affected watcher name (e.g. "Promote", "Retry")
• How many consecutive failures occurred
• A countdown until the next retry attempt
• The error reason (hover for details)

What triggers a watcher breaker

A breaker opens when the watcher's internal logic encounters an error during its scan-and-act cycle. For example:

• The auto-promoter fails to launch a container
• The auto-retrier encounters an unexpected store error
• The auto-submitter fails to commit changes

How it recovers

Watcher breakers use exponential backoff:

After each cooldown, the watcher tries again. A single success resets the breaker completely — the failure counter goes back to zero and the watcher resumes normal operation.

What still works

Watcher breakers only affect automated actions. You can still:

• Manually promote tasks from the board
• Manually trigger refinement or testing
• Use all other UI features normally

Automation toggles (Autopilot, Auto-test, etc.) remain independent of breaker state. The breaker is a transient safety layer, not a configuration change.

Container Breaker

The container breaker protects against the container runtime (Docker or Podman) being unavailable. If the runtime crashes, gets uninstalled, or the daemon stops, wallfacer stops trying to launch containers until the runtime recovers.

What triggers it

The container breaker opens after 5 consecutive runtime errors (configurable). Only true runtime failures count:

• Exit code 125 (container engine failure)
• Connection refused (daemon not running)
• Binary not found

Normal agent failures (exit codes 1–124) do not trip the breaker.

How it recovers

The container breaker uses a three-state model:

1. Closed (normal): All container launches proceed.
2. Open (tripped): All launches blocked for 30 seconds.
3. Half-open (probing): One launch is allowed as a probe.
   • If the probe succeeds → circuit closes, operations resume.
   • If the probe fails → circuit reopens for another 30 seconds.

What you see

The container breaker is not directly shown in the UI. Its effects are visible as:

• Auto-promote stops picking up new tasks
• Auto-retry stops retrying tasks that failed due to container crashes
• The board appears to "pause" temporarily

Once the container runtime is available again, the next probe succeeds and everything resumes automatically.

The container breaker state is available via GET /api/debug/runtime (the container_circuit field) for monitoring.

Configuration

Both breakers are configurable via process-level environment variables (set in your shell, systemd unit, or container environment):

Watcher breaker timing (30s base, 5min cap) is not currently configurable.

Design Rationale

Circuit breakers exist because wallfacer runs automation continuously in the background. Without them, a transient failure (e.g. Docker daemon restarting) would cause every watcher to spam failed attempts, flood logs, and potentially exhaust resources. The breakers provide automatic backoff and recovery with no user action needed.

Key design choices:

• Per-watcher isolation: A failure in auto-test does not affect auto-promote. Each watcher recovers independently.
• User actions bypass breakers: Manual operations always work, even when automation is paused.
• No manual reset: Breakers self-heal. The exponential backoff ensures recovery within 5 minutes at most. A manual reset could cause re-triggering of the same failure.
• Automatic, not configurable toggles: Breakers are a safety mechanism, not a user preference. They open and close based on runtime conditions.