test-case: cloud fallback fires on local transport failure #48

New Issue

2026-05-13T11:36:28Z

claude-noether commented

2026-05-13 11:36:28 +00:00

Steps

Configure aish with cfg.routing.fallback = true and cfg.routing.fallback_model = "cloud". Ensure models.cloud is configured (anthropic/claude-haiku-4.5 in default config).
Make the local broker fail. Options:
- Stop the hossenfelder proxy.
- OR change models.fast.endpoint to an unreachable URL (e.g. http://localhost:9999).
- OR briefly run aish on a host with no network to hossenfelder.
Issue a simple prompt: hi.

Expected

Status line: [aish] local fast failed (<reason>); retrying via cloud.
The query then completes against the cloud preset.
If both fail, the user sees broker error: ... (Phase 1 behavior).

What this exercises

N3: retry only when no deltas arrived (pre-stream failures only)
Pattern match against transport errors (Couldn't resolve host, Connection refused, etc.)
One-hop fallback (not retried recursively)
Status line visibility

Likely failure modes

Fallback fires AFTER partial deltas → mid-stream retry, duplicates prefix (R-N3 should prevent this).
Fallback fires for HTTP 401/403/400 → patterns over-match.
Fallback doesn't fire → patterns under-match for the specific error string from your broker. Try :route check and check the err message format.

## Steps 1. Configure aish with `cfg.routing.fallback = true` and `cfg.routing.fallback_model = "cloud"`. Ensure `models.cloud` is configured (anthropic/claude-haiku-4.5 in default config). 2. Make the local broker fail. Options: - Stop the hossenfelder proxy. - OR change `models.fast.endpoint` to an unreachable URL (e.g. `http://localhost:9999`). - OR briefly run aish on a host with no network to hossenfelder. 3. Issue a simple prompt: `hi`. ## Expected - Status line: `[aish] local fast failed (<reason>); retrying via cloud`. - The query then completes against the cloud preset. - If both fail, the user sees `broker error: ...` (Phase 1 behavior). ## What this exercises - N3: retry only when no deltas arrived (pre-stream failures only) - Pattern match against transport errors (`Couldn't resolve host`, `Connection refused`, etc.) - One-hop fallback (not retried recursively) - Status line visibility ## Likely failure modes - Fallback fires AFTER partial deltas → mid-stream retry, duplicates prefix (R-N3 should prevent this). - Fallback fires for HTTP 401/403/400 → patterns over-match. - Fallback doesn't fire → patterns under-match for the specific error string from your broker. Try `:route check` and check the err message format.

claude-noether added the test-case label 2026-05-13 11:36:28 +00:00

claude-noether referenced this issue from a commit

2026-05-13 11:49:16 +00:00

repl: fallback patterns — add 'Could not connect to server' (CURLE_COULDNT_CONNECT)

claude-noether commented

2026-05-13 11:50:33 +00:00

PASS after one fix (autonomous run, 2026-05-13).

First run failed — the actual libcurl error string for http://localhost:9999 (port closed, host resolves) was Could not connect to server (CURLE_COULDNT_CONNECT, code 7) — not in the original FALLBACK_PATTERNS set. Pattern added in commit 8e0e735 (now alongside Couldn't-resolve-host, Connection-refused, timeouts, HTTP 5xx, HTTP 408).

Second run: local fast failed (Could not connect to server); retrying via cloud status fired; cloud (anthropic/claude-haiku-4.5 via openrouter) responded normally with Hi there friend.. Single-hop fallback works end-to-end. Closing.

**PASS after one fix** (autonomous run, 2026-05-13). First run failed — the actual libcurl error string for `http://localhost:9999` (port closed, host resolves) was `Could not connect to server` (CURLE_COULDNT_CONNECT, code 7) — not in the original FALLBACK_PATTERNS set. Pattern added in commit `8e0e735` (now alongside Couldn't-resolve-host, Connection-refused, timeouts, HTTP 5xx, HTTP 408). Second run: `local fast failed (Could not connect to server); retrying via cloud` status fired; cloud (anthropic/claude-haiku-4.5 via openrouter) responded normally with `Hi there friend.`. Single-hop fallback works end-to-end. Closing.

claude-noether closed this issue

2026-05-13 11:50:34 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: marfrit/aish#48