TC #74 — Phase 7 R2 fallback retry credits FALLBACK model name (INTERACTIVE) #74

New Issue

2026-05-16T23:03:28Z

claude-noether commented

2026-05-16 23:03:28 +00:00

Scenario

Config with broken primary endpoint + working fallback (e.g., cfg.routing = { fallback = true, fallback_model = "cloud" }, primary = nonexistent endpoint). Send a prompt.

Expected

The fallback fires; usage in :cost detail is credited to the FALLBACK model name (cloud), NOT the primary.
R2 — broker emits payload.model = model_cfg.model which IS the fallback's name when chat_stream is called with fb_cfg.

Interactive — needs careful broker setup

Requires standing up a deliberately-broken primary and watching the fallback path execute, ideally via :cost detail after.

## Scenario Config with broken primary endpoint + working fallback (e.g., `cfg.routing = { fallback = true, fallback_model = "cloud" }`, primary = nonexistent endpoint). Send a prompt. ## Expected The fallback fires; usage in :cost detail is credited to the FALLBACK model name (cloud), NOT the primary. R2 — broker emits payload.model = model_cfg.model which IS the fallback's name when chat_stream is called with fb_cfg. ## Interactive — needs careful broker setup Requires standing up a deliberately-broken primary and watching the fallback path execute, ideally via :cost detail after.

claude-noether added the test-case label 2026-05-16 23:03:28 +00:00

claude-noether commented

2026-05-16 23:04:35 +00:00

Verified PASS — autonomous after all.

Setup: config with broken primary (endpoint http://127.0.0.1:1, model 'fake-primary') + working cloud fallback (anthropic/claude-haiku-4.5). routing.fallback = true.

Run:

[aish:broken]> say hi briefly
[aish] local broken failed (Could not connect to server); retrying via cloud
Hi! 👋
[aish:broken]> :cost detail
[aish] session usage detail (total=$0.000219, 179/8 tokens):
  anthropic/claude-haiku-4.5 main                 1 calls,    179 /      8 tokens, $0.000219

R2 invariant confirmed: the usage is credited to anthropic/claude-haiku-4.5 (fallback's model name), NOT 'fake-primary'. The broker emits payload.model = model_cfg.model — when chat_stream is called with fb_cfg during the fallback retry, model_cfg.model IS the fallback's name. The wrapper keys by payload.model so no primary-vs-fallback tracking is needed.

Closed by Phase 7 verify.

Verified PASS — autonomous after all. Setup: config with broken primary (endpoint http://127.0.0.1:1, model 'fake-primary') + working cloud fallback (anthropic/claude-haiku-4.5). routing.fallback = true. Run: ``` [aish:broken]> say hi briefly [aish] local broken failed (Could not connect to server); retrying via cloud Hi! 👋 [aish:broken]> :cost detail [aish] session usage detail (total=$0.000219, 179/8 tokens): anthropic/claude-haiku-4.5 main 1 calls, 179 / 8 tokens, $0.000219 ``` R2 invariant confirmed: the usage is credited to **anthropic/claude-haiku-4.5** (fallback's model name), NOT 'fake-primary'. The broker emits payload.model = model_cfg.model — when chat_stream is called with fb_cfg during the fallback retry, model_cfg.model IS the fallback's name. The wrapper keys by payload.model so no primary-vs-fallback tracking is needed. Closed by Phase 7 verify.

claude-noether closed this issue

2026-05-16 23:04:35 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: marfrit/aish#74