marfrit
cc3be17994
Zero-dependency OpenAI-compatible router: OpenRouter cloud + local model-aware fanout, merged /v1/models catalog with cost tags, background free/cheap-tier auto-curation. Extracted from the fleet's unversioned /opt/llm-proxy.py (kept only as cp-backups) and de-hardcoded for portability: - config dir via LLM_PROXY_CONFIG_DIR (default /etc/llm-proxy) - default backends -> localhost (no fleet hosts); real ones in local-backends.json - friendly-name aliases -> friendly-names.json (none baked in code) - OpenRouter Referer/Title via LLM_PROXY_REFERER / LLM_PROXY_TITLE - removed the ds4-escher one-off filter and the fleet-specific helper scripts Sanitized: no fleet hostnames, IPs, names, or secrets (key comes from env). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
102 lines
3.8 KiB
Markdown
102 lines
3.8 KiB
Markdown
# llm-proxy
|
|
|
|
An OpenAI-compatible HTTP proxy that routes `/v1` requests to either OpenRouter
|
|
(cloud) or local model servers. One file, Python standard library only, no
|
|
dependencies.
|
|
|
|
## What it does
|
|
|
|
- Presents a single OpenAI-compatible `/v1` API in front of many backends.
|
|
- Routes each request by model id: cloud-prefixed ids go to OpenRouter; other
|
|
ids go to the local backend that advertises them.
|
|
- Serves a merged `/v1/models` list — the OpenRouter catalog plus every
|
|
reachable local server's models — with each id tagged by source and cost tier.
|
|
- Maintains the cloud catalog in the background: adds newly-free models, drops
|
|
models that stopped being free, marks unavailable ones, and discovers cheap
|
|
paid models.
|
|
|
|
## Requirements
|
|
|
|
- Python 3, standard library only.
|
|
- `OPENROUTER_API_KEY` in the environment for cloud routing. Without it, only
|
|
local routing works.
|
|
|
|
## Run
|
|
|
|
OPENROUTER_API_KEY=sk-or-... python3 llm-proxy.py 8082
|
|
|
|
The listen port is the first argument (default `8082`). The server binds `0.0.0.0`.
|
|
|
|
## Configuration
|
|
|
|
### Environment
|
|
|
|
| Variable | Default | Purpose |
|
|
|---|---|---|
|
|
| `OPENROUTER_API_KEY` | — | Bearer key for OpenRouter. Substituted server-side; clients never send it. |
|
|
| `LLM_PROXY_CONFIG_DIR` | `/etc/llm-proxy` | Directory holding the files below. |
|
|
| `LLM_PROXY_REFERER` | `http://localhost` | `HTTP-Referer` sent to OpenRouter. |
|
|
| `LLM_PROXY_TITLE` | `llm-proxy` | `X-Title` sent to OpenRouter. |
|
|
|
|
### Files (in `LLM_PROXY_CONFIG_DIR`)
|
|
|
|
| File | In repo | Purpose |
|
|
|---|---|---|
|
|
| `local-backends.json` | example only | Local backends: `{name, host, port, slots}`. Copy `local-backends.example.json`. |
|
|
| `friendly-names.json` | example only | Optional display aliases `{model-id: label}`. Copy `friendly-names.example.json`. |
|
|
| `cloud-catalog.json` | no (state) | Persisted OpenRouter catalog + status. Written by the proxy. |
|
|
| `last-known-models.json` | no (state) | Last-seen model list. Written by the proxy. |
|
|
|
|
If `local-backends.json` is absent, the proxy falls back to one `127.0.0.1:8080`
|
|
backend.
|
|
|
|
## Endpoints
|
|
|
|
- `POST /v1/chat/completions` — routed by model id (see Routing); streaming supported.
|
|
- `GET /v1/models` — merged local + cloud catalog.
|
|
|
|
Resolved requests are forwarded to the chosen backend and the response streamed back.
|
|
|
|
## Routing
|
|
|
|
1. A **cloud-prefixed** model id (`openai/`, `anthropic/`, `google/`,
|
|
`deepseek/`, `qwen/`, `x-ai/`, …) goes to OpenRouter over HTTPS with the
|
|
server-side key.
|
|
2. A **known local** model goes to the backend whose `/v1/models` advertises it
|
|
(cached 30 s).
|
|
3. Unknown model or empty body: the first reachable local backend.
|
|
|
|
The `/v1/models` ids carry display tags — `[$]` paid cloud, `[free]` free cloud,
|
|
`[local]` local. The tag is stripped before routing, so a client may send back
|
|
the exact id it was shown.
|
|
|
|
## Background catalog maintenance
|
|
|
|
A thread runs every 3 hours and:
|
|
|
|
- adds newly-free models over 7 B parameters to the free catalog,
|
|
- removes models that are no longer free,
|
|
- probes each free model with a 1-token request and suffixes `-na` on HTTP 429,
|
|
- discovers paid models under `$0.50` / M-token from known prefixes.
|
|
|
|
Paid models added by hand in `cloud-catalog.json` are never auto-removed.
|
|
|
|
## Signals
|
|
|
|
- `SIGHUP` — reload `local-backends.json`.
|
|
- `SIGUSR1` — drop discovery caches and re-check the cloud catalog now.
|
|
|
|
## Scope
|
|
|
|
llm-proxy has no virtual keys, budgets, caching, observability callbacks, admin
|
|
UI, or native non-OpenRouter providers. It trades those for a single
|
|
dependency-free file plus OpenRouter free/cheap-tier auto-curation. For the
|
|
larger feature set, use a full gateway such as litellm.
|
|
|
|
## bullpen
|
|
|
|
Contributed as an optional component of
|
|
[bullpen](https://git.reauktion.de/marfrit/bullpen). bullpen depends only on a
|
|
generic OpenAI-compatible `/v1` endpoint; llm-proxy is one way to provide it,
|
|
not a requirement.
|