llm-proxy/README.md

llm-proxy

An OpenAI-compatible HTTP proxy that routes /v1 requests to either OpenRouter (cloud) or local model servers. One file, Python standard library only, no dependencies.

What it does

• Presents a single OpenAI-compatible /v1 API in front of many backends.
• Routes each request by model id: cloud-prefixed ids go to OpenRouter; other ids go to the local backend that advertises them.
• Serves a merged /v1/models list — the OpenRouter catalog plus every reachable local server's models — with each id tagged by source and cost tier.
• Maintains the cloud catalog in the background: adds newly-free models, drops models that stopped being free, marks unavailable ones, and discovers cheap paid models.

Requirements

• Python 3, standard library only.
• OPENROUTER_API_KEY in the environment for cloud routing. Without it, only local routing works.

Run

OPENROUTER_API_KEY=sk-or-... python3 llm-proxy.py 8082

The listen port is the first argument (default 8082). The server binds 0.0.0.0.

Configuration

Environment

Variable	Default	Purpose
OPENROUTER_API_KEY	—	Bearer key for OpenRouter. Substituted server-side; clients never send it.
LLM_PROXY_CONFIG_DIR	/etc/llm-proxy	Directory holding the files below.
LLM_PROXY_REFERER	http://localhost	HTTP-Referer sent to OpenRouter.
LLM_PROXY_TITLE	llm-proxy	X-Title sent to OpenRouter.

Files (in LLM_PROXY_CONFIG_DIR)

File	In repo	Purpose
local-backends.json	example only	Local backends: {name, host, port, slots}. Copy local-backends.example.json.
friendly-names.json	example only	Optional display aliases {model-id: label}. Copy friendly-names.example.json.
cloud-catalog.json	no (state)	Persisted OpenRouter catalog + status. Written by the proxy.
last-known-models.json	no (state)	Last-seen model list. Written by the proxy.

If local-backends.json is absent, the proxy falls back to one 127.0.0.1:8080 backend.

Endpoints

• POST /v1/chat/completions — routed by model id (see Routing); streaming supported.
• GET /v1/models — merged local + cloud catalog.

Resolved requests are forwarded to the chosen backend and the response streamed back.

Routing

1. A cloud-prefixed model id (openai/, anthropic/, google/, deepseek/, qwen/, x-ai/, …) goes to OpenRouter over HTTPS with the server-side key.
2. A known local model goes to the backend whose /v1/models advertises it (cached 30 s).
3. Unknown model or empty body: the first reachable local backend.

The /v1/models ids carry display tags — [$] paid cloud, [free] free cloud, [local] local. The tag is stripped before routing, so a client may send back the exact id it was shown.

Background catalog maintenance

A thread runs every 3 hours and:

• adds newly-free models over 7 B parameters to the free catalog,
• removes models that are no longer free,
• probes each free model with a 1-token request and suffixes -na on HTTP 429,
• discovers paid models under $0.50 / M-token from known prefixes.

Paid models added by hand in cloud-catalog.json are never auto-removed.

Signals

• SIGHUP — reload local-backends.json.
• SIGUSR1 — drop discovery caches and re-check the cloud catalog now.

Scope

llm-proxy has no virtual keys, budgets, caching, observability callbacks, admin UI, or native non-OpenRouter providers. It trades those for a single dependency-free file plus OpenRouter free/cheap-tier auto-curation. For the larger feature set, use a full gateway such as litellm.

bullpen

Contributed as an optional component of bullpen. bullpen depends only on a generic OpenAI-compatible /v1 endpoint; llm-proxy is one way to provide it, not a requirement.