# llm-proxy

An OpenAI-compatible HTTP proxy that routes `/v1` requests to either OpenRouter
(cloud) or local model servers. One file, Python standard library only, no
dependencies.

## What it does

- Presents a single OpenAI-compatible `/v1` API in front of many backends.
- Routes each request by model id: cloud-prefixed ids go to OpenRouter; other
  ids go to the local backend that advertises them.
- Serves a merged `/v1/models` list — the OpenRouter catalog plus every
  reachable local server's models — with each id tagged by source and cost tier.
- Maintains the cloud catalog in the background: adds newly-free models, drops
  models that stopped being free, marks unavailable ones, and discovers cheap
  paid models.

## Requirements

- Python 3, standard library only.
- `OPENROUTER_API_KEY` in the environment for cloud routing. Without it, only
  local routing works.

## Run

    OPENROUTER_API_KEY=sk-or-... python3 llm-proxy.py 8082

The listen port is the first argument (default `8082`). The server binds `0.0.0.0`.

## Configuration

### Environment

| Variable | Default | Purpose |
|---|---|---|
| `OPENROUTER_API_KEY` | — | Bearer key for OpenRouter. Substituted server-side; clients never send it. |
| `LLM_PROXY_CONFIG_DIR` | `/etc/llm-proxy` | Directory holding the files below. |
| `LLM_PROXY_REFERER` | `http://localhost` | `HTTP-Referer` sent to OpenRouter. |
| `LLM_PROXY_TITLE` | `llm-proxy` | `X-Title` sent to OpenRouter. |

### Files (in `LLM_PROXY_CONFIG_DIR`)

| File | In repo | Purpose |
|---|---|---|
| `local-backends.json` | example only | Local backends: `{name, host, port, slots}`. Copy `local-backends.example.json`. |
| `friendly-names.json` | example only | Optional display aliases `{model-id: label}`. Copy `friendly-names.example.json`. |
| `cloud-catalog.json` | no (state) | Persisted OpenRouter catalog + status. Written by the proxy. |
| `last-known-models.json` | no (state) | Last-seen model list. Written by the proxy. |

If `local-backends.json` is absent, the proxy falls back to one `127.0.0.1:8080`
backend.

## Endpoints

- `POST /v1/chat/completions` — routed by model id (see Routing); streaming supported.
- `GET /v1/models` — merged local + cloud catalog.

Resolved requests are forwarded to the chosen backend and the response streamed back.

## Routing

1. A **cloud-prefixed** model id (`openai/`, `anthropic/`, `google/`,
   `deepseek/`, `qwen/`, `x-ai/`, …) goes to OpenRouter over HTTPS with the
   server-side key.
2. A **known local** model goes to the backend whose `/v1/models` advertises it
   (cached 30 s).
3. Unknown model or empty body: the first reachable local backend.

The `/v1/models` ids carry display tags — `[$]` paid cloud, `[free]` free cloud,
`[local]` local. The tag is stripped before routing, so a client may send back
the exact id it was shown.

## Background catalog maintenance

A thread runs every 3 hours and:

- adds newly-free models over 7 B parameters to the free catalog,
- removes models that are no longer free,
- probes each free model with a 1-token request and suffixes `-na` on HTTP 429,
- discovers paid models under `$0.50` / M-token from known prefixes.

Paid models added by hand in `cloud-catalog.json` are never auto-removed.

## Signals

- `SIGHUP` — reload `local-backends.json`.
- `SIGUSR1` — drop discovery caches and re-check the cloud catalog now.

## Scope

llm-proxy has no virtual keys, budgets, caching, observability callbacks, admin
UI, or native non-OpenRouter providers. It trades those for a single
dependency-free file plus OpenRouter free/cheap-tier auto-curation. For the
larger feature set, use a full gateway such as litellm.

## bullpen

Contributed as an optional component of
[bullpen](https://git.reauktion.de/marfrit/bullpen). bullpen depends only on a
generic OpenAI-compatible `/v1` endpoint; llm-proxy is one way to provide it,
not a requirement.