Routing by task
A service asks for a class of task rather than a specific model: cheaper, faster or more accurate. The gateway chooses where to send it. The model can be changed for the whole company in one place without touching service code.
Proxy Broker live
Proxy Broker: a gateway to language models
A single entry point for all AI requests in the company. It fails over between providers, cuts spend through routing, and stops one broken service from dragging down the rest.
The problem
When every service talks to the models on its own, three problems arrive at once. Keys spread across projects, nobody sees the total spend, and one provider outage takes down everything that depends on it.
Add rate limits on top: the provider starts refusing at peak load, exactly when the service is needed most.
How it works
Failover on errors
A rate limit or a server error is no longer an error shown to the user; the request moves to the next provider in the list. The failing provider is excluded for a while and checked again later.
Local models as reserve
Models on our own hardware sit at the end of the chain. They cost nothing per request and are always reachable, so a complete failure would mean your own server is down.
Accounting and observability
Every request passes through one point, so it is visible how much was spent, by which service and on which task. Metrics, logs and alerts are collected by a separate stack.
What it gives
- Model spend goes down: cheap tasks no longer land on expensive models
- A provider outage stops being an incident
- Keys and limits live in one place instead of ten projects
- Spend is visible per service and per task
Who it is for
Companies where more than one service talks to language models, and anyone who wants to see and control what those models cost.
Questions
What happens when a provider hits its rate limit?
The request goes to the next provider. A rate limit used to count as an ordinary error and was returned to the user; we changed that because it happens regularly at peak load.
How does the gateway know a provider is broken?
It counts errors over a short window. Once there are too many, the provider is switched off for a while, then probed again with a single request. That way a broken service does not collect the whole load.
Can we run a gateway like this ourselves?
Yes. It is an ordinary containerised service that sits next to your infrastructure. Local models go on your hardware if privacy matters more than quality.
Other projects
Ads Automation marketing Campaign management through the API instead of clicking through the interface. It always starts with one question: are conversions even counted correctly? Read the case → RAG Platform on-premise The model answers questions from your materials rather than from the internet. Everything runs on your own hardware, so documents never leave the company network. Read the case →
Need a similar solution?
Tell us about the task and we will say which parts of this apply to you.