Been working on this on the side. It started because we kept writing the same retry/fallback code every time we hooked up an LLM provider at work, so we finally just wrote it once and split it out into its own thing.
It's called llmrouter. You register your providers (OpenAI, Anthropic, Gemini plus a bunch of OpenAI compatible ones) once at startup, and it takes care of retries, circuit breaking, fallback to a backup provider, and streaming/tool calls, all behind one interface so your actual code doesn't care which provider answered.
Went with Go for it mainly because that's what our backend stack already is, and honestly Go's plain function types make this kind of middleware chaining pretty clean to write and reason about, no interfaces or inheritance gymnastics needed.
There's no shortage of LLM gateways and routers out there already, so this isn't trying to be the one to rule them all. It's just the version we built for ourselves and figured we'd open source.
Middleware is just a plain function, func(Provider) Provider, so if you want to bolt something on it's not a big lift. We also learned the hard way that if you share one circuit breaker across providers, one bad provider trips the breaker for everyone, that one bit us in testing.
- Wrote up how it's put together here if you want the long version
- Github repo: https://github.com/bluefunda/llmrouter
Poke holes in it if you want, still early days.
[link] [留言]