# What is LiteLLM?

LiteLLM is an open-source project that puts one API in front of many language model providers. Your application calls it in the OpenAI request format; it translates that call for whichever provider is actually going to answer, and translates the answer back into the same shape. Anthropic, OpenAI, Mistral, a model you host yourself: the calling code does not change when the model behind it does.

It is developed by BerriAI and covers more than 100 providers across chat completions, embeddings, images, audio and batch endpoints.

## The problem it solves

Two problems, and organizations usually arrive because of the second one.

**Every provider has its own SDK.** Adding a second model means a second client library, a second authentication scheme and a second set of error semantics in your codebase. Switching models means changing application code, which means the choice of model is a deployment rather than a configuration change. Teams end up locked to whichever provider they integrated first, not because it is best but because moving is work.

**Spend controls sit at the wrong level.** Most providers cap spending per account. One team's runaway retry loop, or one enthusiastic prototype, exhausts the budget for everyone. There is no native way to say that this project gets a certain amount per month and that team gets another, which is exactly what a finance department asks for the moment AI spend becomes visible.

A gateway fixes both by being the single place every request passes through: one interface for applications, one place to enforce budgets, rate limits and access rules, and one log of what was asked and what it cost.

## Proxy or SDK, and why it matters

LiteLLM comes in two forms, and picking the wrong one is the most common early mistake.

The **Python SDK** is a library you import. It gives one developer a uniform way to call many models from their own code. Nothing is centralized: budgets, keys and logs live wherever that code runs.

The **proxy server** is a service you deploy. Applications point at it over HTTP, in any language, and it holds the provider keys, the budgets, the rate limits and the audit log. This is what people mean by an AI gateway, and it is the form that answers the spend and access problems above.

If the goal is governance across teams, it is the proxy. If the goal is convenience inside one Python project, the SDK is enough and a deployment is overhead.

## When a gateway is worth it

- **More than one team is calling models.** A gateway earns its keep at the point where you need per-team budgets and per-team keys.
- **You want the model choice to stay open.** Routing lets you move traffic between providers, or fail over when one has an outage, without shipping application code.
- **Someone has to answer what this costs.** Per-user, per-team and per-project attribution is the difference between a chargeback conversation and a guess.
- **Prompts are sensitive.** Every request passing through infrastructure you control means the logs of what was asked stay where you put them.

## When it is not

**A single application calling a single provider** does not need a gateway. It needs the provider's SDK and less infrastructure.

**A gateway is another hop in the request path.** It adds latency and it is a component that can fail. If your architecture cannot tolerate a dependency between the application and the model, that is a real objection.

**It routes; it does not host.** LiteLLM sends your prompts to whichever provider you configure. If the requirement is that prompt text never reaches a third party at all, a gateway alone does not deliver it. You need a model you host, with the gateway pointed at that. [vLLM](https://www.vllm.ch) is the usual answer for the hosting half.

## What running one involves

The proxy is the control point for every model call in the organization, which is a good place to enforce policy and an uncomfortable place to be careless:

- **It holds every provider key.** Secrets management stops being optional at this point.
- **It is on the critical path.** Whatever availability your AI features need, the gateway needs at least as much, which means more than one replica and a real failover story.
- **Authentication has to connect to how you already manage people.** Single sign-on and role-based access, so that access follows joiners and leavers rather than a spreadsheet of keys.
- **The interesting metrics are per team, not per cluster.** Token usage, error rates by provider, latency by model, and cost attributed to the team that spent it.

## Where VSHN fits

VSHN deploys and operates the LiteLLM proxy on Kubernetes and OpenShift in Swiss data centers, on Exoscale, Cloudscale and other Swiss providers, with the budget controls, guardrails and single sign-on integration configured rather than left at defaults. Proxy logs, API keys and request metadata stay in the chosen jurisdiction, and our [sovereignty assessment](/sovereignty/) sets out how that is scored.

The question worth settling before any deployment is which of the two forms you actually need, and whether the models behind the gateway are hosted or third-party. That is a short conversation and it changes the whole architecture.

[Book an architecture review](#contact)