Tagged “gateway”

7 posts · View all tags
The key that expires: giving an agent its own identity at the gateway
IntermediatePart 5

The key that expires: giving an agent its own identity at the gateway

Part 4 ended with a shell, a static key and a model that answered. This replaces that key with a token your identity provider issues, that carries a scope, and that dies after five minutes. authentik issues it over client credentials, the gateway verifies it in twenty-five lines, and three status codes prove the boundary. Includes the requirement authentik doesn't document, and the licence wall you hit if you follow LiteLLM's own guide.

Agentsgatewaylitellmsecurity
September 8, 202618 min read
Read more
The binding that wasn't there: group access control, and what SSO doesn't protect
IntermediatePart 4

The binding that wasn't there: group access control, and what SSO doesn't protect

Part 3 shipped SSO with empty bindings. Closing that gap on the LiteLLM gateway turned up something worse: authentik's wizard let us configure the bindings, showed them in its own table, and saved none of them — no error, no warning. Then, once access control actually worked, the API answered a request from a shell with no account, no session and no membership. Both of those are the post.

gatewaylitellmsecurityhardening
September 3, 202615 min read
Read more
Load test an LLM gateway and find the stall the median hides
AdvancedPart 5

Load test an LLM gateway and find the stall the median hides

Part 4 measured a blocking call in a callback at 200-350ms and said the real damage only shows under concurrency. This runs it: sixteen requests at once, with a control against the upstream so you can tell your gateway's fault from the model's. The medians of the two versions are almost identical. One version drops requests and silently stops masking.

performancegatewaylitellmobservability
August 27, 202619 min read
Read more
Clean traces, untouched answers: masking PII in LiteLLM's logs without corrupting the response
AdvancedPart 4

Clean traces, untouched answers: masking PII in LiteLLM's logs without corrupting the response

Send a gateway's traffic to Langfuse and the model's own reply carries the PII straight back into your traces. Widen Presidio's scope and it masks the answer your users receive instead. Neither setting gives you both, so here is a forty-line callback that does — measured, and with the mistake that quietly makes it slower.

privacypiiguardrailsgateway
August 21, 202615 min read
Read more
Mask PII at the gateway: set up Presidio, plus the one line the docs leave out
IntermediatePart 3

Mask PII at the gateway: set up Presidio, plus the one line the docs leave out

Set up PII masking on a self-hosted LLM gateway with Presidio. Masking can sit in three places — before the model, on the response, or only on the path to your logs — and only one keeps your app working. Follow the documented config and your model's answers come back redacted; here is why, and the single setting that fixes it.

self-hostingprivacypiiguardrails
August 20, 202620 min read
Read more
LiteLLM complexity routing: the right model for each request, and what it costs in latency
IntermediatePart 2

LiteLLM complexity routing: the right model for each request, and what it costs in latency

How LiteLLM's complexity router decides which model answers a request — the seven scoring dimensions, the arithmetic on a real prompt, and a measured comparison of the free keyword scorer against an LLM classifier.

self-hostinggatewayroutingcost
August 19, 202612 min read
Read more
One endpoint, many models: deploy an LLM gateway on a WEC Instance
IntermediatePart 1

One endpoint, many models: deploy an LLM gateway on a WEC Instance

Every app on a box holding the same API key is a problem waiting to happen. A gateway fixes that — one endpoint, scoped keys per app, per-key budgets, and a log of who spent what. Deployed for real on a host already running five other stacks, including the parts that surprised us.

self-hostinggatewaylitellmdocker
August 13, 202614 min read
Read more