Tagged “wec”

9 posts · View all tags
Split one MCP toolbox between two agents so the scheduler cannot issue refunds
IntermediatePart 4

Split one MCP toolbox between two agents so the scheduler cannot issue refunds

Part 3 handed one agent all five tools and let a human pause guard the dangerous one. That guard depends on the model asking. This splits the same MCP server between a scheduler and a billing agent, so the scheduler cannot refund an invoice for the simplest reason available — the tool was never in its list. Then the supervisor goes back on top, and routing turns out not to be the guarantee.

AIAgentsself-hostingmcp
September 4, 202613 min read
Read more
Book appointments and refund invoices from a LangGraph agent over MCP
IntermediatePart 3

Book appointments and refund invoices from a LangGraph agent over MCP

The scheduler and billing agents from Parts 1 and 2 never actually scheduled or billed anything. This gives them a calendar, a customer list and a refund that no model can issue alone — over MCP, on the stateless spec, with the model on WEC Inference. Includes three things nobody documents: your server still defaults to sessions, ctx.elicit is the old API and its era error goes to the model, not to you, and cache=True does nothing unless the server advertises a TTL.

AIAgentsself-hostingmcp
September 3, 202621 min read
Read more
Load test an LLM gateway and find the stall the median hides
AdvancedPart 5

Load test an LLM gateway and find the stall the median hides

Part 4 measured a blocking call in a callback at 200-350ms and said the real damage only shows under concurrency. This runs it: sixteen requests at once, with a control against the upstream so you can tell your gateway's fault from the model's. The medians of the two versions are almost identical. One version drops requests and silently stops masking.

performancegatewaylitellmobservability
August 27, 202619 min read
Read more
Hand work between LangGraph agents without corrupting shared state
IntermediatePart 2

Hand work between LangGraph agents without corrupting shared state

A supervisor routes work to specialists, an agent hands off mid-task, and two agents write the same state key in the same step. One of those raises an error rather than picking a winner — and the routing decision costs 412 output tokens to say one word.

AIAgentsself-hostinglanggraph
August 26, 202618 min read
Read more
Checkpoint a LangGraph agent on a WEC Instance so crashes cost you nothing
IntermediatePart 1

Checkpoint a LangGraph agent on a WEC Instance so crashes cost you nothing

Persist agent state to Postgres on a WEC Instance, call a model on the WEC Inference API, then kill the process mid-run twice and resume both times from exactly where it stopped — plus the human-approval pause, and the interrupt that silently fires your side effects twice.

AIAgentsself-hostinglanggraph
August 26, 202621 min read
Read more
Clean traces, untouched answers: masking PII in LiteLLM's logs without corrupting the response
AdvancedPart 4

Clean traces, untouched answers: masking PII in LiteLLM's logs without corrupting the response

Send a gateway's traffic to Langfuse and the model's own reply carries the PII straight back into your traces. Widen Presidio's scope and it masks the answer your users receive instead. Neither setting gives you both, so here is a forty-line callback that does — measured, and with the mistake that quietly makes it slower.

privacypiiguardrailsgateway
August 21, 202615 min read
Read more
Mask PII at the gateway: set up Presidio, plus the one line the docs leave out
IntermediatePart 3

Mask PII at the gateway: set up Presidio, plus the one line the docs leave out

Set up PII masking on a self-hosted LLM gateway with Presidio. Masking can sit in three places — before the model, on the response, or only on the path to your logs — and only one keeps your app working. Follow the documented config and your model's answers come back redacted; here is why, and the single setting that fixes it.

self-hostingprivacypiiguardrails
August 20, 202620 min read
Read more
LiteLLM complexity routing: the right model for each request, and what it costs in latency
IntermediatePart 2

LiteLLM complexity routing: the right model for each request, and what it costs in latency

How LiteLLM's complexity router decides which model answers a request — the seven scoring dimensions, the arithmetic on a real prompt, and a measured comparison of the free keyword scorer against an LLM classifier.

self-hostinggatewayroutingcost
August 19, 202612 min read
Read more
One endpoint, many models: deploy an LLM gateway on a WEC Instance
IntermediatePart 1

One endpoint, many models: deploy an LLM gateway on a WEC Instance

Every app on a box holding the same API key is a problem waiting to happen. A gateway fixes that — one endpoint, scoped keys per app, per-key budgets, and a log of who spent what. Deployed for real on a host already running five other stacks, including the parts that surprised us.

self-hostinggatewaylitellmdocker
August 13, 202614 min read
Read more