Tagged “observability”

7 posts · View all tags
Load test an LLM gateway and find the stall the median hides
AdvancedPart 5

Load test an LLM gateway and find the stall the median hides

Part 4 measured a blocking call in a callback at 200-350ms and said the real damage only shows under concurrency. This runs it: sixteen requests at once, with a control against the upstream so you can tell your gateway's fault from the model's. The medians of the two versions are almost identical. One version drops requests and silently stops masking.

performancegatewaylitellmobservability
August 27, 202619 min read
Read more
Clean traces, untouched answers: masking PII in LiteLLM's logs without corrupting the response
AdvancedPart 4

Clean traces, untouched answers: masking PII in LiteLLM's logs without corrupting the response

Send a gateway's traffic to Langfuse and the model's own reply carries the PII straight back into your traces. Widen Presidio's scope and it masks the answer your users receive instead. Neither setting gives you both, so here is a forty-line callback that does — measured, and with the mistake that quietly makes it slower.

privacypiiguardrailsgateway
August 21, 202615 min read
Read more
Component-level tracing: debugging agent tool calls
IntermediatePart 7

Component-level tracing: debugging agent tool calls

Your agent's thinking and its actions are two different layers. Build a tool-calling agent on WEC Inference, trace it with Langfuse, then debug two real failures from the trace — including the confident, wrong answer that never throws a stack trace.

AIAgentsinferenceevals
July 22, 202613 min read
Read more
Catch what your tests miss: observe and score your WEC app in production with Langfuse
AdvancedPart 4

Catch what your tests miss: observe and score your WEC app in production with Langfuse

CI evals pass on a fixed test set — but production sends inputs you never tested. Self-host Langfuse on a WEC Instance, trace every real call, auto-score live traffic with an LLM judge, drill into the exact step that broke, and feed failures back to make your evals stronger. Every command — and every dead end — is real.

AIself-hostinginferenceevals
July 7, 202618 min read
Read more
Stop hand-writing test cases: generate an eval dataset with the WEC API
IntermediatePart 3

Stop hand-writing test cases: generate an eval dataset with the WEC API

Two hand-typed test cases aren't an eval. Use the WEC Inference API to generate a labeled evaluation dataset — then validate and curate it, because generated labels aren't automatically correct. The result is real test data at scale; feed it to your harness and coverage surfaces the misclassifications and debatable labels two cases would hide. Every command and result is real.

AIinferenceevalsobservability
July 6, 202613 min read
Read more
Trustworthy JSON: schema-validate your model's structured output
IntermediatePart 2

Trustworthy JSON: schema-validate your model's structured output

LLMs promise JSON and deliver markdown fences and reasoning. Build a Promptfoo eval that classifies support tickets into schema-validated JSON on the WEC Inference API — with transforms to recover messy output, a model-reliability matrix, and a CI gate. Every command and result is real.

AIinferenceevalsobservability
July 1, 202614 min read
Read more
Evaluate your models with Promptfoo on the WEC Inference API
BeginnerPart 1

Evaluate your models with Promptfoo on the WEC Inference API

Stop eyeballing LLM output. Build a real evaluation harness with Promptfoo pointed at the WEC Inference API — assertions, latency guardrails, JSON-schema checks, model-graded rubrics, an all-WEC model comparison, and a CI gate. Every command and result is real.

AIinferenceevalsobservability
June 30, 202612 min read
Read more