Tagged “evals”

7 posts · View all tags
Component-level tracing: debugging agent tool calls
IntermediatePart 7

Component-level tracing: debugging agent tool calls

Your agent's thinking and its actions are two different layers. Build a tool-calling agent on WEC Inference, trace it with Langfuse, then debug two real failures from the trace — including the confident, wrong answer that never throws a stack trace.

AIAgentsinferenceevals
July 22, 202613 min read
Read more
Regression-test your RAG service with DeepEval — and settle a model debate with data
AdvancedPart 6

Regression-test your RAG service with DeepEval — and settle a model debate with data

Wrap the RAG assistant from part 5 in a containerized DeepEval suite judged by gemma4 on the WEC Inference API — no OpenAI key anywhere. Then use it to answer a real question: should we swap our generation model? Same quality, 5.5× the tokens: the eval says no. Every command, number, and error is real.

AIinferenceevalsdocker
July 20, 202619 min read
Read more
The capstone: build, evaluate, and observe a RAG docs assistant on the WEC API
AdvancedPart 5

The capstone: build, evaluate, and observe a RAG docs assistant on the WEC API

Build a production-shaped RAG service in Docker: scrape a real docs site, embed locally, generate on the WEC Inference API — then catch a real hallucination, root-cause it to your own scraper, fix it, and pin it with a regression test. Every command, number, and error is real.

AIinferenceevalsdocker
July 13, 202626 min read
Read more
Catch what your tests miss: observe and score your WEC app in production with Langfuse
AdvancedPart 4

Catch what your tests miss: observe and score your WEC app in production with Langfuse

CI evals pass on a fixed test set — but production sends inputs you never tested. Self-host Langfuse on a WEC Instance, trace every real call, auto-score live traffic with an LLM judge, drill into the exact step that broke, and feed failures back to make your evals stronger. Every command — and every dead end — is real.

AIself-hostinginferenceevals
July 7, 202618 min read
Read more
Stop hand-writing test cases: generate an eval dataset with the WEC API
IntermediatePart 3

Stop hand-writing test cases: generate an eval dataset with the WEC API

Two hand-typed test cases aren't an eval. Use the WEC Inference API to generate a labeled evaluation dataset — then validate and curate it, because generated labels aren't automatically correct. The result is real test data at scale; feed it to your harness and coverage surfaces the misclassifications and debatable labels two cases would hide. Every command and result is real.

AIinferenceevalsobservability
July 6, 202613 min read
Read more
Trustworthy JSON: schema-validate your model's structured output
IntermediatePart 2

Trustworthy JSON: schema-validate your model's structured output

LLMs promise JSON and deliver markdown fences and reasoning. Build a Promptfoo eval that classifies support tickets into schema-validated JSON on the WEC Inference API — with transforms to recover messy output, a model-reliability matrix, and a CI gate. Every command and result is real.

AIinferenceevalsobservability
July 1, 202614 min read
Read more
Evaluate your models with Promptfoo on the WEC Inference API
BeginnerPart 1

Evaluate your models with Promptfoo on the WEC Inference API

Stop eyeballing LLM output. Build a real evaluation harness with Promptfoo pointed at the WEC Inference API — assertions, latency guardrails, JSON-schema checks, model-graded rubrics, an all-WEC model comparison, and a CI gate. Every command and result is real.

AIinferenceevalsobservability
June 30, 202612 min read
Read more