Tagged “promptfoo”

4 posts · View all tags
The capstone: build, evaluate, and observe a RAG docs assistant on the WEC API
AdvancedPart 5

The capstone: build, evaluate, and observe a RAG docs assistant on the WEC API

Build a production-shaped RAG service in Docker: scrape a real docs site, embed locally, generate on the WEC Inference API — then catch a real hallucination, root-cause it to your own scraper, fix it, and pin it with a regression test. Every command, number, and error is real.

AIinferenceevalsdocker
July 13, 202626 min read
Read more
Stop hand-writing test cases: generate an eval dataset with the WEC API
IntermediatePart 3

Stop hand-writing test cases: generate an eval dataset with the WEC API

Two hand-typed test cases aren't an eval. Use the WEC Inference API to generate a labeled evaluation dataset — then validate and curate it, because generated labels aren't automatically correct. The result is real test data at scale; feed it to your harness and coverage surfaces the misclassifications and debatable labels two cases would hide. Every command and result is real.

AIinferenceevalsobservability
July 6, 202613 min read
Read more
Trustworthy JSON: schema-validate your model's structured output
IntermediatePart 2

Trustworthy JSON: schema-validate your model's structured output

LLMs promise JSON and deliver markdown fences and reasoning. Build a Promptfoo eval that classifies support tickets into schema-validated JSON on the WEC Inference API — with transforms to recover messy output, a model-reliability matrix, and a CI gate. Every command and result is real.

AIinferenceevalsobservability
July 1, 202614 min read
Read more
Evaluate your models with Promptfoo on the WEC Inference API
BeginnerPart 1

Evaluate your models with Promptfoo on the WEC Inference API

Stop eyeballing LLM output. Build a real evaluation harness with Promptfoo pointed at the WEC Inference API — assertions, latency guardrails, JSON-schema checks, model-graded rubrics, an all-WEC model comparison, and a CI gate. Every command and result is real.

AIinferenceevalsobservability
June 30, 202612 min read
Read more