Ragas

Wherever the OS grounds decisions in retrieved documents, unmeasured RAG is decoration. Ragas turns retrieval + generation into scored loops so “document-backed” claims can fail CI. Prefer it for RAG-centric research features; prefer Promptfoo/DeepEval when the bottleneck is multi-provider matrices or agent trajectory unit tests.

1. Build or generate an eval dataset (quickstart templates or production-aligned generation).
2. Run metrics (built-in or custom LLM-judged metrics) against app outputs/contexts.
3. Inspect scores/reasons; feed failures back into chunking, retrieval, or prompts.
4. Optionally close the loop with production data for continual improvement.

Related: abm-agent-behavior-mining · agent-evaluation · deepeval · topics/32-eval-harnesses

• Use when: the primary risk is retrieval/generation quality (faithfulness, context relevance, etc.).
• Skip when: you need red-team/security matrices (promptfoo), process mining (pm4py), or AWS conversation harnesses (agent-evaluation) without a RAG corpus.
• Prefer instead: DeepEval if you already standardize on pytest metrics; TruLens if you need OTEL traces + RAG triad feedback in one stack.

• Metric/API churn: README now leads with flexible metrics — pin versions; don’t assume 2023 paper metric names forever.
• Judge dependency: LLM metrics need keys, cost, and calibration against human labels.
• Analytics default-on: opt out explicitly in locked-down environments.
• Not a DMS: evaluates retrieval quality; does not replace document systems of record.