DeepEval

Weekly prompt/tool changes need unit/integration tests that score quality, not screenshots. DeepEval is the research default when you want metrics as code in CI for agents and RAG. Pair with Promptfoo when the bottleneck is declarative provider matrices or security probes; pair with TruLens when you need OTEL-native step traces plus judges.

Scores must attach to cases; the eval store is not the company ledger.

1. Author pytest-style tests that call DeepEval metrics on app outputs (or trajectories).
2. Metrics run LLM judges / local NLP models against expected behaviors.
3. Fail CI when thresholds miss; iterate prompts, models, or architecture.
4. Optionally push runs to Confident AI for dashboards and production monitors.

Related: abm-agent-behavior-mining · agent-evaluation · hal-the-holistic-agent-leaderboard · topics/32-eval-harnesses

• Use when: you want metric libraries as unit tests for RAG/agents, including trajectory and step-level scoring.
• Skip when: you mainly need YAML multi-provider eval + red team CLI (promptfoo), RAG-only research metrics (ragas), or process-mining of XES logs (pm4py / ABM).
• Prefer instead: Promptfoo for red-team breadth; Ragas for classic RAG metric papers; TruLens for OTEL tracing + feedback functions.

• Open core: advanced reporting/monitoring pushes toward Confident AI SaaS.
• Judge cost & drift: LLM-as-judge needs locked models and golden calibration.
• Not a process miner: trajectories here are eval objects, not XES event logs.
• Overlap: metric catalogs overlap Ragas/TruLens — pick one primary for a given CI job to avoid contradictory scores.