Agent Evaluation

Committed criteria need scored agent conversations, not vibe demos. This pack is the AWS-shaped harness for “dataset/scenario → talk to agent → metric → gate merge.” Use it when the install already lives on Bedrock/Q/SageMaker or when you need a conversation-orchestrating evaluator rather than only RAG metric libraries (Ragas) or declarative YAML evals (Promptfoo).

Wire scores to cases so a failing eval is attributable; never treat the harness as process authority.

1. Define scenarios / datasets and a target (AWS service or custom agent API).
2. The evaluator LLM drives multi-turn chats with the target.
3. Responses are scored during the conversation; hooks can assert side effects.
4. Run locally or in CI; fail the build when thresholds miss.

Related: abm-agent-behavior-mining · deepeval · hal-the-holistic-agent-leaderboard · topics/32-eval-harnesses

• Use when: you need multi-turn conversational eval of agents, especially on AWS Bedrock/Q/SageMaker, and want CI hooks.
• Skip when: you only need RAG faithfulness metrics (ragas), pytest-style metric unit tests (deepeval), or red-team/YAML provider matrices (promptfoo).
• Prefer instead: Promptfoo for provider A/B + red team; DeepEval for metric libraries as tests; HAL only for public benchmark calibration (archived harness).

• Evaluator LLM cost/variance: scores depend on another model; calibrate judges.
• AWS gravity: first-class targets are AWS; custom targets work but are more integration work.
• Not production monitoring: conversation harness ≠ live tracing (pair with TruLens/Langfuse-class tools if needed).
• Science shelf: current science/sources mix in HAL/SWE-bench contrast files — Evidence rows cite what exists; deepen sources when claiming Bedrock-specific facts.