Evaluations

Evaluations measure whether an agent or model got better or worse: golden sets, LLM-as-judge, trajectory graders, RAG faithfulness, and leaderboard harnesses. Under Agentic they are the quality gate. They depend on Observability for traces and often target RAG or tool-using agents built with Frameworks.

A good eval names the claim, the dataset, the metric, and the fail condition. A bad eval is a chat vibe check with no freeze of prompts or tools. In OS terms, evals are how you prove criterion-shaped improvements (fewer missed requests, faster comparable estimates) instead of demo theater.

Members:

• Ragas and TruLens focus on RAG/groundedness and feedback-style evaluation.
• DeepEval and Promptfoo are general LLM/agent test harnesses.
• ABM and Agent Evaluation lean into agent behavior analysis.
• HAL: The Holistic Agent Leaderboard and process-mining adjacent packs (PM4Py) extend scoring beyond single-turn Q&A.

Prefer RAG-specific scorers when retrieval is the risk; prefer trajectory harnesses when tool misuse is the risk. Feature maps such as Observability & Auditability and Knowledge Grounding & RAG show the coupling: you cannot evaluate what you cannot see, and you cannot claim grounding without citation checks.

Do not ship a framework upgrade without an eval that would have caught last week’s failure. If the only green check is “the demo still demos,” the cell is unused.

Separate offline gates (must pass before merge) from online monitors (drift after ship). Offline sets should include the failures you already saw in the wild — invisible demand, wrong citations, tool loops — not only happy-path demos. When using LLM-as-judge, freeze the judge model and prompt or your metric moves under you. Publish the eval recipe next to the pack so a second estimator of quality can reproduce the number. An eval that cannot fail is decoration; delete it.

For this research catalog the acceptance test is boring and strict: after wiring the pack, can a second person replay the same scenario and get the same measurable outcome? If not, the page is still a brochure. Document the scenario, the expected number, and the log line that proves it. That habit is what separates a deepened taxonomy cell from a landscape screenshot.

Also record what you refused: tools not connected, corpora not ingested, metrics not collected. Refusals keep the cell honest when marketing pages claim universality. A taxonomy essay that only lists winners teaches less than one that names the skip conditions next to the install path.

Keep the essay technical enough that an engineer can argue install order — and short enough that a reader finishes it before opening the right-rail member list.