Technology · Agentic → Evaluations · Open Source · wiki:deep

HAL: The Holistic Agent Leaderboard

Taxonomy

HAL is Princeton’s Holistic Agent Leaderboard ecosystem: a public leaderboard plus a standardized hal-eval harness for reproducible agent runs across benchmarks (SWE-bench Verified, USACO, AppWorld, CORE-bench, τ-bench, …). The harness supports local conda/Docker and Azure VMs, Weave logging/cost tracking, and encrypted trace upload to Hugging Face for leaderboard submission.

Important (primary README, 2026): leaderboard results are no longer updated through this harness; the repo is archived for historical reference while work shifts to agent reliability. Public results remain on the HAL site and Reliability Dashboard.

RepoDocsStack: agentic-evaluations

Why it matters here

Before locking a coding/tool agent stack, you need external baselines, not only private golden sets. HAL is the research pointer for multi-benchmark agent comparison and published traces — use historical harness docs to understand how public scores were produced. It is not a production monitor and not a substitute for install-specific CI goldens (Promptfoo/DeepEval).

How it works

  1. Install hal from the harness repo (historically recursive clone + conda).
  2. hal-eval runs agents against a chosen benchmark with Weave logging.
  3. Traces/costs are recorded; optional hal-upload sends encrypted traces for leaderboard integration.
  4. Today: prefer reading published leaderboard/reliability pages; treat the harness as archived reference code.

Related: abm-agent-behavior-mining · agent-evaluation · deepeval · topics/32-eval-harnesses

Flow

Scroll inside the canvas to pan

When to reach for it

  • Use when: calibrating agents against public multi-benchmark baselines or studying published HAL traces/methodology.
  • Skip when: you need live CI for your product scenarios (promptfoo / deepeval / agent-evaluation), RAG metrics (ragas), or process mining (pm4py).
  • Prefer instead: SWE-bench (or peers) directly if you only care about one coding benchmark; Promptfoo for private regression suites.

Limits

  • Archived harness: do not plan new PRs or expect leaderboard updates via this repo.
  • Not production observability: Weave in HAL ≠ company tracing stack.
  • Contamination risk: public traces/benchmarks; encrypted upload exists for a reason — still don’t leak private tasks.
  • Science shelf: packs reuse SWE-bench / Ragas contrast sources — fine for contrast Evidence, not for claiming active maintenance.

Linked from

Catalog backlinks — what points here (wiki “what links here”).

Features

Stacks

Peers

What we checked

Claims below are backed by science sources on disk.

Founding paper / HAL harness

Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation · STRONG

“We introduce the Holistic Agent Leaderboard (HAL) to address these challenges.”

HAL harness / leaderboard role

HAL repo · MODERATE

“SWE-bench, an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories.”

Coding-benchmark contrast

SWE-bench · MODERATE

“Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities.”

Eval-peer contrast

RAGAS contrast · MODERATE

“Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue.”

In this research stack

Features and peers linked from the catalog map — not a second product surface.

Features it supports

Same cell

Stack

Document management

No DM vendor crosswalk edges yet.

Source

Primary repo github.com/princeton-pli/hal-harness · Open Source

technologies/hal-the-holistic-agent-leaderboard/README.md

Research inventory

9 tags · 14 out · 15 in · 3 artifacts · 1 gaps · 30 corpus docs

Catalog tags

landscape.layer
Agentic
landscape.subcategory
Evaluations
license_tag
Open Source
maps.dm
absent
maps.features
7
one_liner
Agentic
review.depth
science
slug
hal-the-holistic-agent-leaderboard
title
HAL: The Holistic Agent Leaderboard

Artifacts

  • dm_map · absent
  • features_map · present · technologies/hal-the-holistic-agent-leaderboard/features.md
  • readme · present · technologies/hal-the-holistic-agent-leaderboard/README.md

Out · alternative_to

Out · maps_to

In · alternative_to

In · in_stack

In · maps_to

Corpus tags

category
Agentic → Evaluations
dedication
open-source
feature
outcome-verification
r2-30-evaluator-critic
r2-32-eval-harnesses
r2-42-intervention-preassessment
r2-47-trace-maturity-gate
testing-debugging-optimization
wt-test-debug-evaluate
kind
map_edge
tech_features
tech_quote
tech_readme
tech_science_source
tech_section
needs_deepen
false
quality
ok
section
Evidence
Features map
GitHub map
How it works
Limits & failure modes
Links
Scientific notes
What it is
When to use / skip
Why it matters here
slug
hal-the-holistic-agent-leaderboard
source_id
hal-github
ragas-contrast
swebench
technology
hal-the-holistic-agent-leaderboard

Corpus documents (30)

map_edge · 7

  • hal-the-holistic-agent-leaderboard → outcome-verification
  • hal-the-holistic-agent-leaderboard → r2-30-evaluator-critic
  • hal-the-holistic-agent-leaderboard → r2-32-eval-harnesses
  • hal-the-holistic-agent-leaderboard → r2-42-intervention-preassessment
  • hal-the-holistic-agent-leaderboard → r2-47-trace-maturity-gate
  • hal-the-holistic-agent-leaderboard → testing-debugging-optimization
  • hal-the-holistic-agent-leaderboard → wt-test-debug-evaluate

tech_features · 1

  • HAL: The Holistic Agent Leaderboard · features

tech_quote · 8

  • HAL: The Holistic Agent Leaderboard · hal-github
  • HAL: The Holistic Agent Leaderboard · hal-github
  • HAL: The Holistic Agent Leaderboard · hal-github
  • HAL: The Holistic Agent Leaderboard · ragas-contrast
  • HAL: The Holistic Agent Leaderboard · ragas-contrast
  • HAL: The Holistic Agent Leaderboard · ragas-contrast
  • HAL: The Holistic Agent Leaderboard · swebench
  • HAL: The Holistic Agent Leaderboard · swebench

tech_readme · 1

  • HAL: The Holistic Agent Leaderboard

tech_science_source · 3

  • HAL: The Holistic Agent Leaderboard · hal-github
  • HAL: The Holistic Agent Leaderboard · ragas-contrast
  • HAL: The Holistic Agent Leaderboard · swebench

tech_section · 10

  • HAL: The Holistic Agent Leaderboard · Evidence
  • HAL: The Holistic Agent Leaderboard · Features map
  • HAL: The Holistic Agent Leaderboard · GitHub map
  • HAL: The Holistic Agent Leaderboard · How it works
  • HAL: The Holistic Agent Leaderboard · Limits & failure modes
  • HAL: The Holistic Agent Leaderboard · Links
  • HAL: The Holistic Agent Leaderboard · Scientific notes
  • HAL: The Holistic Agent Leaderboard · What it is
  • … +2 more