Technology · Agentic → Evaluations · Open Core · wiki:deep

Promptfoo

Taxonomy

Promptfoo is a CLI and library for evaluating and red-teaming LLM apps: declarative test cases, automated evals, vulnerability scanning, side-by-side provider comparison (OpenAI, Anthropic, Azure, Bedrock, Ollama, …), CI/CD hooks, and PR code scanning for LLM-related issues. Evals can run locally so prompts need not leave the machine.

Primary README (2026): Promptfoo is now part of OpenAI; the project remains open source and MIT licensed. Node.js ≥ 22.22 (24 LTS recommended); also brew/pip/npx.

RepoDocsStack: agentic-evaluations

Why it matters here

Measurement culture means ship evals with every agent/prompt change. Promptfoo is the Declarative + red-team workhorse for this cell: catch injection/security failures and provider regressions before deploy. Complementary to DeepEval (pytest metrics) and Ragas (RAG metric research); stronger when the job is YAML matrices, security probes, and CI.

How it works

  1. promptfoo init --example getting-started (or author a config).
  2. promptfoo eval runs cases against configured providers.
  3. promptfoo view inspects results; red-team commands produce vulnerability reports.
  4. Wire the same config into CI/CD and optional code scanning on PRs.

Related: abm-agent-behavior-mining · agent-evaluation · deepeval · topics/32-eval-harnesses

Flow

Scroll inside the canvas to pan

When to reach for it

  • Use when: you need declarative multi-provider evals, red teaming, or CI gates on prompts/agents/RAG.
  • Skip when: you only need process mining on XES (pm4py), AWS conversation harness (agent-evaluation), or classic RAG metric papers (ragas) without a CLI matrix.
  • Prefer instead: DeepEval for metric-as-pytest; TruLens for OTEL step traces; HAL for public leaderboard calibration only.

Limits

  • Node runtime floor (≥22.22) — env drift breaks CI.
  • OpenAI ownership: license stays MIT, but roadmap/governance may shift — pin versions.
  • Red team ≠ proof of safety: probes reduce risk; they do not replace policy gates on world-changing tools.
  • Local-by-default is a claim about the tool path — still guard secrets in CI logs.

Linked from

Catalog backlinks — what points here (wiki “what links here”).

Features

Stacks

Peers

Document management

What we checked

Claims below are backed by science sources on disk.

Product / CLI eval framing

promptfoo/promptfoo · MODERATE

“Test your prompts, agents, and RAGs. Red teaming, vulnerability scanning, and evals for LLMs.”

Intro / positioning

promptfoo Intro · MODERATE

“promptfoo is an open-source CLI and library for evaluating and red-teaming LLM apps.”

Why evals / red team

Why choose promptfoo · MODERATE

“Simple, declarative test cases: Define evals without writing code or working with heavy notebooks.”

In this research stack

Features and peers linked from the catalog map — not a second product surface.

Features it supports

Same cell

Stack

Document management

Source

Primary repo github.com/promptfoo/promptfoo · Open Core

technologies/promptfoo/README.md

Research inventory

9 tags · 29 out · 31 in · 3 artifacts · 0 gaps · 34 corpus docs

Catalog tags

landscape.layer
Agentic
landscape.subcategory
Evaluations
license_tag
Open Core
maps.dm
present
maps.features
12
one_liner
Agentic
review.depth
science
slug
promptfoo
title
Promptfoo

Artifacts

  • dm_map · present · technologies/promptfoo/document-management.md
  • features_map · present · technologies/promptfoo/features.md
  • readme · present · technologies/promptfoo/README.md

Out · alternative_to

Out · dm_axis

Out · maps_to

In · alternative_to

In · dm_axis

In · in_stack

In · maps_to

Corpus tags

category
Agentic → Evaluations
dedication
open-core
feature
job-legal
outcome-verification
r2-30-evaluator-critic
r2-32-eval-harnesses
r2-33-skill-registry
r2-42-intervention-preassessment
r2-47-trace-maturity-gate
r2-gap-matrix
testing-debugging-optimization
wt-master-synthesis
wt-reuse-deploy-govern
wt-test-debug-evaluate
kind
map_edge
tech_features
tech_quote
tech_readme
tech_science_source
tech_section
needs_deepen
false
quality
ok
section
Document management map
Evidence
Features map
GitHub map
How it works
Limits & failure modes
Links
Scientific notes
What it is
When to use / skip
Why it matters here
slug
promptfoo
source_id
promptfoo-github
promptfoo-intro
promptfoo-why
technology
promptfoo

Corpus documents (34)

map_edge · 12

  • promptfoo → job-legal
  • promptfoo → outcome-verification
  • promptfoo → r2-30-evaluator-critic
  • promptfoo → r2-32-eval-harnesses
  • promptfoo → r2-33-skill-registry
  • promptfoo → r2-42-intervention-preassessment
  • promptfoo → r2-47-trace-maturity-gate
  • promptfoo → r2-gap-matrix
  • … +4 more

tech_features · 1

  • Promptfoo · features

tech_quote · 6

  • Promptfoo · promptfoo-github
  • Promptfoo · promptfoo-intro
  • Promptfoo · promptfoo-intro
  • Promptfoo · promptfoo-intro
  • Promptfoo · promptfoo-why
  • Promptfoo · promptfoo-why

tech_readme · 1

  • Promptfoo

tech_science_source · 3

  • Promptfoo · promptfoo-github
  • Promptfoo · promptfoo-intro
  • Promptfoo · promptfoo-why

tech_section · 11

  • Promptfoo · Document management map
  • Promptfoo · Evidence
  • Promptfoo · Features map
  • Promptfoo · GitHub map
  • Promptfoo · How it works
  • Promptfoo · Limits & failure modes
  • Promptfoo · Links
  • Promptfoo · Scientific notes
  • … +3 more