Evals

Promptfoo

Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. The docs describe it as enabling "test-driven LLM development rather than trial-and-error" and producing "matrix views that let you quickly evaluate outputs across many prompts." It supports assertion-based scoring, integrations across OpenAI, Anthropic, Azure, Google, HuggingFace, and open-source models, plus automated red-team and pentest runs that produce vulnerability and risk reports.

GitHub

Documentation

📖
Documentation
https://www.promptfoo.dev/docs/intro/

SDKs

📦
GitHubRepository
https://github.com/promptfoo/promptfoo

Other Resources

🔗
RedTeam
https://www.promptfoo.dev/docs/red-team/

API entry from apis.yml

apis.yml Raw ↑ 
name: Promptfoo
description: Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications.
  The docs describe it as enabling "test-driven LLM development rather than trial-and-error" and producing
  "matrix views that let you quickly evaluate outputs across many prompts." It supports assertion-based
  scoring, integrations across OpenAI, Anthropic, Azure, Google, HuggingFace, and open-source models,
  plus automated red-team and pentest runs that produce vulnerability and risk reports.
humanURL: https://www.promptfoo.dev/
baseURL: https://www.promptfoo.dev
tags:
- Open Source
- CLI
- Red Teaming
- Assertions
- Test-Driven
properties:
- type: Documentation
  url: https://www.promptfoo.dev/docs/intro/
- type: GitHubRepository
  url: https://github.com/promptfoo/promptfoo
- type: RedTeam
  url: https://www.promptfoo.dev/docs/red-team/