promptfoo: open-source prompt, agent, and RAG testing

promptfoo

Open-source tool for testing prompts, agents, RAG systems, and AI security behavior.

promptfoo is an MIT-licensed testing and red-teaming tool for prompts, agents, RAG pipelines, and AI application behavior, with declarative configs and CI/CD-friendly workflows.

Best for Teams testing prompts, agents, and RAG systems

First test Use it on one real builder workflow and verify whether it removes manual work without adding fragile setup.

Resource profile

What is promptfoo?

Use this profile to understand what the resource does, where it fits, and how to test it before adopting it.

promptfoo is an MIT-licensed testing and red-teaming tool for prompts, agents, RAG pipelines, and AI application behavior, with declarative configs and CI/CD-friendly workflows.

promptfoo is listed on OpenAgent.bot as a tools resource for open AI builders.

Fit check

Where it fits

Good fit if

Teams testing prompts, agents, and RAG systems
Developers adding AI evaluations to CI/CD
Builders doing red-team or vulnerability checks on AI workflows

Not a fit if

Teams that need only production tracing
Users who want a benchmark score without writing test cases

Evaluation

What to inspect before adopting it

Fit

Workflow, source, license, and deployment match.

Start with one real use case.

Evidence

Official docs, repository, examples, and maintenance signals.

Trust should come from sources.

Risk

Setup burden, operational limits, and failure behavior.

A small first test keeps adoption reversible.

First test

How to evaluate it before committing

Use it on one real builder workflow and verify whether it removes manual work without adding fragile setup.

Keep the first test small enough that you can inspect the source, understand the permissions, and compare the result with nearby OpenAgent resources.

Workflows

Best workflows to test first

Self hosted ai

Use it as a candidate for self hosted ai when the project facts, license, and official links match your deployment requirements.

Compare

How it compares

When to choose promptfoo

Compare it with nearby tools by looking at hosting model, integration surface, license, and whether the official docs show the workflow you need.

Resource	Category	License	Stars
Langfuse	Tools	See repository	28,327
LiteLLM	Tools	See repository	48,958
MCP Inspector	Tools	See repository	9,954