promptfoo
Promptfoo is an open-source CLI and library for testing, evaluating, and security-testing LLM applications. It lets you compare prompt performance across multiple models (GPT, Claude, Gemini, etc.), run automated evaluations, and identify vulnerabilities through red teaming—all with declarative configuration and CI/CD integration.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Repository | promptfoo/promptfoo |
| Owner | promptfoo |
| Primary language | TypeScript |
| License | MIT — OSI-approved |
| Stars | 23k |
| Forks | 2.1k |
| Open issues | 403 |
| Latest release | code-scan-action-0.1.8 (2026-06-16) |
| Last updated | 2026-07-07 |
| Source | https://github.com/promptfoo/promptfoo |
What promptfoo is
TypeScript-based evaluation framework that executes LLM prompts against test cases, computes metrics, and generates comparison reports. Supports local execution, multiple LLM providers via unified API, custom grading functions, and GitHub Actions integration. Now part of OpenAI but remains MIT-licensed and community-maintained.
Get the promptfoo source
Clone the repository and explore it locally.
git clone https://github.com/promptfoo/promptfoo.gitcd promptfoo# follow the project's README for install & configurationNeed it deployed, integrated, or customized instead? DEV.co ships production installs.
Best use cases
Implementation considerations
- Requires valid API keys for LLM providers used in evals; cost scales with number of test cases and model queries. Budget for API spend upfront.
- Configuration via YAML/JSON declarative files; test data organization critical for reproducibility and maintainability across team.
- Custom grading functions (JavaScript/TypeScript or HTTP-based) enable domain-specific scoring but require engineering time to design and validate.
- CI/CD integration via GitHub Actions (or webhook) works well but assumes git-based workflow; self-hosted runners may require network setup for API access.
- Red teaming generates synthetic adversarial inputs automatically; results require human review for false positives and context-specific filtering.
When to avoid it — and what to weigh
- Real-time Production Inference — Promptfoo is an evaluation and testing tool, not a serving framework. Do not use it for runtime request routing or production inference.
- Fully Offline/Air-gapped Environments — Many evaluations require live API calls to LLM providers. Local-only models (Ollama) are supported but not all features work without external connectivity.
- Non-Node.js Teams Without Python Familiarity — Primary distribution is npm; pip install exists but the core codebase is TypeScript. Teams without JavaScript/TypeScript capability may face integration friction.
- Minimal Dependency Constraints — Requires Node.js ^20.20.0 or >=22.22.0 with active version management. Environments on older Node.js or with strict dependency lockdown may struggle with upgrades.
License & commercial use
MIT License. Permissive OSI-approved license allowing commercial use, modification, and redistribution with attribution.
MIT license explicitly permits commercial use. No restrictions on bundling or resale. Promptfoo is now part of OpenAI and remains MIT-licensed; review OpenAI's Terms of Service separately for implications of OpenAI API usage within your product. No commercial support tier documented; community support via Discord.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Low |
| DEV.co fit | Strong |
| Assessment confidence | High |
Promptfoo executes prompts against external LLM APIs—ensure API keys are managed via environment variables and CI/CD secrets, not committed to repos. Local execution means prompts do not transit external networks unless sent to LLM providers. Red teaming may generate/surface harmful outputs; review and filter results before use. No formal security audit or CVE history mentioned; assess custom graders and plugins for injection risks.
Alternatives to consider
LangSmith (LangChain)
Comprehensive LLM ops platform with tracing, evaluation, and dataset management; tighter integration with LangChain ecosystem but steeper learning curve and vendor lock-in.
Braintrust
Evaluation and monitoring platform with built-in evals and dashboarding; more managed/SaaS-focused, less granular control and open-source availability than Promptfoo.
DeepEval
Python-first evaluation framework; better for Python teams but smaller community and fewer out-of-the-box red teaming features than Promptfoo.
Build on promptfoo with DEV.co software developers
Start with Promptfoo's getting-started guide: npm install -g promptfoo && promptfoo init --example getting-started. Deploy evaluations in CI/CD to catch prompt regressions and security issues before production.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
promptfoo FAQ
Does Promptfoo store my prompts or API keys?
Can I use Promptfoo with closed-source or private LLM models?
What is the cost to run evaluations?
Is Promptfoo suitable for production monitoring?
Work with a software development agency
From first prototype to production, DEV.co delivers software development services around tools like promptfoo. Our software development agency staffs experienced software developers and web developers for custom software development, web development, integrations, and ongoing support across ai frameworks and beyond.
Ready to Evaluate Your LLM Prompts?
Start with Promptfoo's getting-started guide: npm install -g promptfoo && promptfoo init --example getting-started. Deploy evaluations in CI/CD to catch prompt regressions and security issues before production.