Promptfoo

What is Promptfoo?

Promptfoo is an MIT-licensed command-line tool and library for evaluating LLM apps. You describe prompts, test cases and assertions in a config file, run them against OpenAI, Anthropic, Azure, Bedrock, Ollama and many other providers, and compare results side by side in a web viewer. It also runs red-team scans for vulnerabilities such as prompt injection and can run in CI. Evals run locally; an optional self-hosted Docker server stores and shares eval results for a team.

Where it fits in a 10–200 person business

• Check that a prompt or model change does not break a support bot's answers before release
• Compare several models on your own test cases to pick the cheapest one that is good enough
• Red-team an internal chatbot or agent for prompt injection and data leaks

These use cases are our editorial suggestions, not claims made by the project.

Can a business use Promptfoo? (license)

This is a plain-English summary of the license text, not legal advice. Third-party models you run with it have their own licenses.

Data & compliance: does Promptfoo phone home?

Checked against the project's own docs and repository on Nov 2, 2026. "Not documented" means we could not find the answer in official sources; it does not mean the feature is missing or that no data is sent. Third-party services you connect (model APIs, vector databases, search engines) have their own data policies. Compare all apps.

What hardware does Promptfoo need?

Server (web UI/API) figures from the docs: "4+ cores recommended", "RAM 8GB+ (16GB recommended for heavy use)", "100GB+ recommended" storage, SSD for the database volume. The server is optional; the CLI alone lists 2+ cores, 4 GB+ RAM, 10 GB+ storage and Node.js 22.22.0 or newer.

Source: www.promptfoo.dev/docs/usage/self-hosting/

How much does it cost to host Promptfoo per month?

How to deploy Promptfoo

Method: Docker (optional results server; the CLI installs with npm)

docker pull ghcr.io/promptfoo/promptfoo:latest
docker run -d \
  --name promptfoo_container \
  -p 3000:3000 \
  -v /path/to/local_promptfoo:/home/promptfoo/.promptfoo \
  -e OPENAI_API_KEY=sk-abc123 \
  ghcr.io/promptfoo/promptfoo:latest
# open http://localhost:3000
# CLI only: npm install -g promptfoo && promptfoo init --example getting-started

From the official instructions: www.promptfoo.dev/docs/usage/self-hosting/. Always check the source for the current version before you run it.

Known gotchas when self-hosting Promptfoo

• The docs say self-hosting is not recommended for production: the server has no built-in authentication, SSO, multi-team support or role-based access control. Keep it on a private network. source
• Results live in a SQLite database in /home/promptfoo/.promptfoo; without that volume mapping, data is lost when the container is recreated. source
• Run only one replica. Jobs live in each server's memory and replicas cannot share the SQLite database, so more than one leads to "Job not found" errors. source
• Basic usage telemetry is on by default (commands and assertion types, plus account email when logged in). Set PROMPTFOO_DISABLE_TELEMETRY=1 to turn it off. source

Alternatives in LLM gateway & observability

• LiteLLM AI gateway that puts 100+ LLM providers behind one OpenAI-compatible API with keys and budgets.
• New API Self-hosted AI API gateway (One API fork) that puts many model providers behind one key, with quotas, usage logs and cost tracking.
• One API Original self-hosted LLM API gateway that puts many model providers behind one OpenAI-style key, with quotas and token management.
• Langfuse LLM observability with tracing, prompt management, evaluations and cost tracking.
• MLflow Open-source platform for LLM tracing, evaluation and prompt management plus classic ML experiment tracking and model registry.
• Opik Open-source LLM observability and evaluation platform for tracing, evals, prompt management and monitoring.
• Phoenix Free self-hosted LLM tracing, evaluation, datasets and prompt management from Arize.

Sources

• GitHub repository: github.com/promptfoo/promptfoo
• Official documentation, Hardware requirements, Deploy instructions, Gotcha 1, Gotcha 2, Gotcha 3: www.promptfoo.dev/docs/usage/self-hosting/
• License text: github.com/promptfoo/promptfoo/blob/main/LICENSE
• Gotcha 4: www.promptfoo.dev/docs/configuration/telemetry/
• Hetzner Cloud pricing (cost estimate): www.hetzner.com/cloud/regular-performance
• GitHub stars, commits and releases: GitHub REST API, fetched Nov 2, 2026