Writing / 2026
When Your Model Vendor Buys Your Referee
OpenAI recommends moving Evals users to Promptfoo, which it owns. Keep control of your evaluation cases, judge choice and data flows.
OpenAI’s Evals dashboard and API are scheduled to shut down on November 30, 2026 . Its migration guide recommends Promptfoo , which is now part of OpenAI . Follow that recommendation and your next comparison of competing models can run through an eval harness owned by one of those competitors. The migration solves an immediate problem while creating a dependency worth examining before the replacement becomes your release gate.
I see a structural conflict here. Ownership gives a model vendor an interest in the tool used to assess its competitors. That establishes no bias in a particular result, and treating it as evidence of manipulated scores would outrun the facts. The useful engineering response is to make the evaluation reproducible outside that ownership boundary. Suspicion contributes nothing to a regression decision. Control over the inputs does.
The counterargument deserves its full weight. Promptfoo remains MIT licensed and forkable, releases have continued, and its repository still lists providers including Anthropic and Ollama . Promptfoo says it will continue supporting a diverse range of providers and models . A useful multi-provider tool gives its owner a reason to preserve that usefulness. I see no reason to abandon working evaluations because the owner changed. I do want an exit that survives beyond permission to copy the code.
That distinction matters because some plugins require Promptfoo’s hosted inference and cannot run locally . A fork cannot reproduce that dependency merely by retaining the source. This is the operational extension of asking who owns your eval framework during vendor negotiation : identify which parts you can actually run yourself. Keep your cases and rubrics in versioned files, with assertions you can inspect. Treat hosted functionality as a dependency with an explicit replacement cost.
Start the inspection with the judge. Promptfoo chooses a default grading provider from credentials available in the environment; different credentials can activate different defaults
. That makes the environment part of the evaluation method. Set defaultTest.options.provider or --grader explicitly. My recommendation is to pin a judge from a vendor outside the comparison, then validate its decisions against hand labels.
Choosing another vendor does not make the judge neutral or correct. It separates an interest while leaving you responsible for grading quality. I’ve already argued for owned cases and versioned judges in The Benchmark You Didn’t Build . The additional responsibility here is to make the ownership boundary visible in the configuration. Someone reviewing a result should be able to identify the target and the judge without reconstructing which credentials happened to be installed on the runner.
Data flow needs the same attention. Promptfoo collects basic usage telemetry by default, excluding prompts, model outputs, test cases, provider API keys and full configuration files. Logged-in telemetry includes email and user ID
. Those exclusions matter; describing this as automatic collection of your test contents would be wrong. Set PROMPTFOO_DISABLE_TELEMETRY=1
when you want telemetry disabled. Review the generation and grading paths separately, because disabling usage telemetry answers only the telemetry question.
For red-team generation and grading, Promptfoo’s documentation says it uses your local OpenAI key by default and proxies requests to its API without a key. It also says remote generation is preferred by default when logged into Promptfoo Cloud
. Hosted grading can send the target prompt and response, along with the criteria. Setting OPENAI_API_KEY routes generation and grading through your OpenAI account instead of Promptfoo servers
. That changes the route while keeping the destination within the same ownership boundary.
In CI, set PROMPTFOO_DISABLE_REMOTE_GENERATION=true and supply your own key or a local model
. The documentation explicitly says this setting does not guarantee network isolation or disable telemetry, sharing or Cloud sync. Sharing can upload prompts and outputs within an eval snapshot, so also set PROMPTFOO_DISABLE_SHARING=true or sharing: false
. Block api.promptfoo.app, api.promptfoo.dev and a.promptfoo.app
at egress to test whether the evaluation still runs. Expect plugins that require hosted inference to stop working. Discover that dependency before it blocks a release.
There is evidence for taking implementation choices this seriously without alleging vendor misconduct. Hugging Face compared MMLU implementations: llama-65b scored 0.637 with the original implementation and 0.488 with EleutherAI’s eval harness, while falcon-40b scored 0.571 and 0.527. The ranking flipped . That example establishes that implementation choices can change the winner. It supplies no evidence about Promptfoo’s behavior. The practical consequence is to review the evaluation method alongside the model whenever a result becomes a reason to switch vendors.
Take an existing suite and run it through Inspect or EleutherAI’s evaluation framework . Compare individual cases and investigate disagreements before relying on an aggregate score. This is my recommendation for exercising the fallback, extending the same dependency discipline I apply to vendor risk . If your CI evaluation gate depends on a hosted component, include that component in the exercise. A runnable alternative gives you evidence about portability that a license alone cannot supply.
Next week, assign an engineer to recreate your Evals graders before the scheduled shutdown. OpenAI’s migration guide says recreated LLM-judge graders should be validated before regression decisions . Validate them against hand labels and commit the explicit judge configuration. Run the migrated suite with the Promptfoo hosts blocked before accepting its first release decision.