OpenAI Evals Goes Read-Only on 31 Oct: Migration Checklist

TL;DR: OpenAI Evals becomes read-only for existing users on 31 October 2026 and the dashboard and API shut down on 30 November 2026. OpenAI points users to Promptfoo as the migration destination. Do not plan against 30 November — plan against 31 October: after that date you cannot change or add anything to your evals, so export your datasets, grader definitions, and recent run results this week, rebuild the suite in Promptfoo, and re-baseline every score before the old numbers disappear. The whole move is an afternoon's work for a typical suite, and the same 200-case suite that costs $1.87 per run on a frontier model costs $0.04 on a cheap one through tokens.bd. All dates and prices checked 28 October 2026.

OpenAI's deprecation wave has been rolling all autumn — the legacy model shutdown on 23 October, and now the platform features. Evals is the one that hurts quietly: unlike a model shutdown, it does not break your production calls on a known date. It breaks your ability to know whether your next deploy is better or worse than your last one. A release gate you cannot update is worse than no gate at all, because it keeps returning confident scores while it stops describing your product.

This post is the checklist for the three days before read-only and the thirty days before shutdown: what each date actually means, what to export, how to rebuild in Promptfoo, what it costs to run evals on each model, and how to point the whole thing at tokens.bd's gateway so one suite can test 60+ models.

What exactly happens on 31 October — and on 30 November?#

Two dates, two different kinds of deadline, and confusing them is the main way teams get caught.

All dates checked 28 October 2026 against OpenAI's own evals guide, which carries the deprecation notice at the top of the page.

The practical consequence: November is a read-only migration window, not a grace period. If your evals live in CI or gate releases, the new home has to be working before 31 October, not before 30 November — because any case you discover in November, any grader you need to fix, any new model you need to test, cannot go into the old suite at all.

What does "read-only" actually stop you doing?#

Here is the honest limit of what is publicly documented: OpenAI has said "read-only" and has not published a run-by-run definition of it. Secondary coverage disagrees on the details — one account says suites still run and report but accept no new cases, another says you can look but not run or modify. Do not build your plan on either reading.

Plan against the strict version, and you are safe under both:

• Assume you cannot create, edit, or add cases after 31 October. Any eval change you are considering belongs in this week's work, not November's.
• Assume you cannot rely on scheduled or CI-triggered runs after 31 October. If a run is part of your release process, its replacement must already exist.
• Assume everything stored only in the Evals dashboard disappears on 30 November. Datasets, grader definitions, run history, baselines — if your only copy is in the dashboard, you do not have a copy.

If November turns out to be more permissive than this, you have lost nothing: you migrated early. If it turns out to be stricter, you are already moved.

What should you export before 30 November?#

Export in this order — the first two items are the ones teams most often forget, and they are the ones that cannot be reconstructed later.

1. Your datasets. Every test case: inputs plus expected outputs (ground-truth labels). These are rows; they export in an afternoon and they are the only part of a suite that is truly portable. If your data went in through the dashboard UI, pull it out via the API (GET /v1/evals/{eval_id} and the dataset endpoints) while reads still work.

2. Your grader definitions. This is the hard one. Graders written in a web form have no diff, no author, and no version history — and when you rewrite them by hand in the new harness, the semantics shift slightly, which shifts every score before and after. Write down, for each grader: what it checks, the exact comparison (equality, contains, threshold), and any model-graded rubric text verbatim. A score only means something as a comparison against the same grader, so a rewritten grader is a new baseline, not a continuation.

3. Your recent run history and baselines. A single score is meaningless without the run it came from: which model version, which prompt version, which grader version, which date. Export per-case results — not just aggregates — with model identifier, grader version, and timestamp attached to every row. An aggregate that hides which cases failed is how a suite passes at 92% while every case covering the feature's actual purpose fails.

4. Your prompt versions. Evals test a prompt as much as a model. If the prompt text lives only inside an eval configuration, copy it into your repository now, versioned in git. This matters twice over: reusable prompt objects (pmpt_ IDs, the v1/prompts API) shut down on the same 30 November date, so prompt-in-dashboard is doubly exposed.

5. Your CI wiring. Grep your repositories this week, not next month, for the three strings that identify exposure to the whole 30 November cluster:

grep -rn "v1/evals\|pmpt_\|v1/prompts" --include="*.py" --include="*.ts" --include="*.js" --include="*.yaml" --include="*.yml" .
grep -rn "platform.openai.com/evaluation" --include="*.md" --include="*.yaml" --include="*.yml" .

Anything that matches is either a migration task (evals, prompts) or a dead link waiting to happen. The same grep catches Agent Builder references if you inventory dashboard-built agents alongside it.

How do you rebuild an eval in Promptfoo?#

Promptfoo is OpenAI's named migration destination — it is now part of OpenAI — and it is the lowest-friction rebuild for a dataset-and-grader suite because the concepts map one to one: a prompt, a set of test cases, and assertions that play the role of graders.

A minimal Promptfoo config for the classic ticket-classification eval looks like this:

# promptfooconfig.yaml
prompts:
  - |
    You are an expert in categorizing IT support tickets. Given the support
    ticket below, categorize the request into one of "Hardware", "Software",
    or "Other". Respond with only one of those words.

providers:
  - openai:chat:openai/gpt-6-luna

tests:
  - vars:
      ticket: "My monitor won't turn on - help!"
    assert:
      - type: equals
        value: "Hardware"
  - vars:
      ticket: "I'm in vim and I can't quit!"
    assert:
      - type: equals
        value: "Software"

Run it with npx promptfoo eval. The mapping from OpenAI Evals concepts is direct:

Three migration traps to avoid:

1. Do not copy scores across the move. Your first Promptfoo run is a new baseline. Run it against the same model and prompt, record the numbers, and compare future runs to the Promptfoo baseline — not to the Evals number. Grader rewrites and harness differences make cross-harness score comparisons unreliable.
2. Split the suite while you move it. One small critical set tied to named user journeys, where a single regression fails the build — and one broad set that reports a trend and blocks nothing. A single aggregate across both is what let broken features ship behind passing suites.
3. Set the grader provider explicitly. Promptfoo's model-graded assertions use a default grader model unless you say otherwise. Point defaultTest.options.provider at a model you actually control, or your grading bill and your grading behaviour will both surprise you.

For multi-turn agent tests, Promptfoo covers simulated users, but a dataset row is still the right home for single-turn checks: keep the policy-answer check as a row, and add conversation tests only for what must happen across turns — clarification, recommendation, objection, next step.

What does the same eval suite cost on each model?#

This post's own computed dataset, on one stated assumption so you can scale it to your suite: 200 cases, each using 800 input tokens and 150 output tokens, at tokens.bd list prices checked 28 October 2026, converted at $1 = ৳124.01 (exchange-rates.org, 28 October 2026). Change the case count or token mix and the totals scale linearly — the ratios between models do not change.

Two conclusions fall out of the table. First, evals are cheap on any model in the bottom three rows — the suite costs less than a cup of tea per run, which means there is no cost excuse for not running it on every commit. Second, the frontier rows are 15x (Sonnet) and 45x (Luna-to-Sol) more expensive per run for the same cases, which is exactly why the suite should run on a cheap model by default and reserve the frontier model for the release-gate run, not the other way round.

Add a model-graded rubric and the grader pass costs roughly what the cheap rows cost again: grading 200 cases with a Haiku-rate grader (400 input + 100 output tokens per case) adds about $0.018 per run — 58% on top of a Haiku run. Set the grader model deliberately; a frontier grader on every case quietly doubles or triples the bill.

Run the suite daily in CI for a month and the gap becomes the budget line: $1.26/month (~৳156) on Luna versus $56.10/month (~৳6,957) on Sol for the same 200 cases. On tokens.bd's own plans, $10 of Weekly 5X credit buys about 238 Luna-rate eval runs and $50 of Monthly Max credit buys about 1,190 — evals are not where your budget goes, whichever plan you are on. Running the suite across all five models in the table costs $2.63 per full matrix run, which is the honest price of knowing which model actually passes your cases instead of assuming it.

How do you run evals through tokens.bd?#

Promptfoo's OpenAI provider accepts an OpenAI-compatible gateway directly: set apiBaseUrl to the gateway and apiKeyEnvar to the credential, and use the model ID exactly as the gateway lists it. For tokens.bd:

providers:
  - id: openai:chat:openai/gpt-6-luna
    config:
      apiBaseUrl: https://tokens.bd/v1
      apiKeyEnvar: TOKENS_BD_API_KEY
      omitDefaults: true

defaultTest:
  options:
    provider:
      id: openai:chat:anthropic/claude-haiku-5-5
      config:
        apiBaseUrl: https://tokens.bd/v1
        apiKeyEnvar: TOKENS_BD_API_KEY

That one change — base URL plus key — is the whole gateway advantage for evals: the same suite, the same assertions, and the model under test becomes a one-line change. Swap openai/gpt-6-luna for deepseek/deepseek-v4.1-flash or anthropic/claude-sonnet-5-5 and the suite measures a different model with no other edits. For teams in Bangladesh, the same key bills in taka through bKash or Nagad with no international card, and per-key spend caps keep a runaway CI job from becoming a budget incident — set a cap on the CI key before you wire it into every pull request.

One caution that applies to any gateway eval run: record the exact model ID in every result row. Gateway catalogs move — models are added, renamed, or retired — and a result row that says only "the cheap model" cannot be compared to anything next quarter.

What about the other 30 November shutdowns?#

Evals does not shut down alone. The same date takes two more platform features, and the same week's grep finds all three:

Our 23 October shutdown checklist covers the model side of this deprecation wave; this post covers the platform side. If you have not yet inventoried either, do both greps in the same session — the model IDs and the platform IDs tend to live in the same repositories.

OpenAI Evals migration FAQ#

Can I still create a new eval after 31 October 2026? No — that is the practical meaning of read-only for existing users. New evals belong in the new harness from now on, not in a dashboard that cannot accept them.

Do my old scores transfer to Promptfoo? No. Scores transfer as history, not as baselines. Keep the exported run history for reference, re-baseline in the new harness against the same model and prompt, and compare future runs to the new baseline.

What if I miss 30 November entirely? Your eval configurations, datasets, and run history on OpenAI's platform are gone, and any CI job calling /v1/evals starts failing. The datasets can usually be rebuilt from your own test data if you kept it elsewhere; the run history and grader definitions usually cannot. Export this week.

Is Promptfoo free? The open-source framework is free to run yourself; Promptfoo also offers a cloud product. Either way, the model calls still bill at the model provider's rates — which is why the cost table above matters more than the harness license.

Should I evaluate on the cheapest model or the model I ship? Both, at different cadences: the cheap model on every commit for regression signal, the shipping model on the release gate for the number that actually matters. The cost table shows why that split is affordable — the daily cheap run is $1.26 a month on Luna, and the gate run is a rounding error next to a bad deploy.

Dates checked 28 October 2026 against OpenAI's evals guide and deprecation coverage; tokens.bd catalog and prices checked the same day on tokens.bd/models. Promptfoo configuration follows Promptfoo's own OpenAI provider documentation. Model list prices change; the provider's pricing page is the authority.

Sources: OpenAI evals guide — deprecation notice · Promptfoo OpenAI provider docs — apiBaseUrl for OpenAI-compatible gateways · OpenAI deprecation calendar — Evals, prompts, Agent Builder dates · OpenAI model deprecation dates — November 2026 shutdowns · Evals shutdown — multi-turn agent test migration · tokens.bd model catalog · tokens.bd quickstart — base URL and model IDs · USD/BDT rate, exchange-rates.org