OpenAI Research

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

Wed, 29 Jul 2026 15:00:00 GMT

Loading…

A sped-up video of GPT‑5.6 Sol attempting to solve puzzles in the ARC-AGI-3 benchmark, with the official harness (left) and our Responses API harness (right), which retains reasoning and enables compaction. On the leaderboard for this game⁠(opens in a new window), no frontier model solves any level beyond the first. With our harness, GPT‑5.6 Sol solves all six.

When we first saw GPT‑5.6 Sol’s low scores on the ARC-AGI-3⁠(opens in a new window) benchmark, we were puzzled.

GPT‑5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture⁠(opens in a new window) and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%.

Were 2D puzzle games unusually difficult for our models? Or was something else going on?

Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting. In the case of ARC-AGI-3, we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set.

With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%. Scores measure Relative Human Action Efficiency (RHAE⁠(opens in a new window))—a metric comparing model performance to a human baseline. Based on official gameplay logs⁠(opens in a new window), we estimate the average human tester scored 48%. Models are not told how they will be scored, and cannot see their score throughout—actions only return a text representation of each frame and what level they are on.

ARC-AGI-3

ARC-AGI-3 is a benchmark designed to measure how well AI agents learn and reason. Agents explore unfamiliar 2D games and infer how they work without explicit instructions. You can play 25 demo games at arcprize.org/tasks⁠(opens in a new window).

ARC-AGI-3 uses an intentionally generic harness, without tools or special features. ARC’s reasoning was that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model’s features and quirks.

In gaming, GPT‑5.6 Sol has beaten Pokémon FireRed with a vision-only harness (as streamed by GPT_Plays_Pokemon⁠(opens in a new window)), Slay the Spire with Codex computer use (as streamed by EpochAI⁠(opens in a new window)), and the first stages of Baba Is You (as shared by Piotr Migdał & Piotr Grabowski⁠(opens in a new window)). What was so different about ARC-AGI-3?

Ethan Mollick shows⁠(opens in a new window) GPT‑5.6 Sol in Codex beating a randomized daily challenge in Slay the Spire 2, a game released after GPT‑5.6 Sol’s knowledge cutoff.

Inspired by ARC’s analysis of GPT‑5.5’s shortcomings⁠(opens in a new window), we examined some of the GPT‑5.6 Sol’s attempts. Like ARC, we saw that the model didn’t appear too bright. It dwelled a long time on each action and struggled to make progress.

But as we looked deeper, we discovered much of the model’s confusion was not inherent to the model itself, but due to settings in the harness.

First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.

Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.

Together, these two features of the harness—discarding reasoning and rolling truncation—helped explain why GPT‑5.6 Sol was struggling to learn over time.

Agents do best when they remember what they’ve done

Our models are trained to think with private reasoning messages before they output replies or tool calls. These private thinking messages are retained as part of the conversation history. If a conversation grows too long, we summarize it and continue.

This is how our models are trained, and also how they are deployed in ChatGPT and Codex. To better match our production setup, we implemented the ARC-AGI-3 harness with our Responses API⁠(opens in a new window). Our API makes it easy to manage context: for GPT‑5.6, passing the previous response ID automatically retains reasoning across tool calls and turns.

With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent strategies.

The next improvement came from replacing rolling truncation with compaction⁠(opens in a new window), another setting in the Responses API.

The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded.

Rolling truncation has two drawbacks. First, the model loses earlier observations and actions. Second, it spends much of the tasks operating with a fuller context window, which can slightly impair performance.

When we enabled compaction on ARC-AGI-3, GPT‑5.6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens.

To illustrate the effect of retaining reasoning and enabling compaction, here’s an animation showing GPT‑5.6 Sol’s 175K context window as it solves a series of ARC-AGI-3 puzzles with each harness.

The two central columns depict how the model context window is used differently by each harness. With better memory of its past, GPT‑5.6 Sol thinks less per action and proceeds much faster. Note: our implementation uses a limit of 175,000 tokens instead of characters, but this ends up being quite similar, as the vast majority of text is action grids which are tokenized at a 1:1 ratio by our tokenizer.

Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.

Conclusion and recommendations

We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. This isn’t the first time we’ve been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages.

If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products:

Use our Responses API, not our legacy Chat Completions API
Retain reasoning
Use compaction

And if you’re comparing models, we recommend relying on evals that use the settings above, which best match real-world use in ChatGPT and Codex.

We are grateful to ARC for their years of creative work on AGI evaluation, and for their analysis that inspired us to take a closer look here.

If you want to test your own mettle against frontier models, try the public games yourself at arcprize.org/tasks⁠(opens in a new window).

Separating signal from noise in coding evaluations

previously⁠, , — Wed, 08 Jul 2026 13:00:00 GMT

Loading…

Accurately measuring our models’ capabilities is important for sound deployment and safety decisions, including decisions under OpenAI’s Preparedness Framework⁠(opens in a new window). With each model release, we report results for a variety of external and internal benchmarks to track model progress. When evaluations have flaws that affect results, they can give a false understanding of capabilities, misrepresenting safety cases and affecting research priorities.

We recently investigated how one of the most widely used coding benchmarks, SWE-bench Verified, had fundamental design and contamination issues, and found that the eval no longer provided meaningful signal on software development capabilities. At the time, we encouraged the wider community to switch to SWE-Bench Pro.

SWE-Bench Pro⁠(opens in a new window) was designed to improve on SWE-bench Verified by testing models on longer horizons and more realistic coding tasks to better track agentic coding capabilities. As in SWE-bench Verified, tasks are sourced programmatically from the history of feature changes in a set of public and private repositories. Models are required to implement a solution that passes new tests for a feature, without breaking existing functionality. On the 731-task public split, frontier models improved from a pass rate of 23.3% to 80.3% in eight months.

We’ve since performed a similar audit on SWE-Bench Pro, reviewing the dataset using a datapoint analysis pipeline. The pipeline reviewed model attempts at the task, task metadata, and failure traces to flag likely evaluation flaws. Each flagged task was then assessed through multiple investigator-agent passes and independently reviewed by five experienced software engineers, with disagreements escalated for further investigation.

We find evidence of breaking issues in a significant portion of the dataset. Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%).

The issues primarily fell into four categories:

Overly strict tests¹ enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.
Underspecified prompts² omit requirements that hidden tests enforce and that are not reasonably inferable.
Low-coverage tests under check the requested feature, so incomplete fixes can pass.
A misleading prompt points models toward the wrong behavior or contradicts what tests require.

Our findings point to the difficulty of curating hard but fair benchmarks and the growing utility of agents for scalable data quality checks. In light of these results, we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results.

Methodology

Our aim is to ensure that task failures reflect genuine model limitations, and task successes reflect complete and valid solutions to the prompt requirements. To check the quality of the data used in the evaluation, we created a quality assurance pipeline to assess whether each datapoint accurately reflects model capabilities.

An initial data quality pipeline flags problems for review. We validate with a deeper agent-assisted audit of flagged tasks and a human annotation campaign working with experienced engineers.

An initial automated filter reviews the instructions given to the model, attempts by the model to solve the task, and the tests used to grade these attempts to flag likely broken or problematic examples. This filter flagged 286 potentially broken tasks. We then conducted a deeper review of that subset in two ways: a human-supervised agent review, which conducts extensive checks with investigator agents and a final human judgment; and a human annotation campaign working with experienced software developers.

Human-supervised agent review

Each flagged problem is audited with Codex-based investigator agents that were given access to the task repository and environment. This helps them distinguish reasonable task ambiguity, which can often be resolved by studying nearby code and repository conventions, from true underspecification. The agent can run tests, inspect files in the repo, and investigate model attempts and their common failure modes on the task. After several independent repeats of these deeper audits, a researcher reviewed the summaries, made a final judgment, and labeled the likely issues.

Human annotation campaign

In parallel, we ran a human annotation campaign over the flagged subset. We worked with experienced software engineers who were trained on the benchmark goals, issue taxonomy, and edge cases before reviewing tasks. Each task was reviewed by five engineers.

Reviewers formed an independent judgment from the visible problem statement, test cases, and the ground-truth reference solution (known as the gold patch) before using the pipeline analysis or transcript as supporting context. The reviewers then assigned a label and severity rating based on concrete evidence, and escalated disagreements or low-confidence cases for further review.

Human reviewers were more likely than the investigator agents to mark tasks as broken. There was also some disagreement on categories between the two review paths, but in no flagged task was “not broken” the most common human label. Of the categories the agent pipeline flagged, reviewers’ judgments overlapped in 74% of cases.

Compared with the agent pipeline, the human reviewers were also more likely to select multiple labels for a task, indicating that they found tasks to be broken in multiple ways or did not fit cleanly into a single category. This suggests the agent-plus-reviewer pipeline resulted in conservative labeling: it captured the same broad failure modes humans identified, while undercounting cases where reviewers saw additional or overlapping issues. The largest difference was in low-coverage tests, which humans selected as the most common issue for 9.4% of the benchmark compared with 4.1% from the agent pipeline.

Failure modes

In several cases the task prompt prescribed a specific implementation, but the hidden test cases expected different behavior.

This task involves normalizing table-of-contents entries and rendering them back to Markdown via TocEntry.to_markdown(). The task prompt specifies serialization down to character-level spacing, describing how exact spacing and pipes are enforced, and gives examples such as " | Chapter 1 | 1" and "** | Chapter 1 | 1":

None

1"[space]| Chapter 1 | 1"
2"**[space]| Chapter 1 | 1"
3"[space]| Just title | "

The hidden test_to_markdown assertions instead require " | Chapter 1 | 1" and "** | Chapter 1 | 1":

None

1"[space][space]| Chapter 1 | 1"
2"**[space][space]| Chapter 1 | 1"
3"[space][space]| Just title | "

There are two leading spaces in the hidden tests, but the example given to the model only contains one leading space. If a model rightly follows the given prompt, that one-character difference would fail the hidden test cases and the task would be marked incorrect.

Discussion

The issues we have identified, coupled with similar cases in SWE-bench Verified, highlight the importance of rigorously checking benchmarks. Issues and pull requests from open-source repositories were originally created for human collaboration, often through long back-and-forths between maintainers and contributors. As a result, problem descriptions, merged code, and unit tests do not always line up to form clean, isolated tasks for evaluating models reliably. In particular, tests included in pull requests can be overly strict because they are written to validate a specific change, rather than to define an implementation-agnostic standard for solving the task.

At the same time, evaluation flaws are easier to detect now than they would have been even a short time ago. As model capabilities improve, we can use those models to inspect prompts, tests, patches, traces, and edge cases with much greater depth and consistency, helping surface benchmark issues that were previously costly or impractical to find at scale.

We hope the wider evaluation community will develop new benchmarks built by experienced software developers specifically to test model capabilities. That approach can preserve the high bar and realism we want to measure model capabilities, and allows for better human oversight throughout the process. Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.

Ultimately, an eval should provide meaningful signal through benchmarks that are hard to game, easy to trust, and genuinely reflective of model capability or alignment. Because these results inform OpenAI’s deployment and safety decisions, the evals we track need to be valid and informative.

Introducing GeneBench-Pro

Tue, 30 Jun 2026 00:00:00 GMT

Loading…

Scientific data rarely arrive with instructions. Researchers must decide whether a pattern reflects biology or noise, whether the data can support the question being asked, and how each result should change what they do next. AI agents are increasingly capable of executing complex analyses, but real scientific research also depends not simply on recalling facts or following a predefined workflow but also on making these higher-order judgments.

Today, we’re introducing GeneBench-Pro—a challenging, research-level benchmark for testing whether models can handle the kind of judgment-heavy analysis that real-world computational biology requires. It expands on GeneBench⁠(opens in a new window) to cover harder, more realistic tasks across genomics, quantitative biology, and translational medicine, capturing the complexity, iterative nature, and ambiguity of scientific research in computational biology.

To date, there have been few convincing assessments of the system-level judgment calls that make real-world computational research difficult. These include handling ambiguity, revising assumptions, choosing the correct analysis path, and knowing when a result is decision-ready. Because these skills are difficult to formalize, they are also difficult to assess rigorously, even as weaknesses in them increasingly constrain overall AI performance.

GeneBench-Pro is designed to precisely measure these higher-level capabilities. Within GeneBench-Pro, we define “research taste” as the chains of judgment calls that shape an analysis: which questions the data can support, how early diagnostics should change the model or estimand, and when an initial plan needs to be revised. Each GeneBench-Pro problem gives the model a realistic and messy dataset, brief experimental context, and a target estimand tied to a downstream decision. To answer correctly, the model must explore the data, choose an appropriate analytical approach, engage in an iterative process of experimentation, and supply a final answer.

Dataset construction

In biology, the cost of data generation (e.g., genome sequencing) has fallen dramatically, and some researchers now argue⁠(opens in a new window) that the limiting factor is no longer sample collection but downstream computation and analysis. GeneBench-Pro is built to assess progress in addressing that bottleneck, with 129 questions covering a broad range of computational biology settings and methods.

Domain Atlas: 129 problems in 10 domains and 21 sub-domains

Click on a dot above to learn about a benchmark problem.

This atlas provides a preview of the breadth of GeneBench-Pro. Visit the case studies page to explore 10 representative questions in more detail.

GeneBench-Pro is also designed to avoid common benchmark failures. Many long-horizon biology benchmarks construct multi-step questions around messy historical datasets, where there may be no single correct path through the analysis. An agent might choose one defensible cutoff, while another might choose a different but equally defensible option, reflecting the arbitrary choices made by the benchmark creator more than any fundamental differences in model performance. The reverse can also happen: if a problem is too numerically insensitive, an agent can make fundamental errors in an analysis and still produce a passing result.

To avoid these failure modes, each GeneBench-Pro problem is built synthetically: we know the full causal structure and directly simulate the data-generating process. That enables us to tune the complexity of each problem, ensure that reasonable differences in subjective analytical choices still produce accepted numerical results, and verify (through ablation studies) that plausible but incorrect analyses fail. We then audit problem drafts through detailed trace analyses to check for information leakage and unintended solution pathways. This gives us confidence that getting the right answer depends on choosing the correct analytic pathway and not on exploiting a shortcut or matching an arbitrary author preference.

We sent 82 of the 129 GeneBench-Pro questions to external domain experts, including graduate students, postdoctoral researchers, industry scientists, and professors. Reviewers assessed each problem’s realism, whether the target answer was identifiable, and whether the methods and estimators were appropriate. Feedback was used to improve problems.

“The problems I reviewed would have been challenging for a graduate student to complete without iterated feedback from an experienced supervisor. The data contained technical and quality control issues that required thoughtful and reflective data analysis with awareness of potential pitfalls to complete successfully; they were not simply applying some off-the-shelf method to clean and well curated data.”

Alexander Strudwick Young, Assistant Professor in Human Genetics at UCLA

“Even if current models aren’t able to reliably run independent analyses from beginning to end, ones that perform well on GeneBench-Pro problems clearly would be able to assist researchers in determining correct workflows and exploring data. I could see that greatly improving the pace, thoroughness, and reproducibility of research.”

Jennifer Grundman, PhD Candidate in Human Genetics at UCLA

1 of 2

“The problems I reviewed would have been challenging for a graduate student to complete without iterated feedback from an experienced supervisor. The data contained technical and quality control issues that required thoughtful and reflective data analysis with awareness of potential pitfalls to complete successfully; they were not simply applying some off-the-shelf method to clean and well curated data.”

Alexander Strudwick Young, Assistant Professor in Human Genetics at UCLA

Evaluation and grading

Each GeneBench-Pro problem is a self-contained scientific analysis. Agents receive access to an isolated workspace with a short prompt, data files, and a standard bioinformatics stack including Python, scientific computing libraries, and basic genomics packages like PLINK 2.0 (although the problems do not require domain-specific tooling).

Structural variant-guided tumor therapy benefit-risk decision

A molecular tumor board registry contains trial-eligible advanced solid-tumor cases considered for a TXR1-directed inhibitor. Estimate, for tumors with SV-driven TXR1 target-mediated activation at time zero, the marginal effect of TXR1i versus non-TXR1 systemic therapy on week-16 clinical benefit as if all patients had an assessable week-16 visit. Also estimate the 8-week treatment-limiting toxicity/discontinuation risk under TXR1i in the same target population. Report net clinical utility = benefit risk difference (percentage points) - 0.35 * toxicity risk (percentage points), and choose therapy_class_code 1 if TXR1i has positive net utility and 0 otherwise.

Use percentage-point units for all non-code quantities. Positive benefit means TXR1i improves week-16 clinical benefit relative to non-TXR1 systemic therapy.

These data came from a real experiment; you will be graded not just on numerical correctness but the quality of analytical reasoning you exhibit; do not attempt to take any shortcuts.

Return your final answer as exactly one JSON object.
Do not wrap the JSON in markdown.
Do not add prose before or after the JSON.
Do not omit any keys shown in the example.
Return the JSON object in your final answer:

JSON

1{
2  "answer": {
3    "therapy_class_code": <int>,
4    "benefit_rd_pp": <float>,
5    "toxicity_dropout_risk_pp": <float>,
6    "net_clinical_utility_pp": <float>
7  },
8  "reasoning": "<description of method and QC>"
9}

Because we control the full data-generation process, we can grade correctness deterministically against known targets, avoiding model-choice variability and verbosity effects found in standard rubric-based evaluation.

Each problem also comes with rich metadata, including the intended analysis structure, attached data files, a detailed multi-page case study, and expert review outcomes. We are fully open-sourcing 10 representative GeneBench-Pro questions on Hugging Face⁠(opens in a new window), with an interactive web interface for browsing them. Finally, we will provide a 50-question subset to Artificial Analysis⁠(opens in a new window) for independent, third-party benchmarking in the near future.

Results

Our strongest model, GPT‑5.6 Sol, attains a pass rate of 28.7% at the highest reasoning level (31.5% with Pro mode enabled). That is a sharp increase from when we began building the original GeneBench; at that time, our best frontier model, GPT‑5, scored below 5%. Progress on this benchmark suggests that frontier models are improving quickly, even on less tangible, systems-level scientific reasoning. At the current pace, this benchmark may be saturated by the end of the year.

The results also show the impact of scaling test-time compute. At the lowest reasoning level, GPT‑5.6 Sol only achieves a single-digit passrate. At the highest reasoning level, GPT‑5.6 Sol solves nearly six times as many questions as GPT‑5.2 does while using about two-thirds as many tokens.

Comparisons across model families suggest that GPT models are among the strongest systems at high-level scientific reasoning under quantitative uncertainty. The performance gap between GPT‑5.6, GPT‑5.5 and leading open-source models such as GLM 5.2 is significantly larger than we would expect when extrapolating from coding benchmarks⁠(opens in a new window), indicating that open-source models are more specialized for coding than for broader reasoning ability.

We used frontier GPT models to evaluate and harden problems during development. As such, we suspected GeneBench-Pro might be biased against GPT models relative to other model families. However, competitor models at best matched the performance of the corresponding GPT model at the time of release, and tended to fall short considerably.

These evaluation results—as high as 31.5% on GPT‑5.6 Sol (Pro)—are striking given the difficulty of the GeneBench-Pro questions. In a survey, our reviewers estimated that a typical GeneBench-Pro problem would take a human expert around 20–40 hours to complete. At a conservative $200 per hour, that puts the human labor cost of a single problem in the thousands of dollars. Current AI agents are still too unreliable to replace human experts, but the cost gap is large, with inference costs at only several dollars per problem. That means even partial automation at current capabilities could create meaningful economic and scientific value.

“The benchmarks are motivated by a diverse range of biological questions, but … the actual challenge comes from exploratory data analysis and reasoning upon these discoveries: identifying patterns and artifacts, and deciding whether the data should be excluded or adjusted. This resembles the messy nature of real biological datasets. Reviewing these evaluations highlights how important clear solver contracts are for agent-based scientific problem solving. Different prompt wording or task specification can greatly affect which analyses appear permissible.”

Cyrillus Tan, Postdoctoral Research Associate at the New York Genome Center

“I liked [the questions] mostly. They tended to have a mix of: (1) Required knowledge of the subject, such as C>T bias in ancient DNA, (2) Data discrepancies, such as ancestry swaps, (3) A kind of knowledge of the right analytical tools for the job and how to implement them. It seemed like most of the agents failed on (2). They aren’t cautious enough about data issues. Maybe that highlights a weakness of current models. And a lot of biological data has irregularities.”

Lex Flagel, Director of Data Science at Gencove

1 of 2

“The benchmarks are motivated by a diverse range of biological questions, but … the actual challenge comes from exploratory data analysis and reasoning upon these discoveries: identifying patterns and artifacts, and deciding whether the data should be excluded or adjusted. This resembles the messy nature of real biological datasets. Reviewing these evaluations highlights how important clear solver contracts are for agent-based scientific problem solving. Different prompt wording or task specification can greatly affect which analyses appear permissible.”

Cyrillus Tan, Postdoctoral Research Associate at the New York Genome Center

“I liked [the questions] mostly. They tended to have a mix of: (1) Required knowledge of the subject, such as C>T bias in ancient DNA, (2) Data discrepancies, such as ancestry swaps, (3) A kind of knowledge of the right analytical tools for the job and how to implement them. It seemed like most of the agents failed on (2). They aren’t cautious enough about data issues. Maybe that highlights a weakness of current models. And a lot of biological data has irregularities.”

Lex Flagel, Director of Data Science at Gencove

Still, the fact that frontier models still solve fewer than a third of these problems shows that there is substantial room for improvement. Models can make partial progress on challenging problems, but they struggle to close the inferential loop. This failure pattern mirrors the contrast between human experts and novices. Experts use their experience to frame the problem and adapt their approach, while novices make observations but struggle to integrate them into the broader context of the problem.

Problem: Pharmacogenomic time-to-event response with time-varying treatment

Treatment initiation, genotype-specific response, delayed pharmacodynamics, prevalent-user flags, and longitudinal biomarkers jointly determine the causal survival estimand.

GPT-5.5 pattern

Handles treatment timing with a conventional Cox outcome model but does not address treatment-confounder feedback.

Fit a counting-process Cox model with treatment as a time-varying exposure, effective only after treat_start+90 days ... The model included G, treatment×G, baseline severity, age, and sex.

GPT-5.6 Sol pattern

Uses a more appropriate causal inference method to properly account for treatment-confounder feedback.

Used a new-user marginal structural Cox model: excluded 818 flagged prevalent users, modeled treatment initiation with stabilized inverse-probability weights using baseline covariates and current biomarker, and treated exposure as time-varying with a 90-day efficacy lag.

Achieving near-perfect performance will require evaluations that both reliably measure progress and identify where models still fail. Benchmarks like GeneBench-Pro can help to turn a vague capability deficiency into something we can diagnose and improve.

If agents can reliably automate this class of analysis, they could significantly accelerate scientific discovery. Human genetic evidence is already central to target prioritization and translational follow-up, because mechanisms with genetic support are much more likely to lead to approved treatments.

Meanwhile, sequencing costs have plummeted, and biobank-scale datasets now link molecular, phenotypic, and health-record information at unprecedented breadth. The limiting factor is shifting from data generation to turning the information into actionable insights. Models that can consistently perform analyses now handled by teams of human experts could transform industrial research by accelerating hypothesis triage, target follow-up, and the iteration cycle between data generation and decision-making.

GeneBench-Pro represents an initial effort to evaluate the more abstract skills involved in good scientific judgment possessed by experienced. These skills allow them to intuit and identify the most promising initial analyses, iterate and revise their thinking when data contradict initial assumptions, and arrive at conclusions upon which downstream clinical, academic, or business decisions may depend.

We anticipate that as model capabilities advance, benchmarks that probe model abilities at these higher levels of abstraction will become increasingly useful, beyond those that simply test book knowledge or the ability to execute routine analyses.

A near-autonomous AI chemist improves a challenging reaction in medicinal chemistry

Wed, 17 Jun 2026 10:00:00 GMT

OpenAI’s work in science is motivated by a simple belief: advanced AI can become a powerful partner for scientists, helping them explore more ideas, connect distant concepts, design better experiments, and accelerate discoveries that benefit humanity. We have already shared early examples of models contributing to novel results in mathematics, including work on the unit distance problem⁠, in theoretical physics, through a new result on gluon amplitudes⁠, and in biology, where GPT‑5 helped lower the cost of cell-free protein synthesis⁠ in an automated lab. We also introduced GPT‑Rosalind⁠, a purpose-built model to support life sciences research and drug discovery workflows.

This project extends that trajectory into medicinal chemistry, where progress cannot be measured by reasoning alone. A hypothesis has to work in the lab with real molecules, instruments, and experimental noise. Working with Molecule.one⁠(opens in a new window), we connected GPT‑5.4 to Maria—an agentic chemistry AI integrated with a high-throughput laboratory for autonomous research—and gave it an open-ended goal: to improve one of several important reaction classes. The system generated research proposals, designed and ran experiments, analyzed experimental data, and proposed follow-up experiments. Humans remained in the loop by designing steering and grading prompts and selecting proposals to test. They also made limited corrections to experimental plans, assisted with basic laboratory operations, and independently validated the final result.

The most promising proposal, OAI-M1-03, focused on a difficult but useful version of Chan–Lam coupling, a reaction chemists use to form carbon-nitrogen bonds. Starting from the open-ended goal of improving Chan–Lam coupling for process chemistry, GPT‑5.4 independently identified primary sulfonamides as a challenging, high-value substrate class and suggested that mild oxidants, including TEMPO, could improve the reaction.

Across two cycles of experimentation in Maria Lab that idea produced a significant improvement. Under the optimized conditions, measured yields improved for 88% of the boronic acids and 83% of the sulfonamides tested. The mean yield rose from 16.6% to 25.2%, and the share of reactions above 30% yield increased from 15.6% to 37.5%. Human chemists then repeated representative reactions at bench scale. Those experiments confirmed the microliter-scale results, showing higher yields for 11 of 14 substrate pairs, with a more than twofold increase in most cases. That matters because medicinal chemists need reactions that work not just in micro-liter screening experiments, but also in practical lab workflows used during drug discovery.

Improvements in this area of medicinal chemistry are particularly exciting because synthesis is often a major bottleneck in drug discovery: scientists can only test the molecules they can make or otherwise obtain. The sulfonamide group appears in medicines across a wide range of therapeutic areas, including anticancer drugs, antimicrobials, and diuretics, yet the Chan–Lam coupling of primary sulfonamides with boronic acids has historically given low yields. Making this form of the reaction more reliable could give medicinal chemists a broader and more practical way to produce and explore potentially useful molecules.

While this is still an early result, it provides another concrete example of the broader direction we are working toward: AI systems that can become valuable partners to scientists across much of the research loop. The model reviewed the literature, proposed an unexpected idea, helped design and analyze experiments, and arrived at a scientific finding that human chemists could evaluate.

Maria Lab: Molecule.one's specialized high-throughput laboratory that ran 10,080 reactions in OAI-M1-03

Why the chemistry problem matters

Organic chemistry underpins all small-molecule medicines, as well as products in agriculture, electronics, and materials science. A reaction is especially useful when it can make the same kind of chemical bond reliably across many different starting materials. When reactions produce low yields or too many unwanted byproducts, chemists may have to abandon otherwise promising molecules or spend significant time developing a different route. This makes synthesis a major bottleneck in drug discovery: scientists can generally only test the molecules they can make or otherwise obtain.

Chan–Lam coupling is useful in medicinal chemistry because it forms carbon-nitrogen bonds, which are common in medicines. However, the reaction does not work equally well for every class of molecule. In particular, coupling primary sulfonamides with boronic acids has historically produced low yields. Sulfonamides are an important family of molecules found in medicines used in oncology and infectious disease. Making this reaction more reliable could give medicinal chemists a broader and more practical way to produce and explore potentially useful molecules.

Connecting GPT‑5.4 to Maria AI and Lab

The combined system paired complementary capabilities. Prompts written by scientists working with Maria AI were used with GPT‑5.4 within a harness to generate and rank thousands of possible research proposals. Human chemists reviewed the small subset of proposals that ranked highest according to the system and selected four for laboratory testing. Maria AI then translated selected high-level plans into detailed lab instructions, ran thousands of high-throughput experiments, analyzed the raw data, and returned structured results to GPT‑5.4.

One of the four selected proposals, OAI-M1-03, suggested using mild oxidants such as TEMPO to improve the performance of the Chan-Lam reaction for sulfonamide synthesis. Chemists found the suggestion both surprising and interesting. We share the detailed findings from OAI-M1-03 in this blog post and in the paper⁠(opens in a new window). We’re also sharing the rewritten model chain-of-thought⁠(opens in a new window) for OAI-M1-03.

The final research proposal was then used by Maria to generate experimental grids, with slight corrections by humans. The largest human correction was to avoid dimethyl sulfoxide, or DMSO, as a solvent because chemists were concerned it could react with the stronger oxidants used as comparisons.

The full process took three months, from the first prompt on March 4th to sharing the OAI-M1-03 results with independent experts on June 4th.

We describe this workflow as near-autonomous, not fully autonomous, because human chemists still made important decisions throughout the process. The model proposed the key research ideas, while human chemists provided high-level steering and judgment, corrected experimental details, helped prepare lab consumables and reagents, and repeated key experiments by hand.

What we found

OAI-M1-03 identified TEMPO as a useful additive for the primary sulfonamide Chan-Lam coupling studied here. Under the optimized conditions, the reaction improved in two ways: average yield went up, and more substrate combinations reached practically useful yields.

Across two cycles, Maria ran a total of 10,080 reactions – more than a chemist running three reactions every day would run in a decade. That scale mattered because chemistry results can be misleading when they are tested on only a few examples. A reaction can look promising on one pair of starting materials, but fail across a broader set of molecules. Thousands of reactions made it possible to identify TEMPO among ten tested oxidants, see the effect repeat across diverse combinations, and find its limitations.

After analyzing the first round of data, the system proposed a more focused second round of experiments to test follow-up hypotheses. One useful follow-up finding was that TEMPO could be replaced by a much cheaper analog, 4-hydroxy-TEMPO, with little loss in performance.

The result also held up beyond Maria Lab’s microliter-scale screening format. Human chemists reproduced representative reactions manually at bench scale and observed an increase in yield for 11 of 14 substrate pairs; for eight pairs the increase was greater than twofold. That replication matters because very small-scale experiments can sometimes introduce artifacts that disappear at a larger scale. Bench-scale validation is also customary before research is published in a scientific journal.

Reaction vials from the manual bench-scale validation.

TEMPO improves product formation at bench scale

Four external chemistry experts reviewed the preprint describing OAI-M1-03. Their assessments supported our view that the result was novel and worth sharing with the scientific community. The stronger test will come next: whether independent labs can reproduce the result, and whether chemists find it useful across a broader range of molecules.

“The merger of high throughput experimentation and modern AI represents a new frontier of scientific discovery. This new reaction is a powerful demonstration, where exceptionally mild conditions and a practical oxidant enable a nicely general substrate scope for one of the more popular reactions in drug synthesis.”

—Tim Cernak, Associate Professor of Medicinal Chemistry, University of Michigan

Of the other three proposals generated by GPT‑5.4 and tested by Maria during the three-month period, OAI-M1-02 and OAI-M1-04 were experimentally proven in the Maria Lab, while OAI-M1-01 was disproven. Analysis of these results is ongoing.

Limitations

This work shows that a model can make a useful contribution in organic chemistry. It did more than summarize the literature or suggest a one-off experiment: it proposed a specific surprising hypothesis and surfaced it for human review, designed experiments, interpreted experimental data, and designed follow-up experiments.

It does not show that AI can independently run a chemistry research program from end to end. Human judgment remained essential, and the workflow depended on specialized high-throughput infrastructure. It also does not establish that the method will generalize to other coupling reactions, other substrate classes, or manufacturing conditions.

The yield estimates came from a high-throughput platform, and bench validation covered 14 representative substrate pairs. More work is needed to characterize the reaction mechanism, define the substrate scope, measure performance under different laboratory conditions, and reproduce the result independently.

Preparedness

Chemistry capabilities require careful treatment because the same tools that can support medicine and materials science could also be misused. We deliberately scoped this work to a legitimate medicinal-chemistry problem: improving a known coupling reaction used to make drug-like molecules. The experiments did not involve toxins, chemical weapons, or requests to design harmful compounds. These results should not be read as evidence that the system can help with those harmful applications. The project did not test or demonstrate that.

We assess and mitigate emerging risks from advanced model capabilities through our Preparedness Framework⁠, including risks related to chemical and biological domains. The model used in this work had already undergone relevant evaluations with the UK AI Security Institute, and the system was designed to refuse requests focused on harmful applications. The experimental workflow added another layer of control: human chemists selected which proposals entered the lab, reviewed experimental plans, and retained control of the physical infrastructure.

We think this is the responsible way to study AI's potential in experimental chemistry: choose a problem space with clear scientific value, pair model-level safeguards with expert oversight, and evaluate the system through constrained physical experiments. As these capabilities improve, we will continue to assess emerging risks, strengthen safeguards, and be specific about what a result does and does not imply.

What’s next

The immediate next steps are scientific: test a broader range of starting materials, investigate why the additives improve the reaction, map where the effect works and fails, and support independent replication. Together, these studies will determine how broadly the method can be applied and how useful it is in practical medicinal chemistry workflows.

Our longer-term goal is to make AI systems reliable scientific partners that help researchers generate hypotheses, design experiments, interpret results, and decide what to test next, while remaining grounded in expert judgment, reliable measurement, and strong safeguards. Organic chemistry is a particularly high-leverage area because progress in small-molecule discovery and manufacturing depends on being able to make molecules reliably. Scientists can only test molecules they can make, and better synthesis can expand the range of ideas they can explore across medicine, agriculture, electronics, energy, and materials science. This result is one early example of that broader direction: a frontier model, specialized agents, an automated laboratory, and human chemists working together to move faster through the research loop and produce findings the scientific community can evaluate, reproduce, and build on.

We are grateful to the Molecule.one team and to the independent chemists who reviewed this work.

Introducing LifeSciBench

Wed, 17 Jun 2026 00:00:00 GMT

Loading…

Agentic AI systems are becoming increasingly capable of performing scientific tasks. However, their usefulness to life science researchers depends on how well they handle the complexity of real research. That work rarely looks like a single fact-recall question or a clean prediction problem. Researchers interpret incomplete evidence, reconcile conflicting results, design difficult experiments, troubleshoot assays, evaluate translational risk, and decide what to do next under uncertainty.

Current benchmarks do not fully capture these capabilities. Many life science evaluations focus on narrow domains or isolated skills, resulting in questions with structured question formats and clean reference answers. While valuable, they often fail to truly assess whether a model can contribute across the broader span of research-level work.

We designed LifeSciBench to help close this gap. Every task is grounded in the judgment of practicing life scientists with Ph.D.-level training and direct experience advancing drug discovery programs in biotech and pharmaceutical settings.

LifeSciBench includes 750 expert-authored tasks spanning seven workflows and seven biological domains.

1,062

Task artifacts

173

Scientist contributors

19,020

Rubric criteria

453

Expert reviewers

What LifeSciBench measures

LifeSciBench measures whether AI systems can support realistic life science research tasks, not just answer biology questions. To define the benchmark taxonomy, we surveyed practicing life scientists about the workflows they use most often in applied research settings. Then, we grouped their responses into seven recurring categories: evidence handling, analysis, design and optimization, scientific reasoning, validation and operations, translation, and scientific communication.

Each task is structured like a request a scientist might give to a knowledgeable collaborator: scientific prompt, any relevant context or artifacts, and a free-response answer. Expert-written rubrics evaluate whether a model can produce the right answer for a specific problem, with the right level of detail, justification, caveats, and formatting a scientist would expect.

Dataset construction

LifeSciBench evaluates scientific reasoning alongside the less well-defined, practical skills necessary for real-world scientific use. Its tasks ask models to work through realistic research problems: interpreting evidence, making domain-grounded judgments, and communicating conclusions that would be useful to expert reviewers. Many tasks also require models to handle uncertainty and reason over supporting data files rather than relying on prompt text alone.

The benchmark is designed to reflect the complexity of life science work. Overall, 79% of tasks require multiple reasoning or decision-making steps, with an average of four steps per task. LifeSciBench includes 1,062 attached artifacts spanning figures, PDFs, tables, sequence files, structure or chemical files, and web references. More than half of tasks (53%) require models to interpret or synthesize information from at least one artifact.

Tasks were created by 173 expert scientists across different life science disciplines. Each scientist had Ph.D.-level training and biotechnology or pharmaceutical industry experience. Tasks could undergo as many revision cycles as needed before acceptance, with no fixed cap on the number of rounds; accepted tasks averaged six self-directed automated review cycles and completed at least two rounds of expert reviews. Reviews were anchored in either a verifiable correct answer or strong expert consensus, with at least 90% agreement among reviewers in the relevant domain. This process helped ensure that accepted tasks were scientifically grounded, clear enough to grade, and representative of applied research.

Grading and rubric breakdown

LifeSciBench tasks are graded with a detailed, task-specific rubric that breaks down the expected response into specific scientific claims, calculations, decisions, justifications, and so on. Across the benchmark, expert-developed rubrics include 19,020 criteria—an average of 25 per task—to assess both scientific correctness and usefulness for research decisions.

This design reflects how scientific work is evaluated in practice: many life science tasks cannot be graded by checking the final answer alone. A response may reach the correct high-level conclusion but still be judged incomplete if, for example, it overlooks a key assay limitation or fails to proactively bring up a highly consequential biological nuance. Conversely, a partial response may contain high-quality reasoning even if it does not fully solve the task.

The granular rubrics capture this nuance. LifeSciBench evaluates not only final-answer accuracy, but whether a model reaches its answer in a scientifically valid and operationally useful way.

Extracting, reconciling, and auditing scientific evidence from papers, figures, tables, and experimental records.

Eval Example

We’re preparing for a Type B FDA meeting on AAV9-microDys-X, an AAV9-based micro-dystrophin gene therapy for Duchenne muscular dystrophy that expresses a 138 kDa construct from an MCK promoter, and we want a hard-nosed critique of whether our current package really supports accelerated approval on micro-dystrophin expression as a surrogate endpoint reasonably likely to predict clinical benefit.

Study context: open-label Phase 1b/2 in 12 ambulatory boys age 4–7 with confirmed DMD and out-of-frame rod-domain deletions. The package is:

Pre-treatment vastus lateralis biopsies: 0–3% of healthy-control dystrophin by quantitative Western blot using MANEX1A against the N-terminal actin-binding domain.
12-week post-treatment contralateral vastus lateralis biopsies: mean micro-dystrophin 38% of healthy control (range 18–61%) by the same Western blot, normalized to total protein by Coomassie staining.
Post-treatment immunofluorescence: sarcolemmal signal in 75–95% of fibers using a polyclonal anti-dystrophin C-terminal antibody.
48-week function: mean NSAA change +1.4 points from baseline versus −0.6 in an external published natural-history registry cohort (p = 0.03 by unpaired t-test).
Safety: transient transaminitis in 8/12 patients managed with steroid taper; one resolved myocarditis; no deaths.
Biodistribution/persistence: AAV9 vector genomes detectable in muscle at 12 weeks at a mean of 2.3 vector genomes per nucleus.
Eligibility: no baseline anti-AAV9 neutralizing antibodies (titer <1:400) and no exon-44 deletions.

Please pressure-test this package item by item: where would FDA or a skeptical reviewer say the evidence, as presented, fails to support our conclusion, and what additional data, analyses, or design changes would be needed to close those gaps if possible? For anything not specified, assume what is known from other clinical-grade 138 kDa micro-dystrophin programs and relevant clinical trials through December 31, 2025 also applies.

Candidate Response

Bottom line: As presented, this package is not strong enough to support accelerated approval on micro-dystrophin expression as a surrogate endpoint reasonably likely to predict clinical benefit. A skeptical FDA reviewer would likely view the biomarker, functional, durability, safety, and generalizability evidence as materially under-supported.

Package Item	Key Failure Mode	What Would Be Needed
Western blot quantification	MANEX1A binds an N-terminal epitope shared by endogenous full-length dystrophin and the micro-dystrophin transgene, so the assay does not cleanly distinguish transgene from residual/revertant dystrophin. Quantifying a 138 kDa micro-dystrophin against a healthy full-length dystrophin standard is also invalid.	Use a recombinant micro-dystrophin standard and an orthogonal method that distinguishes transgene from endogenous dystrophin, such as targeted mass spectrometry or a transgene-specific/epitope-specific assay.
Immunofluorescence	The C-terminal polyclonal antibody is poorly suited because the 138 kDa construct lacks the C-terminal domain. Many DMD patients have revertant fibers, and revertant dystrophin can retain C-terminal epitopes. Revertant fibers may expand clonally with age, biasing IF signal, especially in older boys.	Repeat IF with an antibody against an epitope present in the transgene but absent from revertant dystrophin. Quantify transgene-positive fibers separately from revertant fibers.
Surrogate endpoint validity	The package conflates protein amount with clinical function. “38% of healthy-control protein mass” does not mean 38% of normal dystrophin function because micro-dystrophin is structurally truncated.	Empirically validate the relationship between micro-dystrophin mass-percent, sarcolemmal localization, downstream functional restoration, and clinical benefit before treating expression as a surrogate endpoint.
Biopsy design	Pre- and post-treatment contralateral vastus lateralis biopsies introduce left-right and intramuscular spatial variability. Disease progression and fibro-fatty replacement can also change total-protein-normalized signal.	Standardize biopsy site using consistent anatomical landmarks, normalize to muscle-specific proteins, and measure fibro-fatty composition in parallel.
NSAA comparator/statistics	An external natural-history cohort is not a randomized concurrent control. Trial eligibility, supportive care, participation effects, baseline NSAA, steroid regimen, age, and exon class can all bias the comparison. An unpaired t-test is not sufficient. Also, a +1.4 NSAA change is within test-retest variability for this age group.	Run a randomized concurrent placebo-controlled study, or at minimum use adjusted analyses accounting for baseline NSAA, age, steroid regimen, exon class, and other confounders.
Age-window confounding	Boys age 4–7 are in a developmental window where untreated ambulatory DMD patients may gain motor function before decline dominates. A 48-week NSAA change mixes developmental gain, disease progression, and possible treatment effect.	Use a concurrent randomized control with age stratification to separate developmental trajectory from treatment effect.
Prior clinical precedent	Open-label micro-dystrophin functional signals have not reliably predicted confirmatory benefit; published precedent includes micro-dystrophin gene therapy confirmatory trials failing to reproduce open-label NSAA improvements.	Do not rely on open-label NSAA change as decisive support. Require controlled functional evidence.
Structural limits of the construct	The 138 kDa construct deletes spectrin repeats R16/17, which contain nNOS-binding sites. Loss of nNOS recruitment can impair functional sympatholysis and ischemia protection during exercise, creating a mechanistic ceiling on rescue independent of expression level.	Add mechanistic studies showing whether this specific construct restores relevant dystrophin-associated complex function, nNOS localization, exercise physiology, and muscle protection.
AAV durability	Vector genomes at 12 weeks do not establish durable expression. AAV9 genomes are largely non-integrating episomes and may decline over time. Vector-genome persistence is not the same as persistent protein expression.	Measure longitudinal transgene protein expression and functional biomarker durability beyond 12 weeks.
Immune/safety profile	Transaminitis in 8/12 patients is consistent with immune response to AAV-transduced cells, but the mechanism is not established. One myocarditis case is concerning given AAV9 cardiac tropism.	Provide deeper immune monitoring, liver/cardiac safety characterization, and intensified cardiac follow-up.
Patient selection/generalizability	Excluding anti-AAV9 neutralizing-antibody-positive patients limits generalizability. Excluding exon-44 deletions limits applicability to that DMD subgroup. n=12 is too small to characterize safety and efficacy across the broader DMD population.	Broaden eligibility where possible or pre-specify stratified analyses by antibody status, genotype/exon class, age, and baseline function before using the result to support broad approval.

Regulatory conclusion: The package may show biological activity, but it does not yet establish that the measured micro-dystrophin expression is a reliable surrogate reasonably likely to predict clinical benefit. The main gaps are assay specificity, invalid quantification standards, possible revertant-fiber confounding, lack of a randomized control, age-related NSAA confounding, uncertain durability, and unresolved safety/generalizability issues.

To close the gap, the program would need a controlled, age-stratified clinical design with transgene-specific expression assays, orthogonal protein quantification, tissue-composition controls, longitudinal durability data, mechanistic functional assays for the truncated construct, and stronger safety monitoring, especially hepatic and cardiac.

Rubric Criteria & Grades

Criterion

Points

Identifies assay/measurement problems in micro-dystrophin quantification, including MANEX1A epitope sharing, invalid full-length dystrophin standards, and need for recombinant or orthogonal transgene-specific measurement.

+24

Explains why micro-dystrophin expression level is not automatically a valid surrogate for functional clinical benefit.

+22

Flags biopsy-site, tissue-composition, and age-window confounding that weaken expression and NSAA interpretation.

+19

Critiques the NSAA comparator/statistics, especially reliance on external natural-history controls.

+12

Addresses AAV durability, immune response, transaminitis, myocarditis, and need for longer-term expression/safety follow-up.

+15

Notes patient-selection/generalizability gaps, including anti-AAV9 exclusion, exon-44 exclusion, and small sample size.

Validating LifeSciBench

We validated LifeSciBench through an independent expert review. Feedback came from 453 reviewers who were not involved in writing the tasks. Of those reviewers, 97% held a Ph.D. or equivalent doctorate, with an average of 12 years of field experience and 14 peer-reviewed publications; 88% reported receiving at least one award or fellowship.

Reviewers scored whether each task reflected the qualities needed for a strong benchmark question: alignment with real-world research work, appropriate testing of scientific reasoning and domain expertise, grounding in evidence or expert consensus, and overall usefulness for assessing model performance. Agreement exceeded 96% in every category.

Real-world relevance

Does this task reflect realistic real-world life science work?

Strong agree: 90.4%
Overall agree: 98.3%

Scientific reasoning / domain skill

Does this task test and grade the right scientific reasoning and life science domain skills?

Strong agree: 86.4%
Overall agree: 98.1%

Scientific grounding

Is this task scientifically grounded, answerable, and anchored in appropriate evidence, data, artifacts, or expert consensus?

Strong agree: 77.1%
Overall agree: 96.5%

Overall usefulness

Overall, is this a strong life science evaluation task?

Strong agree: 79.1%
Overall agree: 96.6%

Reviewer comments reinforced the quantitative ratings:

1 of 3

“Overall it is a strong task because it has one correct core interpretation while still leaving room to separate better answers by how carefully they bound the uncertainty.”

Results

We report two complementary metrics. Pass rate is the percentage of tasks on which a model meets the task-level success threshold of 70%. Score is the average rubric reward, giving partial credit for individual criteria even when the full task is not solved. Both matter because a response to a scientific task can be partially correct or useful without meeting every requirement for a complete answer.

Model performance varies substantially by task type, workflow, and response format.

Where AI systems show early strength

LifeSciBench shows that frontier models are relatively strongest on tasks involving scientific synthesis, communication, and structured interpretation. Absolute pass rates are still modest, so these benchmark domains are far from saturated, but GPT‑Rosalind shows meaningful progress over GPT‑5.5, improving overall exact pass rate from 25.7% to 36.1%.

The strongest directions of progression in model capabilities appear in Scientific Communication and Translation. For example, the Scientific Communication pass rate increases from 56.3% for GPT‑5.5 to 71.1% for GPT‑Rosalind; this category is small (n=9), so it should be interpreted cautiously, but it suggests frontier models are improving rapidly in their ability to organize evidence and produce convincing expert-facing explanations. Translation (the "bench-to-bedside" process of drug development) shows a similar pattern, rising from 36.8% for GPT‑5.5 to 57.7% for GPT‑Rosalind, suggesting models are quickly improving on their ability to connect preclinical evidence to clinical implications.

Rubric-level results point in the same direction. On tasks requiring expert-useful or actionable outputs, GPT‑Rosalind scores 44.7%, compared with 29.1% for GPT‑5.5. On tasks requiring uncertainty and caveat handling, it scores 44.8%, compared with 29.3%. This pattern suggests models are most useful when the task has a clear evidence boundary and calls for structured scientific judgment.

GPT‑Rosalind leads performance across scientifically-valuable tasks identified by industry and academic experts.

GPT‑Rosalind improves performance over GPT‑5.5 across core life-science workflows, with the strongest gains in translation and scientific communication.

Where AI systems still fall short

Performance remains much weaker on artifact-heavy, design-heavy, and operationally constrained scientific work. Namely, Design, Optimization, & Prediction remains one of the hardest workflows, with GPT‑Rosalind passrate at 30.7%; Analysis is similarly difficult at 30.3%.

Artifact use is a particularly clear gap. While GPT‑Rosalind performs better than GPT‑5.5 in artifact-heavy settings, its pass rate still drops from 45.1% on text-only tasks to 28.1% on tasks with artifacts or URLs. GPT‑5.5 shows the same pattern, dropping from 29.9% to 21.9%. A more detailed analysis confirms that frontier models struggle at extracting information from complex figures or large sequence files and integrating that information into the final answer.

Pass rates drop when tasks require source-grounded reasoning or working with artifacts

The answer format also matters. Tasks requiring exact sequence, structure, or construct-level outputs show lower pass rates: GPT‑Rosalind reaches only 14.8% on numeric tasks and 24.0% on sequence or structure outputs. Construct-generation tasks are also brittle, with GPT‑Rosalind at 27.3% and showing little improvement over GPT‑5.5. Some of this gap may reflect a stricter grading surface for exact-answer tasks, where small differences in calculation or formatting can cause a response to fall under pass threshold. Still, these failures are scientifically meaningful because many life science workflows require outputs that are exact enough to be used directly, such as in CRISPR/HDR donor design or siRNA design.

Models also often get part of the way there without fully solving the task. In roughly 14% of tasks, models earned substantial rubric credit despite failing the exact-pass threshold. For GPT‑Rosalind, 109 tasks had pass rates below 20% while still earning at least 50% rubric reward. In practice, this means models may identify relevant evidence or produce a plausible partial answer, but still fail because they miss a key constraint, use the wrong evidence, make an incomplete calculation, or do not connect their reasoning to a scientifically useful final decision.

Limitations & what’s next

LifeSciBench is a step toward measuring how useful AI systems can be for life science research, but it is not a substitute for studying models in live research environments. The benchmark focuses on self-contained tasks that reflect recurring industry workflows, while leaving many scientific specialties and task types outside its current scope. Real research is iterative: scientists gather new evidence, revise hypotheses, design follow-up experiments, and adapt their plans as results emerge.

Strong performance on LifeSciBench should therefore be interpreted as evidence of realistic task-level capability, not as a direct measure of downstream research impact. The benchmark is grounded in industry workflows, but it does not capture the full diversity or dynamics of live research programs, where progress depends on factors that unfold over time.

The next step is to connect benchmark performance to deployment studies in live research workflows. While LifeSciBench was developed with practicing scientists, measuring whether AI systems accelerate discovery or improve R&D outcomes will require studying model use and performance in real research settings, over longer horizons, and across multiple rounds of reasoning, feedback, and experimental follow-up.

Get involved

Help shape the next generation of life science AI benchmarks, or request access to GPT-Rosalind.

Join as a contributor Request access

Predicting model behavior before release by simulating deployment

Tue, 16 Jun 2026 00:00:00 GMT

Introduction

Before releasing a new model, labs need to understand not just what it can do, but how it is likely to behave in real-world use, including where it might introduce new risks. This becomes even more important as capabilities increase. As part of our pre-deployment safety review, we leverage targeted evaluations, red-teaming, and other checks to understand model behavior. We’ve now started using a method for simulating model deployments before they happen, which adds a complementary signal: a deployment-like preview of how a candidate model may behave before it reaches users.

Deployment Simulation is a method for simulating a future deployment before it happens. We do so by replaying previous conversations in a privacy-preserving manner with a new candidate model. This enables us to study how the new model responds in realistic contexts before release, including whether new undesired behaviors emerge and how often they may appear.

Across multiple GPT‑5‑series Thinking deployments, Deployment Simulation improved our estimates of undesired model behavior rates, helped surface novel forms of misalignment before release, and helped reduce the risk that models would be able to tell they were being tested. We also applied the method to challenging agentic rollouts, showing that it can extend beyond standard chat to more complex agent settings involving tool use, and can also be used for risk assessment before internal model deployments.

We have already used insights from Deployment Simulation during model development to identify blind spots in traditional evaluations and inform mitigations and deployment decisions. As we make the pipeline easier to run, we expect it to play a larger role in the future model development process.

How Deployment Simulation works

Pre-deployment evaluations used across the industry generally consist of a mix of synthetic, manually written, or production prompts intentionally selected to be difficult, high severity, or adversarial. These evaluations have generally had two intertwined goals: assessing how the model responds when stress-tested in situations that have a very small chance of occurring in deployment traffic, and gaining a general understanding of undesired model behaviors, including finding novel undesired behaviors and predicting their deployment-time frequencies.

While traditional evaluations remain crucial for the first goal, Deployment Simulation represents a significant step forward with regards to the second goal for non-tail risks (in our experiments, our approach can’t be expected to measure behaviors that occur with frequency less than 1 in 200,000 messages). Deployment Simulation’s strength comes from side-stepping some core challenges with traditional evaluations:

Coverage: it is difficult to be confident that evaluation prompts will cover all possible types of undesired model behavior. Measuring new types of undesired behavior requires creating new evaluations, and knowing what harms to measure. This can be time and labor intensive, and evaluations can saturate over time.
Selection biases: traditional evaluations generally are built with specific undesired model behaviors in mind, selected in ways that bias towards contexts have previously occurred. This may make their estimates less informative about incidence of such behaviors in the full breadth of the deployment distribution.
Being recognizable as tests by models: models have been increasingly able to determine they are being tested, which can distort their behavior and our downstream measurements of their safety. While many of the traditional evaluations we use at OpenAI are also drawn from production traffic and don’t suffer from this issue, this is a common concern for many traditional evaluations used across the industry.

The main technique behind Deployment Simulation is simple: we take recent conversations from deployment, remove the original assistant response from the older model, and regenerate it with a candidate model to be released. We evaluate the completions by searching for new failure modes, and estimate deployment-time undesired behavior frequency based on frequency in this deployment-like distribution. We can perform the same measurements after release on real deployment traffic, enabling us to validate our pre-deployment forecasts.

Deployment Simulation significantly mitigates the three limitations listed above: sampling bias is addressed by using a distribution of evaluation prompts representative of recent usage. Deployment Simulation with representative prompts can also mitigate concerns about coverage: we get greater coverage of undesirable model behaviors by simply simulating more traffic. This tradeoff between compute and coverage is important, because it makes the quality of this axis of risk assessment scale with compute, rather than the manual effort required to build more evaluations. Models also don’t seem to be able to distinguish between conversations obtained via deployment simulation and real deployment traffic.

Figure 1: Overview of our Deployment Simulation pipeline

Area	Traditional Evaluations	Deployment Simulation
Coverage	Better for low prevalence but high severity risks that we want to make sure we have coverage for	Better for understanding the spectrum of risks we’re likely to see in deployment traffic
Representativeness / bias	Focused on evaluating targeted past or predicted risks, even for behaviors that have extremely low prevalence in deployment	Representative of risks that occur above a certain prevalence in current deployment traffic
Evaluation awareness	Potentially obvious test cues (e.g., with synthetic data) unless evals are created with deployment data	Deployment-like context seems to significantly mitigate eval awareness
Effort	Custom prompt distributions and setups requires significantly more manual effort	Requires a one-time infrastructure cost, and then reuses prefixes from deployment and graders from known undesired behaviors

High level comparison between traditional evaluations and Deployment Simulation

How we tested Deployment Simulation

To evaluate Deployment Simulation, we pre-registered predictions for the deployment-time frequency of 20 types of undesirable behavior for GPT‑5.4 Thinking. We also ran retrospective studies across other deployments of Thinking models of the GPT‑5‑series. The model behaviors we analyze cover both misalignment and disallowed content categories which we have been reporting in system cards (e.g. the model lying about tools, or outputting disallowed sexual content). While for these results we only track 20 categories of undesired model behavior, and only look for novel misaligned behaviors, Deployment Simulation is a general technique which could be applied to simulate any deployment-time model property.

The production traffic we leveraged was also from Thinking models of the GPT‑5‑series. Before running our evaluations, we automatically removed account-linked identifiers and identifiable information. In accordance with our privacy policy, we only analyzed ChatGPT traffic from users who allow their data to be used for model improvements. We report only aggregate findings from these evaluations. In total, we analyzed approximately 1.3 million de-identified conversations across GPT‑5 Thinking through GPT‑5.4 deployments, spanning August 2025 to March 2026.

We evaluate the quality of the pre-deployment risk-assessment produced by Deployment Simulation in three main ways:

Taxonomy coverage: did post-release auditing surface any important misaligned behaviors missing from our initial bad behavior taxonomy?
Directional accuracy: did the simulation correctly predict whether deployment prevalence would increase or decrease for known behaviors?
Rate calibration: were our pre-deployment estimates close to the rates we later observed in deployment? Achieving good rate calibration requires more than directional accuracy, making this the highest possible quality standard.

We also measure how much models are able to tell they are in an evaluation during Deployment Simulation, and test how robust the method is to different forms of deployment traffic and to challenging tool use settings.

Deployment Simulation significantly expands pre-deployment risk assessment

Across GPT‑5‑series Thinking deployments, Deployment Simulation outperformed challenging-prompt baselines in determining actual risk rate, was able to surface calculator hacking before release, and appeared potentially indistinguishable from deployment to the model.

Overview of our predictions. For a visualization of all predictions for Thinking model deployments of the GPT‑5‑series, see Figure 2. The predictions for GPT‑5.4 Thinking were pre-registered, while the others were retroactive predictions. In aggregate, our predictions had a median multiplicative error of 1.5x (e.g. if the undesirable behavior rate is 10 in 100k, this would correspond to estimating a rate of 15 or 6.67 in 100k). While tail errors of our method can be roughly 10x larger, we expect they can be significantly reduced with further improvements to our simulation pipeline, as discussed below.

Figure 2. Deployment Simulation frequency predictions of deployment-time undesirable behavior based on prior deployment data. The notation 5.1 → 5 indicates that we are resampling prefixes from GPT‑5 with GPT‑5.1, that is, simulating GPT‑5.1’s deployment based on data from GPT‑5. The highest rate of misaligned behavior was “calculator hacking” in GPT‑5.1, which we have discussed previously⁠(opens in a new window).

Comparing Deployment Simulation predictions to baselines. The most important categories to estimate correctly for pre-deployment risk-assessments are ones that have large changes of incidence after the model’s deployment (e.g. >= 1.5x). On this subset, Deployment Simulation is substantially better than baselines both at predicting whether a certain undesirable behavior will increase or decrease in prevalence with a model’s deployment (Figure 3, left), and at estimating its exact deployment-time incidence (Figure 3, right). We use two main baselines: a Challenging Prompts baseline which we currently use for system cards and launch decisions, and a naive baseline of using rates from the previous deployment as estimates for the new model.

Figure 3. Deployment Simulation predicts both the direction of incidence changes (left) and exact production rates (right) better than static evals.

Sources of simulation error in our pipeline. We also studied the current sources of error in our pipeline. Specifically, large tail errors make the pipeline less trustworthy for deployment decisions: for example, when seeing a large predicted increase in an undesirable behavior of a certain type, if the pipeline is known to have even occasional large errors, it raises doubt about whether this is a reliable estimate or an artifact of the pipeline. There are two main sources of possible error:

Resampling environment fidelity error: to estimate production rates accurately, our simulation pipeline needs to be as realistic as possible. This is in part an engineering problem (sampling from the model using the same pipelines used for production traffic), and in part an interesting challenge for tools which need read/write access to external resources which are constantly changing (e.g. web searches or interacting with local files). We can isolate the simulation fidelity error by comparing predictions from an actual deployment and its simulation (that is, simulating the deployment of the already deployed model).
Prompt distribution shift error: a major concern with Deployment Simulation is that any model’s deployment will cause a distributional shift in inputs, especially if users adapt to new model affordances. We isolate prompt distribution error by comparing estimates when simulating the deployment using recent prompts vs prompts from the realized deployment.

As seen in Figure 4, we find that a large fraction of the error in our pipeline–especially for tail errors–is currently due to simulation fidelity. This is good news, as we expect simulation fidelity to be generally easier to improve than reducing prompt distribution shift. We also find that prompt distribution shift error may be reduced by using the most recent data from the previous deployment. More detail can be found in the research paper.

Figure 4: Resampling environment fidelity seems to be the largest source of error in our estimates, especially for the most extreme errors.

Automated auditing found the one new misalignment introduced in these deployments. In the deployment window we studied, our automated auditing pipeline based on simulated deployments would have surfaced ‘calculator hacking’ before release, which was the only novel misalignment we identified in the analyzed ChatGPT traffic. Calculator hacking is a form of reward hacking which involves the model using a browser tool as a calculator while presenting the action as a search. This does not mean large-scale auditing will catch every new failure, but it shows why realistic contexts matter: they can elicit behaviors that narrower eval sets may never surface directly, even if they were not present in the original traffic used to seed the simulation.

Reducing evaluation awareness

Another advantage of using deployment-like contexts is that they may reduce distortions that arise when models recognize they are being tested. In prior research, we as well as others⁠(opens in a new window) have found that models are often aware of when they’re being tested. For example, models may reason in their chain-of-thought about what they’re being tested on, what an evaluator might like, and – if they’re misbehaving – whether they’ll be caught. This poses a problem for alignment evaluations: if a model is only well behaved because it thinks it’s being tested, how do we build confidence that it’ll behave safely once it learns it’s been deployed?

We show that Deployment Simulation substantially reduces levels of evaluation awareness exhibited by models relative to traditional evaluations which are not based on real deployment data. We measure evaluation awareness in three ways: by asking a model to classify whether a trajectory came from production or an evaluation, monitoring chain-of-thought for explicit statements about being evaluated for alignment, as well as for metagaming⁠(opens in a new window): any reasoning about oversight. Simulated deployment traffic closely matched real deployment traffic across all three measures. For example, GPT‑5.2 labeled real production traffic as evaluation-like 5.4% of the time and simulated traffic 5.1% of the time, while traditional evaluations (such as SWE-Bench or MonorepoBench, which are composed primarily of synthetic examples) were classified as evaluations almost 100% of the time.

Figure 5. Evaluation-awareness comparison: real production traffic and simulated traffic show similar rates for eval detection, verbalized alignment evaluation awareness, and metagaming, while traditional eval datasets report much higher rates.

Tool simulation for agentic trajectories

To test whether the method could extend beyond standard chat modes, we also studied simulating deployments for challenging agentic coding settings. Specifically, we use 120,000 internal employee agentic trajectories from GPT‑5.4 to simulate the internal deployment of coding agents based on GPT‑5.5. This serves two purposes: this shows how deployment simulation can also be used for risk-assessment for internal deployments, and it demonstrates that it is possible to make simulation fidelity high even in agentic settings via careful tool simulation.

Above we discussed that the simulation fidelity is one of the main sources of error in our current pipeline. One of the challenges with making deployment simulations realistic is that it requires tools to act in realistic ways, even when they need read/write access to external resources which are constantly changing (e.g. web searches or interacting with local files).

This is especially challenging for settings such as agentic deployments: a single agent trajectory can depend on hundreds of tool calls, repository state, network responses, or transient failures. This makes resampling with real tool access both challenging and potentially dangerous. In light of this, instead of applying tool calls to live systems, we simulate them with another LLM. We show that giving this LLM extra affordances is essential to realism and therefore estimation quality.

We give the tool-simulator model the original final-turn rollout, the repository from the time of the conversation, a database of tool-call/response pairs, and read-only connectors. In a test where a discriminator tried to tell simulated from real rollouts, realism improved from an 11.6% win rate to 49.5%, near the 50% chance level.

Figure 6: One can make resampling realism almost indistinguishable from real sampling even in tool-heavy settings when simulating tool calls with increased affordances.

As additional evidence of simulation fidelity, when simulating GPT‑5.4 Codex thinking traffic with GPT‑5.4, the misalignment rates are, just about, within error bars. By contrast, in our pre-deployment testing GPT‑5.5 was more misaligned in most categories.

Figure 7: With tool simulation, behavior rates for conversations resampled with GPT‑5.4 and actual GPT‑5.4 production traffic are quite similar – indicating high resampling environment fidelity. See the GPT‑5.5 system card⁠(opens in a new window) for more context on the categories.

These results suggest that Deployment Simulation can extend to complex agent settings when the surrounding tool environment is simulated with sufficient fidelity.

WildChat and external auditing

A natural question is whether external auditors can get some of the same benefits without access to private production traffic. We tested this with 100,000 WildChat conversations used as prefixes. WildChat was less accurate than recent OpenAI production data, but still informative: recent production data had lower average multiplicative error than WildChat (1.75× vs. 2.44×), while WildChat often stayed within roughly 3× of production rates and identified the better model at above-chance rates. We see WildChat-style evaluations as a promising complement to developer-run Deployment Simulation. See this companion Alignment blogpost⁠(opens in a new window) for more results on this point.

Limitations

Deployment Simulation is a complement to, not a replacement for, adversarial evaluations, red-teaming, and targeted tail-risk analysis. The detection of some undesired model behaviors relies on chain-of-thought monitorability: if future models reason less legibly, omit key steps, or learn to obscure intent, we may not be able to detect these forms of undesirable behavior. Furthermore, because production data is private, developers currently have stronger access to representative contexts than external auditors. Better public datasets could narrow that gap.

Deployment Simulation is most useful for behaviors that occur often enough to appear in sampled prefixes. A failure that occurs once in ten million conversations is unlikely to be discovered in a million-sample run, so adversarial evaluations, red-teaming, and targeted tail-risk analysis remain necessary. We currently focus on simulating a single assistant response, as in preliminary experiments we did not find meaningful improvements to our estimates by simulating multiple turns. This may be an important area for improving simulation fidelity in the future.

The method also depends on the prefix distribution—the mix of conversations that are being resampled. Historical traffic may not match how users interact with a more capable model after a major product change, new tool release, world event, or seasonal shift. Analysis in the full paper suggests that this issue can be mitigated by using the most recent data available.

Conclusion

Deployment simulation is a new approach to pre-deployment risk assessment that helps frontier labs and evaluators predict how language models may behave in the real world and understand the risks they pose before deployment. It complements existing safety evaluations, red-teaming, and targeted analysis by adding a more production-like prediction layer that can improve estimates of deployment behavior, reduce evaluation-awareness effects, and make pre-deployment predictions checkable after release. Used alongside traditional evaluations, Deployment Simulation can help make model risk assessment more realistic, more quantitative, and more useful for deployment decisions.

Dreaming: Better memory for a more helpful ChatGPT

Thu, 04 Jun 2026 09:00:00 GMT

Loading…

Today we’re beginning to roll out a more capable and scalable system for synthesizing memory, developed to tackle the staleness, correctness, and scalability challenges that we observe when memory is applied to the hundreds of millions of users and multi-year time horizons in ChatGPT.

Memory is what helps ChatGPT learn your preferences, projects, and constraints, allowing future conversations to start from shared context rather than from scratch.

Over the last two years, memory has grown into a critical part of the ChatGPT experience, helping ChatGPT better understand your context so it can help you accomplish meaningful goals over time. This is central to making ChatGPT more useful: knowing you, helping you, and doing more for you.

This update is available to Plus and Pro users in the US today, and will roll out to additional countries and Free and Go users over the coming weeks.

How memory has evolved

Memory first launched in April 2024 (also known as saved memories). The feature let you ask ChatGPT to remember information and carry it forward into future chats.

Saved memories were only written during the conversation and relied on strong cues to decide when to trigger memory, such as an instruction to "remember I’m traveling to Singapore in July." In practice, interacting with this system could feel like talking to someone who took a few notes, but still forgot everything that wasn’t written down. Saved memories also tend to go stale over time and eventually become incorrect or irrelevant.

In April 2025, we updated ChatGPT’s memory by giving the model the ability to reference chat context outside of the saved memories list; this was done by introducing the first version of dreaming—a method for ChatGPT to automatically curate memories in the background by referencing chat history.

In contrast to saved memories, dreaming leverages a background process that allows ChatGPT to learn from many conversations and synthesize ChatGPT’s memory state in order to always provide the freshest, most relevant context to your conversations. Dreaming also makes it easier for memory to include context that occurs naturally in conversation, without relying on explicit requests to remember something.

Over the last year, dreaming supplemented saved memories to create a step-function improvement in ChatGPT's ability to personalize responses and offset the staleness of saved memories. However, it historically was never sufficient as a standalone memory system.

Today, we are launching a significantly more capable and compute-efficient memory architecture built on top of dreaming.

The memories synthesized by dreaming are reviewable through a summary of them made visible in the memory summary page. From the memory summary, you can quickly glean the highlights of what ChatGPT knows about you, add or update information about yourself, and provide instructions on what topics ChatGPT should bring up and when. If you want to drill down into a particular area to learn more, just chat with the model.

How we evaluate memory

When we think about what "good memory" looks like in ChatGPT, a few things come to mind:

Carry forward useful context: You tell ChatGPT something once, and it remembers that information in your subsequent chats.
Follow preferences and constraints: If you describe a preference (e.g., you’re vegetarian), then ChatGPT should take actions that are consistent with that preference going forward.
Stay current over time: Memory should account for the passage of time. Imagine "The user is planning their birthday party for next Saturday"; eventually, Sunday arrives.

We can evaluate how ChatGPT Plus and Pro memory has improved over time with respect to each of the three memory objectives above. We do this for each of:

2024: Saved memories
2025: Saved memories + Dreaming V0
2026: Dreaming V3

Carrying forward context

When you start a new chat with ChatGPT, you don’t have to introduce yourself from scratch. ChatGPT can save you time and build on prior context, especially for complex, long-running projects.

For example, imagine you’re using ChatGPT to shop for new camera gear that’s compatible with your current camera. If you’ve discussed your camera setup with ChatGPT in the past, you can ask for products that are compatible with "my photography setup" and get tailored recommendations that meet your needs.

Without memory

What would I need to buy to use TTL for my underwater photography setup?

To use TTL flash underwater, you need a compatible chain from camera → housing/trigger → cable → strobe. What you buy depends mostly on whether your setup uses fiber-optic TTL or electrical TTL.

Basic shopping checklist

TTL-capable underwater strobe
Look for strobes that support your TTL type, such as Inon S-TTL, Sea & Sea DS-TTL, Ikelite DS TTL, or compatible Backscatter/Nauticam systems. TTL has to be supported by the strobe, not just the camera.
TTL trigger or TTL converter
This is the key piece. It translates your camera’s flash metering into something the underwater strobe can understand. Examples include Nauticam optical TTL converters, Sea & Sea TTL converters, Backscatter Smart Control optical TTL triggers, or Ikelite DS Link TTL converters, depending on housing and camera brand. Backscatter notes that Nauticam optical TTL converters can work with fiber-optic-connected Inon and Sea & Sea TTL strobes, while Ikelite’s DS Link converters are made around Ikelite DS strobes and electrical sync setups. (Backscatter.com⁠(opens in a new window))
Connection cable
For fiber optic TTL, buy one fiber optic cable per strobe. Ikelite’s fiber-optic guide says each strobe needs its own light path, converter/adapter, and cord. (Ikelite⁠(opens in a new window))
For electrical TTL, buy the correct sync cord, such as Ikelite-style, Nikonos 5-pin, or housing-specific bulkhead cable.
Housing compatibility parts
You may need a hot-shoe adapter, bulkhead, fiber optic window/adapter, or housing-specific TTL board. Many TTL triggers mount inside the housing on the camera hot shoe.
Strobe arms, clamps, tray, and mounting hardware
TTL controls exposure, but you still need to physically mount and aim the strobes.
Batteries and spares
Bring strobe batteries, trigger batteries if required, spare fiber optic cables or sync cord O-rings, and silicone grease for serviceable O-rings.

Which route should you choose?

Fiber optic TTL is usually easiest if your housing has optical ports or can use the camera’s pop-up flash/LED trigger. It is common with compact, mirrorless, Nauticam, Sea & Sea, AOI, and Olympus/OM TG-style setups.

Electrical TTL is often best if you use Ikelite DS strobes or an Ikelite-style housing/bulkhead. Ikelite says its DS Link TTL converters are faster and more reliable than fiber-optic TTL triggers, and they do not require batteries. (Backscatter.com⁠(opens in a new window))

The compatibility rule

You need all three to match:

camera brand/protocol + housing/trigger type + strobe TTL system

For example:

Sony camera + Nauticam housing + optical TTL trigger + Inon/Sea & Sea-compatible fiber optic strobe
Nikon/Canon/Sony camera + Ikelite housing/bulkhead + Ikelite DS Link TTL converter + Ikelite DS strobe
Compact camera with built-in flash + fiber optic cable + S-TTL/DS-TTL-capable strobe

So the practical answer is: buy a TTL-compatible strobe, the correct TTL trigger/converter for your camera and housing, and the right fiber optic or electrical sync cable. The exact part numbers depend on your camera, housing, and strobe model.

The model produces a generic response that leaves the user to do complicated compatibility checks on their own.

With memory

What would I need to buy to use TTL for my underwater photography setup?

I’ll verify this against your actual setup: Sony A1 II in Nauticam NA-A1II, with Backscatter Mini Flash 3 and Inon Z-330 strobes. The key question is whether you want TTL with the Mini Flash 3, the Z-330s, or both.

For your setup — Sony A1 II + Nauticam NA-A1II + Backscatter Mini Flash 3 + Inon Z-330 — there are two different TTL paths:

Best buy for Mini Flash 3 TTL

Buy:

Backscatter Smart Control TTL LED Nauticam Flash Trigger for Sony
SKU: BS-TR-SN2

Backscatter lists it as custom-designed for Nauticam full-frame Sony housings, TTL-compatible with Mini Flash 3, and the NA-A1II housing page offers it as the “Backscatter Smart Control TTL Flash Trigger” option. (Backscatter.com⁠(opens in a new window))

You’d use normal optical fiber cables from the housing to the Mini Flash 3. On the MF-3, you’d use SC or SC Macro mode; Backscatter says SC / SC Macro are the TTL modes for Mini Flash 3, with SC Macro specifically tuned for close macro work. (Backscatter.com⁠(opens in a new window))

For Inon Z-330 TTL

For the Inon Z-330, the relevant unit is:

UW Technics 11075-HSS TTL Converter for Sony / Nauticam

UW Technics explicitly lists Inon Z-330 as a compatible TTL strobe and supports optical TTL output. (UWTechnics⁠(opens in a new window))

But I would not buy this for Mini Flash 3 TTL. Backscatter’s own UW Technics listing says Backscatter strobes are manual / HSS only on that converter, while the Backscatter Smart Control trigger is the one listed for Sony TTL with the Mini Flash 3. (Backscatter.com⁠(opens in a new window))

Practical recommendation

Since you shoot macro and have the Mini Flash 3, I’d buy the Backscatter Smart Control TTL LED Nauticam Flash Trigger for Sony — BS-TR-SN2.

The catch: there probably is no single trigger that gives true TTL for both the Mini Flash 3 and Inon Z-330 at the same time. The Backscatter trigger is the right TTL solution for the MF-3; the UW Technics converter is the right TTL solution for the Z-330. So choose based on which strobe you actually want TTL for.

For your use case, I’d prioritize Mini Flash 3 TTL / SC Macro and treat the Z-330s as manual when needed.

The model remembers the user’s camera setup and recommends a compatible product.

We can construct an eval from examples that resemble this where the model is asked to respond to a prompt that requires it to recall factual information about the user. The model is then rewarded if it responds in a way that correctly uses the relevant context. In this evaluation, the new dreaming-based system improves the model's ability to recall relevant facts.

Following preferences

Memory also helps ChatGPT respond in ways that better match your preferences and constraints.

Imagine that you’re planning a trip to Singapore. Two months before your trip, you ask ChatGPT to help with an itinerary. ChatGPT already knows from past travel planning that you enjoy wildlife photography, prefer hotels with strong AC, and enjoy a quiet dinner over a crowded bar.

Without memory

Help me plan a trip to Singapore. I’m going to be there for a work trip in July and will have a few days free at the end.

Great add-on city for a work trip: Singapore is compact, safe, excellent for solo exploring, and easy to do in 2–4 free days. July is hot and humid; expect daytime highs around 31–33°C / 88–91°F, warm nights, and sudden rain, so plan outdoor time early morning or after 5 p.m. and keep midday for museums, malls, cafés, or hotel downtime. Singapore’s official weather service notes typical daily highs of 31–33°C and lows of 23–25°C. (Singapore Weather Service⁠(opens in a new window))

A strong 3-day plan

Day 1: Marina Bay, Gardens, and skyline

Start late afternoon at Marina Bay and walk the waterfront: Merlion Park, Helix Bridge, ArtScience Museum exterior, then into Gardens by the Bay. Book Cloud Forest, Flower Dome, and either the OCBC Skyway or Supertree Observatory if views matter to you. Gardens by the Bay lists Cloud Forest and Flower Dome as open daily 9 a.m.–9 p.m.; Supertree Observatory is also listed as 9 a.m.–9 p.m. with last admission at 8:30 p.m., but check maintenance closures before booking. (Gardens by the Bay⁠(opens in a new window))

Evening: stay for the Supertree light show area, then dinner at Lau Pa Sat for an easy first-night hawker experience near the CBD. It has 80+ stalls and local staples like satay, laksa, nasi lemak, bak kut teh, kaya toast, and char kway teow. (Laupasat⁠(opens in a new window))

Day 2: Culture neighborhoods and food

Morning: Chinatown. Visit Buddha Tooth Relic Temple, walk the shophouse streets, and eat at Maxwell Food Centre or Chinatown Complex Food Centre. Visit Singapore lists Maxwell, Chinatown Complex, Old Airport Road, Tiong Bahru Market, Newton, and others among popular hawker centres. (Visit Singapore⁠(opens in a new window))

Afternoon: Kampong Glam and Haji Lane for Sultan Mosque, murals, boutiques, and cafés. Then head to Little India around golden hour for temples, flower garlands, Tekka Centre, and dinner.

Evening: either cocktails around Ann Siang / Club Street, or keep it low-key with a night walk along the Singapore River: Boat Quay to Clarke Quay to Robertson Quay.

Day 3: Nature or beach day

Pick one based on your energy.

For nature: do Singapore Botanic Gardens early, then the National Orchid Garden, followed by lunch in Dempsey Hill or Tiong Bahru. This is the best “green Singapore” day without leaving the city.

For beach/resort mode: go to Sentosa. Choose the southern beaches, Fort Siloso Skywalk, or a beach club. I would skip trying to do every Sentosa attraction unless you love theme parks.

For local food focus: do Tiong Bahru Market for breakfast, National Gallery Singapore midday, then Old Airport Road Food Centre for dinner. The official events page also shows exhibitions and cultural events, so it is worth checking once your exact July dates are fixed. (Visit Singapore⁠(opens in a new window))

Best places to stay after the work part

Marina Bay / CBD is easiest if you want polished, walkable, skyline-heavy Singapore and minimal transit.

Chinatown / Telok Ayer / Tanjong Pagar is my top pick for a few free days: great food, MRT access, heritage streets, cocktail bars, and close to work districts.

Bugis / Kampong Glam is better if you want boutique hotels, cafés, street art, and a slightly more relaxed neighborhood feel.

Orchard is convenient for shopping and rainy-day malls, but less atmospheric.

What to book ahead

Book Gardens by the Bay conservatories/Skyway, any special restaurants, and possibly a cocktail bar. For hawker centres, do not over-plan; just go hungry, carry tissues or wet wipes, and be ready to share tables.

July packing notes

Bring breathable work clothes, comfortable walking shoes, a small umbrella, and a light layer for aggressive air-conditioning. I would also plan one “rain-flexible” indoor block each day, because showers can arrive suddenly even when the forecast looks fine.

My suggested balance: Day 1 Marina Bay/Gardens, Day 2 Chinatown–Kampong Glam–Little India, Day 3 Botanic Gardens or Sentosa. That gives you the Singapore highlights without turning your post-work break into another work project.

The model produces a generic response that is more touristy, doesn’t help with hotel booking, and largely ignores the user's interests.

With memory

Help me plan a trip to Singapore. I’m going to be there for a work trip in July and will have a few days free at the end.

I’ll build this around your known constraints: a few free days after work, hot/humid July weather, wildlife/photography/nature bias, strong AC needs for sleep, and good but not overly social dining. I’ll verify current July logistics and then give a practical short itinerary rather than a generic Singapore list.

Here’s how I’d structure it: 2–3 free days, mostly nature / wildlife / photography, with heat-managed mornings and AC breaks. July is hot and humid: Singapore’s July long-term stats are about 31.4°C / 88.5°F average daily max, 25.4°C / 77.7°F min, 146.6 mm rain, and 14 rain days, so outdoor plans should be early morning or evening, not midday. (National Environment Agency⁠(opens in a new window))

Best plan for 3 free days

Day 1 — Gardens by the Bay + Marina Bay, low-friction

Morning: Cloud Forest + Flower Dome. This is the obvious “touristy” pick, but it fits you well because it’s visually interesting, good for photography, and climate-controlled. Gardens by the Bay lists Cloud Forest and Flower Dome-type attractions running 9:00am–9:00pm, and Cloud Forest currently has the Jurassic World experience listed as ongoing. (Gardens by the Bay⁠(opens in a new window))

Late afternoon / evening: Supertree Grove + Borealis / light show. Gardens currently lists Borealis every Sat, Sun & Mon, 8:00–8:30pm at Supertree Grove, so try to align this with one of those nights. (Gardens by the Bay⁠(opens in a new window))

Dinner: do something easy nearby: Marina Bay / Tanjong Pagar / Telok Ayer. For you, I’d prioritize reservation-based restaurants with real tables over hawker centers on the first night, because you dislike cramped solo seating.

Day 2 — Mandai wildlife day

Morning to afternoon: Bird Paradise + River Wonders. Bird Paradise is probably the best fit photographically; River Wonders is less “wildlife photography” and more controlled exhibits, but it pairs well logistically. River Wonders is listed as daily 9:00am–6:00pm, last entry 5:00pm. (Mandai Wildlife Reserve⁠(opens in a new window))

Evening: Night Safari. This is the rare Singapore attraction that is genuinely better at night and avoids the worst heat. Night Safari is listed as daily 6:00pm–12:00am, last entry 11:15pm, with tram plus walking trails and over 900 nocturnal animals. (Mandai Wildlife Reserve⁠(opens in a new window))

This is a full day. I would not add Singapore Zoo unless you have 4 free days; Mandai can become exhausting fast.

Day 3 — Nature reserve / macro-ish walking + food

Early morning: Singapore Botanic Gardens or MacRitchie / Windsor Nature Park. For a few free days, Botanic Gardens is the safer pick: easier transport, less sweaty, better bailout options, and more consistent casual photography. MacRitchie / Windsor is better if you want more actual forest, but it is hotter, sweatier, and less predictable.

Midday: AC break.

Late afternoon: Joo Chiat / Katong or Chinatown / Telok Ayer for walking, architecture, coffee, and dinner. This gives you a different texture from Marina Bay without turning the day into a logistics project.

My ranking for you

Cloud Forest / Gardens by the Bay — best combination of photogenic, iconic, heat-managed.
Bird Paradise + Night Safari — best wildlife-focused day.
Botanic Gardens early morning — best low-stress outdoor nature.
Joo Chiat / Katong — best neighborhood walk.
Hawker centers — worth doing, but not ideal for your seating preferences unless timed off-peak.

What I’d avoid or deprioritize

I would skip Sentosa unless you specifically want beach/resort stuff. I would also skip a packed “greatest hits” day of Merlion / Orchard / Little India / Chinatown / Clarke Quay; it’s inefficient and hot. For you, the trip will be better if each day has one strong anchor plus one easy evening plan.

Hotel / sleep note

Since this is a work trip, your hotel may be fixed. If you can choose or extend at the same hotel, the hard gate is still AC. Singapore hotels are generally AC-heavy, but your requirement is unusually strict.

One thing I need to make this concrete

Send the exact free dates and whether your work hotel is fixed. Then I’d turn this into a day-by-day plan with restaurant targets and a hotel/AC callout.

The model produces a response that is personalized to the user’s interests in wildlife photography, quiet dinners, and their priorities when booking a hotel.

Preferences can take several forms:

Instructions for how ChatGPT should respond ("don’t bring up Stan again").
Your personal preferences or constraints ("I’m vegetarian")
Implicit preferences that shape what’s relevant to you ("I live near San Francisco" → local options should be tailored to this area)

In developing the new memory system, we improved ChatGPT’s ability to apply relevant preferences from past conversations. Following the "I’m vegetarian" example above, we can evaluate whether the model correctly leverages memory to produce vegetarian-friendly dining options when a vegetarian user asks for meal prep suggestions.

Staying current over time

Time doesn’t stop when your chat ends.

Traditional memory systems can become stale. For example, you tell ChatGPT "I’m in Singapore and need a dinner recommendation for tonight." Then, time passes, your trip ends, and you wonder why ChatGPT still thinks you’re in Singapore.

With dreaming, memories are automatically updated as time passes, allowing ChatGPT to revise its memory from "You’re going to Singapore in July" to "You went to Singapore in July 2026" when the trip ends. Then, when you’re back home, ChatGPT can again provide recommendations that are tailored to your home location and time zone.

Stale memory

Help me find something for take-out tonight that’s still open.

It’s about 5:19 AM Sunday in Singapore, so your safest take-out/delivery bets are 24-hour places.

Best bets right now:

McDelivery / McDonald’s Singapore — easiest islandwide option; McDonald’s says McDelivery is available 24 hours, 7 days a week. Breakfast menu should be on now. (McDonald's Singapore⁠(opens in a new window))
Srisun Express — good for prata, Indian Muslim food, local comfort food; their site says they’re halal-certified and open 24/7. (Sri Sun Express⁠(opens in a new window))
89.7 Supper Club — good if you’re near Changi Village or Geylang East; their outlets page says both locations are open 24 hours, 7 days a week. (89.7 Supper Club⁠(opens in a new window))
126 Dim Sum Wen Dao Shi — worth checking on GrabFood/foodpanda if you’re in the East; a 2025 guide says it runs till 5:30 AM on weekdays and round-the-clock on weekends. Since it’s Sunday morning, it may still be available. (Eatbook⁠(opens in a new window))
Prata around Geylang / Serangoon / Bedok / Jalan Kayu — look in GrabFood or Google Maps for “prata open now”; guides list multiple 24-hour prata/supper spots, including Mufiz Prata Corner and RK Eating House. (Eatbook⁠(opens in a new window))

For the fastest path: open GrabFood / foodpanda / Deliveroo, set filter to Open now, then search “prata,” “McDonald’s,” “Srisun,” “89.7,” “dim sum,” or “nasi lemak.” If you tell me your neighborhood or nearest MRT, I’ll narrow it to specific nearby places.

The model thinks the user is still in Singapore.

With memory

Help me find something for take-out tonight that’s still open.

I’ll use Portola Valley / Ladera as the starting point and prioritize places that are plausibly open for takeout tonight, then narrow to low-friction options rather than a huge list.

For tonight, Saturday May 30, near Ladera / Portola Valley, I’d start here:

Alpine Inn — safest casual pickup option. Open Saturday until 9:30 pm and has Order Takeout on its site. Good for burgers / tavern food. (Rossotti's Alpine Inn⁠(opens in a new window))
Taverna Portola Valley — nearby at Ladera Country Shopper; open daily, Saturday dinner until 9:00 pm. Better if you want Greek / Mediterranean. (TAVERNA Portola Valley⁠(opens in a new window))
Portola Bistro — also nearby; Saturday dinner until 9:30 pm. Good option for Italian / Mediterranean / pizza-ish dinner. (Portola Bistro -⁠(opens in a new window))
Amigos Grill — Mexican, very close in Ladera Country Shopper. Official site says call to order at 650-851-3997; DoorDash / order.online lists pickup/delivery ordering until 8:40 pm. (Amigos Grill⁠(opens in a new window))
State of Mind Slice House, Palo Alto — best fallback if you want pizza and can drive farther; Friday–Saturday hours are 11 am–10 pm. (State of Mind Slice House⁠(opens in a new window))

My pick: Alpine Inn for easiest local takeout, Taverna if you want something less heavy, State of Mind if it’s later and you want the most reliable post-9 pm option.

The model provides responses that are relevant to the user’s home location.

In our memory evaluations, we measure whether ChatGPT can correctly respond to prompts where the passage of time materially affects the correct answer or recommendation. Dreaming provides a substantial lift in this area:

A more scalable foundation for the future

At OpenAI, our mission is to ensure that artificial general intelligence benefits all of humanity.

While dreaming-based memory has been available to Plus and Pro users for some time, we are only now able to offer Free users a version that meets our quality bar and is practical to serve at scale. Recent improvements reduced the compute required to serve dreaming to Free users by approximately 5x, making it possible to begin rolling out dreaming to Free users over the coming weeks and to increase memory capacity for Plus and Pro users.

Looking ahead, dreaming now provides us with a shared memory foundation for all users. This update represents our most capable memory system yet, and we’ll continue improving it.

To learn more about this release and memory user controls, visit our Memory FAQ⁠(opens in a new window).

An OpenAI model has disproved a central conjecture in discrete geometry

Wed, 20 May 2026 00:00:00 GMT

Loading…

For nearly 80 years, mathematicians have studied a deceptively simple question: if you place $n$ points in the plane, how many pairs of points can be exactly distance $1$ apart?

This is the planar unit distance problem, first posed by Paul Erdős in 1946. It is one of the best-known questions in combinatorial geometry, easy to state and remarkably difficult to resolve. The 2005 book Research Problems in Discrete Geometry, by Brass, Moser, and Pach, calls it “possibly the best known (and simplest to explain) problem in combinatorial geometry.” Noga Alon, a leading combinatorialist at Princeton, describes it as “one of Erdős’ favorite problems.” Erdős even offered a monetary prize for resolving this problem.

Today, we share a breakthrough on the unit distance problem. Since Erdős’s original work, the prevailing belief has been that the “square grid” constructions depicted further below were essentially optimal for maximizing the number of unit-distance pairs. An internal OpenAI model has disproved this longstanding conjecture, providing an infinite family of examples that yield a polynomial improvement. The proof has been checked by a group of external mathematicians. They have also written a companion paper explaining the argument and providing further background and context for the significance of the result.

The result is also notable for how it was found. The proof came from a new general-purpose reasoning model, rather than from a system trained specifically for mathematics, scaffolded to search through proof strategies, or targeted at the unit distance problem in particular. As part of a broader effort to test whether advanced models can contribute to frontier research, we evaluated it on a collection of Erdős problems. In this case, it produced a proof resolving the open problem.

This proof is an important milestone for the math and AI communities. It marks the first time that a prominent open problem, central to a subfield of mathematics, has been solved autonomously by AI. It also demonstrates the depth of reasoning these systems now support. Mathematics provides a particularly clear testbed for reasoning: the problems are precise, potential proofs can be checked, and a long argument only works if the reasoning holds together from beginning to end. The method by which the problem was solved is also notable. The proof brings unexpected, sophisticated ideas from algebraic number theory to bear on an elementary geometric question.

Fields medalist Tim Gowers, writing in the companion paper, calls the result “a milestone in AI mathematics.” According to leading number theorist Arul Shankar, “In my opinion this paper demonstrates that current AI models go beyond just helpers to human mathematicians – they are capable of having original ingenious ideas, and then carrying them out to fruition”.

Mathematicians on the result

1 of 4

“This has been one of Erdős' favorite problems, I have heard him myself mentioning the problem multiple times in his lectures. I believe it would be fair to say that every mathematician working in Combinatorial Geometry thought about this problem, and lots of mathematicians working in other areas spent at least some time thinking about it… The solution of the problem by the internal model of Open AI is, in my opinion, an outstanding achievement, settling a long-standing open problem. The fact that the correct answer is not $n^{1+o(1)}$ is surprising, and the construction and its analysis apply fairly sophisticated tools from algebraic number theory in an elegant and clever way.”

Noga Alon

“The model’s CoT is deeply interesting. It is noteworthy that a significant majority of the thoughts are trying to construct a counterexample to the widely believed upper bound, rather than trying to prove it. This argues that the model has some combination of good intuition, willingness to try approaches considered long-shot by the community, and a predisposition to attempt constructions.… In my opinion this paper demonstrates that current AI models go beyond just helpers to human mathematicians – they are capable of having original ingenious ideas, and then carrying them out to fruition.”

Arul Shankar

The proof is available here⁠(opens in a new window). The companion paper by leading external mathematicians is available here⁠(opens in a new window). You can find an abridged version of the model’s chain of thought here⁠(opens in a new window).

Previously known construction of many unit distances from a rescaled square grid.

The unit distance problem

Let $u(n)$ be the largest possible number of unit-distance pairs among $n$ points in the plane. Examples attaining linear growth rate are easy to construct: placing $n$ points in a line gives $n-1$ pairs, while a square grid gives about $2n$ pairs. The previously best known construction, coming from a rescaled square grid, turns out to give even more: $n^{1 + C / \log \log(n)}$ for a constant $C$ . Since $\log \log(n)$ tends to infinity with $n$ , the additional term in the exponent tends to $0$ , meaning these constructions achieve growth only slightly faster than linear. For decades, it was widely believed that this rate was essentially the best possible, and no construction could improve significantly over the square grid. In technical terms, Erdős conjectured an upper bound of $n^{1+o(1)}$ in which the additional $o(1)$ indicates a term tending to $0$ with $n$ .

Our new result disproves this conjecture. More precisely, for infinitely many values of $n$ , the proof constructs configurations of $n$ points with at least $n^{1+\delta}$ unit-distance pairs, for some fixed exponent $\delta > 0$ . (The original AI proof does not give an explicit $\delta$ , but a forthcoming refinement due to Princeton mathematics professor Will Sawin has shown one can take $\delta=0.014$ .)

The history of the problem helps to see why the result is surprising. The best known lower bound had been essentially unchanged since Erdős’s original 1946 construction. The best upper bound, $O(n^{4/3})$ , dates to work by Spencer, Szemerédi, and Trotter in 1984, and despite later refinements and related structural work by Székely, Katz and Silier, Pach, Raz, and Solymosi and by others, the upper bound has remained essentially unchanged. As evidence in favor of the conjecture, Matoušek and Alon-Bucić-Sauermann studied the problem with non-Euclidean distances in the plane, and proved that "most" of these non-Euclidean distances obey the conjecture in some sense.

Surprisingly, the key ingredients of the construction come from a very different part of mathematics known as algebraic number theory, which studies concepts like factorization in extensions of the integers known as algebraic number fields.

After verifying the initial proof, we investigated the success rate of our models on this problem with varying amounts of test-time compute. The results are shown here.

New techniques from algebraic number theory

At a high level, the proof begins with a familiar geometric idea and pushes it in an unexpected direction.

Erdős’s original lower bound can be understood through the Gaussian integers: numbers of the form $a+bi$ , where $a$ and $b$ are integers and $i$ is the square root of $-1$ . The Gaussian integers extend the ordinary integers and, like them, enjoy properties such a unique factorization into primes. Such extensions of the ordinary integers or rationals are known as algebraic number fields. The new argument replaces the Gaussian integers by more complicated generalizations from algebraic number theory with richer symmetries that can create many more unit-length differences.

The precise argument uses tools such as infinite class field towers and Golod–Shafarevich theory to show the number fields required for the argument actually exist. These ideas were well-known to algebraic number theorists, but it came as a great surprise that these concepts have implications for geometric questions in the Euclidean plane.

What this means for mathematics

This result marks an important moment in the interaction between AI and mathematics: an AI system has autonomously resolved a longstanding open problem at the center of an active field. It also offers an early glimpse of a new kind of collaboration between AI and human mathematicians. In this case, the companion work by external mathematicians paints a substantially richer picture than the original solution alone.

As Thomas Bloom writes in the companion note:

“When assessing the importance and influence of an AI-generated proof, a question I ask myself is: has this taught us something new about the problem? Do we understand discrete geometry better now? I think the answer is a moderated yes: this shows that there is a lot more that number theoretic constructions have to say about these sorts of questions than we suspected; moreover, that the number theory required can be very deep. No doubt many algebraic number theorists will be taking a close look at other open problems in discrete geometry in the coming months.”

The unexpected connection between algebraic number theory and discrete geometry revealed by the solution is part of what makes the result notable. It does not simply settle a specific conjecture, but may provide mathematicians with a bridge to begin exploring further related problems.

Bloom also points toward a broader possibility:

“The frontiers of knowledge are very spiky, and no doubt the coming months and years will see similar successes in many other areas of mathematics, where long-standing open problems are resolved by an AI revealing unexpected connections and pushing the existing technical machinery to its limit. AI is helping us to more fully explore the cathedral of mathematics we have build over the centuries; what other unseen wonders are waiting in the wings?”

This result provides a promising example: AI contributing not only a solution, but a mathematical discovery whose significance becomes clearer and richer through subsequent human understanding.

Why this matters

The takeaway is bigger than this particular result. Better mathematical reasoning can make AI a stronger research partner: something that can hold together difficult lines of thought, connect ideas across distant areas of knowledge, surface promising paths experts may not have prioritized, and help researchers make progress on problems that would otherwise be too complex or time-intensive to tackle.

Those capabilities matter beyond mathematics. If a model can keep a complicated argument coherent, connect ideas across distant areas of knowledge, and produce work that survives expert scrutiny, those are also useful abilities in biology, physics, materials science, engineering, and medicine, and they are part of our longer-term path toward more automated research: systems that can help scientists and engineers explore more ideas and pursue harder technical questions.

AI is about to start taking a very serious role in the creative parts of research, and most importantly AI research itself. While this progress is not unexpected, it reinforces the urgency we feel about understanding this next phase of AI development, the challenges of aligning very intelligent systems, and the future of human-AI collaboration.

That future still depends on human judgment. Expertise becomes more valuable, not less. AI can help search, suggest, and verify. People choose the problems that matter, interpret the results, and decide what questions to pursue next.

What Parameter Golf taught us about AI-assisted research

Tue, 12 May 2026 00:00:00 GMT

Loading…

We launched Parameter Golf to engage and support the machine learning research community in exploring a new, tightly constrained machine learning problem. We wanted the challenge to be interesting enough to reward real technical creativity, while remaining conceptually simple and easy to verify.

Participants had to minimize held-out loss on a fixed FineWeb dataset while staying within a 16 MB artifact limit, including both model weights and training code, and a 10-minute training budget on 8×H100s. We provided a baseline, dataset, and evaluation scripts so participants could fork the repo, improve the model, and submit their results through GitHub.

Over the course of eight weeks, we received more than 2,000 submissions from over 1,000 participants. We were impressed by the technical breadth, creativity, and rule-bending across the submissions, from careful optimizer tuning and quantization work to new modeling ideas and test-time training.

One of the most exciting parts of the challenge was seeing how widely participants used AI coding agents. Agents helped lower the cost of experimentation, made it easier for more people to participate, and changed the pace of the competition. They also created new challenges for submission review, attribution, and scoring.

The challenge also became a meaningful talent discovery surface for us. That was one of our goals for Parameter Golf, and it was a useful signal that open-ended technical challenges can reveal exceptional machine learning taste and persistence.

In this post, we highlight some of the submissions we found surprising and interesting, and share what we learned from running a coding contest in the age of powerful AI agents.

Technical impressions

Record track

We judged and independently reproduced each submission on the record-track leaderboard, and verified that each submission was record-breaking at the time it was submitted. Several themes stood out.

Training optimization

Some of the strongest results came from careful tuning of existing components.

Submission	Contributor	Technique	Why it mattered
#60	@notapplica	Combined prior wins from #50, #42, and likely #39, then made a deeper model work with Muon weight decay, spectral embedding initialization, residual-mix scheduling, and compiled evaluation.	A strong example of disciplined leaderboard work: identifying which existing improvements matter and combining them cleanly.

Quantization

Several submissions pushed hard on compression and export.

Submission	Contributor	Technique	Why it mattered
#414	@signalrush	Used GPTQ-lite to quantize weights after training.	The first leaderboard submission to successfully use GPTQ-lite, leading to better evaluation.
#1060	@dexhunter	Built on #634 by @raahilshah to successfully use full Hessian GPTQ.	Extended earlier quantization work into a stronger compression path.

Test-time and evaluation strategies

Some submissions pushed the boundary between model improvement and evaluation strategy. These approaches were valid under the rules, but they required careful review from us as organizers.

Submission	Contributor	Technique	Why it mattered
#77	@samacqua	Used score-first, per-document LoRA test-time training: score first, adapt only on already-scored chunks, and reset at document boundaries.	Pushed the boundary between model improvement and evaluation strategy while staying reviewable under the rules.
#1019	@abaybektursun	Used self-generated GPTQ calibration: generate calibration text from the trained model, then build GPTQ Hessians from those activations.	A creative calibration strategy that required careful review from organizers.

New modeling and data ideas

A few submissions introduced modeling or data ideas that were especially creative.

Submission	Contributor	Technique	Why it mattered
#1729	@romeerp	Introduced the CaseOps tokenizer: lossless capitalization operator tokens with original-byte BPB sidecar accounting.	A creative tokenizer and data-representation idea.
#265	@unnir	Introduced XSA, an efficient partial Exclusive Self Attention approach with GQA-aware grouped views.	Brought an efficient attention variant into the challenge.
#65	@aquariouseworkman	Introduced SmearGate and BigramHash: a learned previous-token embedding blend plus adjacent-token-pair hash features.	Added new feature mechanisms from scratch.
#1204	@msisovic	Introduced mini depth recurrence: repeated layers 4 and 5, delayed recurrence until mid-training, and partially untied the repeated MLPs.	The first accepted leaderboard row to make recurrent layers work effectively.

We chose to highlight these nine submissions because they represent the range of results we hoped the challenge would surface. Some participants found wins through careful tuning. Others pushed quantization and low-rank techniques. Some explored edges of the evaluation rules. And several introduced modeling or data ideas, from the literature or from scratch, that produced unexpected gains.

Nonrecord track

The nonrecord track was home to many creative submissions. We highlighted 15 favorites, including approaches ranging from non-autoregressive text modeling to dynamic tokenization.

Because this track was more experimental, we focused less on raw performance and more about whether the approach was technically interesting. Three submissions stood out in particular:

These were our favorite three nonrecord submissions, even though they were not necessarily the top three by performance.

That said, the nonrecord track was still competitive. Half of nonrecord leaderboard entries beat the naive baseline of 1.22 BPB, and the top-ranked entry reached 1.12 BPB.

We found this encouraging. Even against strong transformer baselines, alternative approaches could sometimes hold their own against the dominant architecture.

We also think that this track benefits especially from the availability of strong coding agents. Agents made it much cheaper to prototype speculative ideas, including approaches that may previously have felt too time-consuming or uncertain to try in a short competition.

Takeaways

A major difference between Parameter Golf and earlier competitions like it was the widespread use of coding agents. The vast majority of submitters mentioned using agents as part of their work.

That lowered the barrier to entry. Participants could set up experiments faster, inspect unfamiliar code, and test ideas with less friction. Runpod’s sponsorship of $1,000,000 in compute also played a major role in making the challenge accessible to more people.

At the same time, agent use created new issues for submission and scoring. Many submissions were small changes to existing top scorers, rather than fundamentally new approaches. This was often useful: strong ideas spread quickly and were refined by others. But it also created noise. When submissions that fell outside the competition guidelines produced unusually strong scores, other agents sometimes copied those ideas and continued down the same invalid path.

The volume of submissions also changed how we had to run the competition. We could not manually inspect every submission and still keep the leaderboard moving. During the challenge, we developed an internal Codex-based triage bot to monitor new submissions and flag them for human review. This became especially important during periods when we received hundreds of submissions a day.

AI agents also became part of the community around the challenge. For much of the competition, @notapplica and their coding agent ran a “Live Updates” bulletin, tracking major events, explaining leaderboard approaches, and helping other participants follow the competition. Community review tools also appeared to help less experienced participants check whether their submissions were within the rules and avoid common invalid approaches.

What’s next?

Our primary goal was to launch a challenge that eligible participants⁠(opens in a new window) could take part in and experience machine learning research. Parameter Golf brought in a wide range of technically strong and creative submissions, and it gave us a clearer view of how open research competitions may change as AI agents become more capable and widely used.

We are thinking about launching more challenges like this in the future. If you’re interested, please fill out the challenge participant form⁠(opens in a new window).

Introducing OpenAI Privacy Filter

Wed, 22 Apr 2026 00:00:00 GMT

Loading…

Today we’re releasing OpenAI Privacy Filter, an open-weight model for detecting and redacting personally identifiable information (PII) in text. This release is part of our broader effort to support a more resilient software ecosystem by providing developers practical infrastructure for building with AI safely, including tools⁠ and models⁠ that make strong privacy and security protections easier to implement from the start.

Privacy Filter is a small model with frontier personal data detection capability. It is designed for high-throughput privacy workflows, and is able to perform context-aware detection of PII in unstructured text. It can run locally, which means that PII can be masked or redacted without leaving your machine. It processes long inputs efficiently, making redaction decisions in a quick, single pass.

At OpenAI, we use a fine-tuned version of Privacy Filter in our own privacy-preserving workflows. We developed Privacy Filter because we believe that with the latest AI capabilities, we could raise the standard for privacy beyond what was already on the market. The version of Privacy Filter we are releasing today achieves state-of-the-art performance on the PII-Masking-300k benchmark, when corrected for annotation issues we identified during evaluation.

With this release, developers can run Privacy Filter in their own environments, fine tune it to their own use cases, and build stronger privacy protections into training, indexing, logging, and review pipelines.

A small model with frontier personal data detection capability

Privacy protection in modern AI systems depends on more than pattern matching. Traditional PII detection tools often rely on deterministic rules for formats like phone numbers and email addresses. They can work well for narrow cases, but they often miss more subtle personal information and struggle with context.

Privacy Filter is built with deeper language and context awareness for more nuanced performance. By combining strong language understanding with a privacy-specific labeling system, it can detect a wider range of PII in unstructured text, including cases where the right decision depends on context. It can better distinguish between information that should be preserved because it is public, and information that should be masked or redacted because it relates to a private individual.

The result is a model that is strong enough to deliver frontier-level privacy filtering performance. At the same time, the model is small enough to be run locally–meaning data that has yet to be filtered can remain on device, with less risk of exposure, rather than needing to be sent to a server for de-identification.

Model overview

Privacy Filter is a bidirectional token-classification model with span decoding. It begins from an autoregressive pretrained checkpoint and is then adapted into a token classifier over a fixed taxonomy of privacy labels. Instead of generating text token by token, it labels an input sequence in one pass and then decodes coherent spans with a constrained Viterbi procedure.

This architecture gives Privacy Filter a few useful properties for production use:

Fast and efficient: all tokens are labeled in a single forward pass.
Context aware: the language prior enables PII spans to be detected based on surrounding context.
Long-context: the released model supports up to 128,000 tokens of context.
Configurable: developers can tune operating points to trade off recall and precision depending on their workflow.

The released model has 1.5B total parameters with 50M active parameters.

Privacy Filter predicts spans across eight categories:

private_person
private_address
private_email
private_phone
private_url
private_date
account_number
secret

The account_number category helps mask a wide variety of account numbers, including banking info like credit card numbers and bank account numbers, while secret helps mask things like passwords and API keys.

These labels are decoded with BIOES span tags, which helps produce cleaner and more coherent masking boundaries.

Example input text

Subject: Q2 Planning Follow-Up

Hi Jordan,

Thanks again for meeting earlier today. I wanted to follow up with the revised timeline for the Q2 rollout and confirm that the product launch is scheduled for September 18, 2026. For reference, the project file is listed under 4829-1037-5581. If anything changes on your side, feel free to reply here at maya.chen@example.com or call me at +1 (415) 555-0124.

Best,

Maya Chen

Text after masking personal identifiers

Subject: Q2 Planning Follow-Up

Hi [PRIVATE_PERSON],

Thanks again for meeting earlier today. I wanted to follow up with the revised timeline for the Q2 rollout and confirm that the product launch is scheduled for [PRIVATE_DATE]. For reference, the project file is listed under [ACCOUNT_NUMBER]. If anything changes on your side, feel free to reply here at [PRIVATE_EMAIL] or call me at [PRIVATE_PHONE].

Best,

[PRIVATE_PERSON]

How we built it

We developed Privacy Filter in several stages.

First, we built a privacy taxonomy that defines the types of spans the model should detect. This includes personal identifiers, contact details, addresses, private dates, many different kinds of account numbers such as credit and banking information, and secrets such as API keys and passwords.

Second, we converted a pretrained language model into a bidirectional token classifier by replacing the language modeling head with a token-classification head and post-training it with a supervised classification objective.

Third, we trained on a mixture of publicly available and synthetic data designed to capture both realistic text and difficult privacy patterns. In parts of the public data where labels were incomplete, we used model-assisted annotation and review to improve coverage. We also generated synthetic examples to increase diversity across formats, contexts, and privacy subtypes.

At inference time, the model's token-level predictions are decoded into coherent spans using constrained sequence decoding. This approach preserves the broad language understanding of the pretrained model while specializing it for privacy detection.

How Privacy Filter performs

We evaluated Privacy Filter on standard benchmarks and on additional synthetic and chat-style evaluations designed to test harder, more context-sensitive cases.

On the PII-Masking-300k⁠(opens in a new window) benchmark, Privacy Filter achieves an F1 score of 96% (94.04% precision and 98.04% recall). On a corrected version of the benchmark that accounts for dataset annotation issues identified during review, the F1 score is 97.43% (96.79% precision and 98.08% recall).

We also found that the model can be adapted efficiently. Fine-tuning on even a small amount of data quickly improves accuracy on domain-specific tasks, increasing F1 score from 54% to 96% and approaches saturation on the domain-adaption benchmark we evaluated.

Beyond benchmark performance, Privacy Filter is designed for practical privacy filtering in noisy, real-world text. That includes long documents, ambiguous references, mixed-format strings, and software-related secrets. The model card ⁠(opens in a new window)also reports targeted evaluation on secret detection in codebases and stress tests across multilingual, adversarial, and context-dependent examples.

Limitations

Privacy Filter is not an anonymization tool, a compliance certification, or a substitute for policy review in high-stakes settings. It is one component in a broader privacy-by-design system.

Its behavior reflects the label taxonomy and decision boundaries it was trained on. Different organizations may want different detection or masking policies, and those policies may require in-domain evaluation or further fine-tuning. Performance may also vary across languages, scripts, naming conventions, and domains that differ from the training distribution.

Like all models, Privacy Filter can make mistakes. It can miss uncommon identifiers or ambiguous private references, and it can over- or under-redact entities when context is limited, especially in short sequences. In high-sensitivity domains such as legal, medical, and financial workflows, human review and domain-specific evaluation and fine-tuning remain important.

Availability

We are releasing OpenAI Privacy Filter to support stronger privacy protections across the ecosystem.

The model is available today under the Apache 2.0 license on Hugging Face⁠(opens in a new window) and Github⁠(opens in a new window). It is intended for experimentation, customization, and commercial deployment, and it can be fine-tuned for different data distributions and privacy policies.

Alongside the model, we are sharing documentation covering the model architecture, label taxonomy, decoding controls, intended use cases, evaluation setup, and known limitations, so teams can understand both what the model does well and where it should be used carefully.

Looking ahead

Privacy protection for AI systems is an ongoing effort across research, product design, evaluation, and deployment.

Privacy Filter reflects one direction we believe is important: small, efficient models with frontier capability in narrowly defined tasks that matter for real-world AI systems. We are releasing it because we think privacy-preserving infrastructure should be easier to inspect, run, adapt, and improve.

Our goal is for models to learn about the world, not about private individuals. Privacy Filter helps make that possible.

We’re releasing this preview of Privacy Filter to receive feedback from the research and privacy community and iterate further on model performance.

OpenAI Research

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

ARC-AGI-3

Agents do best when they remember what they’ve done

Conclusion and recommendations

Separating signal from noise in coding evaluations

Methodology

Human-supervised agent review

Human annotation campaign

Failure modes

OpenLibrary-77c16d5

None

None

Discussion

Introducing GeneBench-Pro

Dataset construction

Domain Atlas: 129 problems in 10 domains and 21 sub-domains

Evaluation and grading

Structural variant-guided tumor therapy benefit-risk decision

JSON

Results

Problem: Pharmacogenomic time-to-event response with time-varying treatment

GPT-5.5 pattern

GPT-5.6 Sol pattern

A near-autonomous AI chemist improves a challenging reaction in medicinal chemistry

Why the chemistry problem matters

Connecting GPT‑5.4 to Maria AI and Lab

What we found

TEMPO improves product formation at bench scale

Limitations

Preparedness

What’s next

Introducing LifeSciBench

What LifeSciBench measures

Dataset construction

Grading and rubric breakdown

Eval Example

Candidate Response

Rubric Criteria & Grades

Validating LifeSciBench

Real-world relevance

Scientific reasoning / domain skill

Scientific grounding

Overall usefulness

Results

Where AI systems show early strength

Where AI systems still fall short

Limitations & what’s next

Get involved

Predicting model behavior before release by simulating deployment

Introduction

How Deployment Simulation works

How we tested Deployment Simulation

Deployment Simulation significantly expands pre-deployment risk assessment

Reducing evaluation awareness

Tool simulation for agentic trajectories

WildChat and external auditing

Limitations

Conclusion

Dreaming: Better memory for a more helpful ChatGPT

How memory has evolved

How we evaluate memory

Carrying forward context

Without memory

Basic shopping checklist

Which route should you choose?

The compatibility rule

With memory

Best buy for Mini Flash 3 TTL

For Inon Z-330 TTL

Practical recommendation

Following preferences

Without memory

A strong 3-day plan

Day 1: Marina Bay, Gardens, and skyline

Day 2: Culture neighborhoods and food

Day 3: Nature or beach day

Best places to stay after the work part

What to book ahead

July packing notes