GPQA is a set of graduate-level multiple-choice questions in biology, physics and chemistry, built so that people with PhDs in the field can answer them while skilled outsiders with unlimited web access mostly cannot. GPQA Diamond is its strictest subset of 198 questions, and it is the version AI labs quote most often in model announcements. Every figure comes from the GPQA paper unless another source is linked.
On this page
Who built GPQA and why
GPQA stands for “Graduate-Level Google-Proof Q&A”. David Rein and seven co-authors at New York University posted the paper on arXiv on November 20, 2023. The benchmark was not designed as a leaderboard. Its stated purpose is research on scalable oversight: how a person can check the answers of an AI system that knows more than they do. For that, the authors needed questions that experts can settle, that well-resourced non-experts get wrong, and that the strongest models of the time also got wrong.
How the questions were written and checked
The authors hired 61 contractors through Upwork, each with a completed PhD or one in progress in their field. Every question then went through four stages:
- Question writing. An expert writes a question with four answer options and an explanation of the correct answer.
- First expert validation. A second expert in the same area answers it without seeing the solution, then gives feedback.
- Revision. The writer revises the question using that feedback.
- Second expert and non-expert validation. Another expert answers the revised question, and three non-experts try it too.
The non-experts are the important part. They are themselves PhD-level contractors, but from other fields: a physicist cannot act as a non-expert on any physics question, and organic chemists are kept away from biology questions and vice versa. They had unrestricted internet access and no time limit, but were not allowed to use AI assistants. They had to spend at least 15 minutes per question and averaged 37 minutes (median 30). That is what “Google-proof” means here: searching the web for half an hour did not reliably produce the answer.
Payment was tied to the outcome. Writers earned bonuses when experts answered correctly and when non-experts answered incorrectly, so the incentive was to write questions that are both objective and hard.
Three subsets: Extended, main and Diamond
The authors collected 564 questions, held back 18 as an unreleased test set and released the remaining 546 in three nested versions:
| Subset | Questions | Selection rule | Expert accuracy | Non-expert accuracy |
|---|---|---|---|---|
| GPQA Extended | 546 | All released questions | 64.8% | 34.1% |
| GPQA (main set) | 448 | At least one of two experts agrees with the answer, and no more than two of three non-experts get it right | 71.9% | 30.4% |
| GPQA Diamond | 198 | Both experts agree with the answer, and at most one of three non-experts gets it right | 81.3% | 22.1% |
The paper notes that the main-set and Diamond accuracies are skewed by selection: the questions were chosen partly because experts answered them correctly and non-experts did not. An expert “agrees” either by answering correctly at the first attempt or by clearly explaining their own mistake after seeing the writer’s answer.
By field, the 546 Extended questions split into 227 physics, 214 chemistry and 105 biology. Organic chemistry (144 questions) and molecular biology (85) are the largest subdomains.
What a GPQA Diamond score means
Each question has four options, so random guessing scores about 25%. In the original paper, GPT-4 with few-shot chain-of-thought prompting scored between 38% and 40% across the three subsets, roughly the level of skilled non-experts with web access and far below the experts. A version of GPT-4 with web search reached 39.4% on the Extended set.
Lab-reported scores today sit far higher. For example, Moonshot AI’s Kimi K3 model card reports 93.5% on GPQA Diamond with reasoning effort set to “max”, temperature 1.0 and top-p 0.95. Scores like that are above the 81.3% expert accuracy measured on the same subset, but the two numbers are not a like-for-like contest: the expert figure comes from individual contractors answering questions under the paper’s protocol, and it is shaped by the selection rule described above.
When you read a GPQA Diamond number in a model announcement, check three things:
- Who ran it. Most scores are self-reported by the lab. Our LLM leaderboard marks each GPQA Diamond row as lab-reported, maintainer-run or independently evaluated, with the source and date.
- The setting. Reasoning effort, sampling temperature, the number of attempts averaged and whether tools were allowed all move the score. Labs state these in footnotes, as the Kimi K3 card does.
- The subset. “GPQA” and “GPQA Diamond” are different test sets of 448 and 198 questions; a score on one is not a score on the other.
Limits the authors point out
The paper lists its own limitations, and they matter more now that scores are crowded near the top:
- Small size. With only 448 questions in the main set, the authors write that detecting a difference between two models with over 80% statistical power needs a large gap, such as 50% versus 60%. Diamond, at 198 questions, is smaller still: one question is worth about half a percentage point, so a gap of one or two points between two models can come down to a handful of questions.
- Contamination risk. The authors ask people not to post examples from the dataset in plain text or images online, and they embedded a canary string in the data so that training pipelines can filter it out. Whether a given model saw the questions during training cannot be checked from a score alone.
- Who wrote the questions. Experts were recruited through Upwork without controls for region or background, and the authors make no claim that GPQA represents the questions that come up in real scientific work.
- A proxy, not the target. GPQA was built to test oversight of systems that know more than their supervisors; the authors call it an imperfect proxy for questions nobody can yet answer.
How GPQA fits with other benchmarks
GPQA Diamond tests whether a model can pick the right answer to a hard, closed science question. It says nothing about writing working code, which is what SWE-bench and HumanEval measure, or about long multi-step tasks with tools. Labs usually report it alongside those benchmarks in the evaluation tables of their model and system cards. New results as they are published are in AI benchmark news.
