i made agi take the SAT
what happens when u give today’s frontier AI models a standardized test designed for 17 year olds? i tested 20 AI models from US and Chinese providers on official digital SAT practice tests from College Board — the full thing: reading & writing AND math, all 4 modules, figures included.
spoiler: the frontier scores 1440-1485 — 95th-97th percentile among actual SAT takers. reading is basically solved. math with figures is where they bleed. and none of them are getting into MIT. (part 2 below has the bell curve and the actual admissions math.)
Imagen 4 prompt: robot taking standardized test, stressed, pencil in hand, scantron sheet, realistic
tldr
- gemini-3.1-pro leads at 1485 (97th percentile), claude-opus-4.8 right behind at 1470 — statistically tied
- four models put up a perfect 800 on reading & writing. that section no longer discriminates at the frontier.
- best math score: 685. figure questions (graphs, geometry diagrams) are the universal weak spot
- DeepSeek matches the frontier on text questions (93.1%) — it just can’t see images, so it can’t take the full test
- every score comes with a confidence interval, a shuffle-robustness check, and a contamination probe (data below, not vibes)
the setup
i built a benchmark harness that runs models through complete official digital SAT practice tests (#10 and #11) — 120 questions each: 66 reading & writing + 54 math across 4 modules. this includes:
- vocabulary in context (what does “obfuscate” mean in paragraph 3)
- reading comprehension, text structure, grammar, rhetorical synthesis
- algebra, advanced math, problem-solving & data analysis, geometry/trig
- figure questions: 34 of 120 questions require actually reading a scatterplot, graph, or diagram — these get sent to the models as images
how models answer: chain-of-thought, one question per request, fresh context every time, final answer demanded on its own Answer: X line. responses with no parseable answer are recorded as unparseable — counted against the model like a blank bubble, never coerced into a guess. API failures are retried until every model has a complete, error-free answer sheet.
how scores are computed: raw section scores go through the official College Board conversion table for that exact test form (the real scale is nonlinear, floor 200 per section). uncertainty via item bootstrap (2,000 resamples). models whose intervals overlap the leader are marked statistically tied — the SAT’s own standard error is ~30-40 points, so a 20-point gap on one sitting means nothing.
providers tested:
- US: OpenAI, Anthropic, Google, xAI, Meta
- China: DeepSeek, Zhipu AI (GLM), Moonshot (Kimi), MiniMax, Alibaba (Qwen), StepFun
- EU: Mistral
two full tests × 20 models × CoT, plus a direct-answer condition, a shuffled-options robustness run, a no-passage contamination probe, and a 426-question AGIEval anchor: ~14,000 graded responses, ~$50 in API costs.
the results
live interactive version with CI bars: vincentwi.com/sat-benchmark
| Band | Model | Total | 95% CI | Pctl (SAT takers) | RW | Math |
|---|---|---|---|---|---|---|
| 1 | gemini-3.1-pro | 1485 | 1425–1560 | 97 | 800 | 685 |
| 1 | claude-opus-4.8 | 1470 | 1405–1555 | 97 | 790 | 680 |
| 1 | gemini-3.5-flash | 1445 | 1365–1520 | 95 | 800 | 645 |
| 1 | gpt-5.5 | 1445 | 1375–1520 | 95 | 800 | 645 |
| 1 | grok-4.5 | 1445 | 1370–1515 | 95 | 800 | 645 |
| 1 | claude-sonnet-5 | 1440 | 1365–1520 | 95 | 785 | 655 |
| 1 | gpt-5.2 | 1435 | 1360–1510 | 95 | 795 | 640 |
| 1 | o4-mini | 1380 | 1300–1465 | 92 | 765 | 615 |
| 1 | gemini-2.5-flash | 1365 | 1290–1450 | 91 | 775 | 590 |
| 1 | claude-haiku-4.5 | 1345 | 1260–1445 | 89 | 735 | 610 |
| 2 | gpt-4.1 | 1310 | 1230–1400 | 87 | 750 | 560 |
| 2 | llama-4-maverick | 1305 | 1215–1390 | 86 | 740 | 565 |
| 2 | grok-4.3 | 1285 | 1210–1375 | 84 | 775 | 510 |
| 2 | mistral-large-3 | 1265 | 1170–1360 | 83 | 705 | 560 |
text-only models — no vision means the 34 figure questions are unanswerable, so no honest full-test score. on the text-question subset:
| Model | Text-only accuracy |
|---|---|
| DeepSeek V4 Pro | 93.1% |
| Qwen3.5-397B | 91.5% |
| Kimi K2.6 | 91.1% |
| GLM-5.2 | 89.9% |
| Step-3.7-Flash | 86.6% |
| MiniMax M3 | 85.5% |
for scale: gemini-3.1-pro and opus get 93.9% on the same subset. DeepSeek is statistically indistinguishable from the frontier on text. it just can’t see.
things to notice
- reading & writing is done. four 800s. the digital SAT’s RW section no longer separates frontier models.
- math with figures is the moat. best math: 685. the misses concentrate hard in figure questions — reading a scatterplot, a geometry diagram, a graphed system of equations. text-only math is much stronger.
- grok-4.3 posted 21 unparseable responses (reasoning loops that never emit an answer). on a real test, blank bubbles score zero, so that’s what it gets. models increasingly fail by not answering, not by being wrong.
- the whole top-10 is one statistical band. anyone ranking models on 20-point differences is selling you something.
the contamination asterisk (with data)
these are public practice tests. every model has seen them in training. instead of hand-waving, i measured it two ways:
1. the no-passage probe. 8 models got RW questions with the passage deleted — just the question and four options. chance is 25%:
| model | accuracy without the passage |
|---|---|
| gemini-3.1-pro | 72.1% |
| grok-4.5 | 62.3% |
| deepseek-v4-pro | 52.5% |
| gpt-5.5 | 50.8% |
| claude-opus-4.8 | 50.8% |
| gpt-4.1 | 49.2% |
| qwen3.5-397b | 49.2% |
| claude-haiku-4.5 | 29.5% |
some of that is legit (distractors are often implausible alone), but 72% without the passage is recognition, not reading. interestingly haiku, the small model, sits near chance — contamination capacity seems to scale with model size.
2. the AGIEval anchor. the old-SAT AGIEval set has been in training corpora since 2023 — maximally burned in. the same models average 97-99% on it, saturated flat. the digital-SAT tests still show real spread, so they’re less contaminated — but “less” is not “not”. read the leaderboard as an upper bound.
robustness checks
- shuffled answer options: re-lettering every MCQ moved scores -60 to +90 — noise, no position memorization at the letter level
- direct vs CoT: forcing answer-only costs most models 20-60 points, almost all in math. reasoning tokens are load-bearing for arithmetic, decorative for reading
- two test forms: pt10 and pt11 agree within each model’s CI, so this isn’t single-form luck
part 2: ok but would they get in anywhere
every model on this list beats ~19 out of 20 human test-takers. so… ivy league material, right?
no. here’s the bell curve, and then the uncomfortable math.
hover any dot to see where it sits, and toggle a school to overlay the mid-50% band of its actual enrolled students.
the purple curve is the actual SAT-taker score distribution (normal fit to College Board’s official percentile table: mean 1041, sd ~223). the models all live in the right tail — impressive — but look where the MIT line is. every single model is to the left of it.
the admissions math
using each school’s enrolled-student SAT mid-50% range (25th-75th percentile of admitted students who submitted scores, from IPEDS federal data) plus their actual admit rates:
| School | Admit rate | SAT mid-50% (composite) | best model (1485) lands… |
|---|---|---|---|
| Harvard | 3.5% | 1490–1580 | below the 25th percentile |
| Stanford | 3.9% | 1500–1580 | below the 25th |
| Columbia | 4.2% | 1490–1580 | below the 25th |
| Princeton | 4.5% | 1490–1580 | below the 25th |
| Yale | 4.5% | 1500–1580 | below the 25th |
| MIT | 4.8% | 1530–1580 | 45 points below the 25th |
| UPenn | 5.9% | 1500–1570 | below the 25th |
| Duke | 6.8% | 1490–1570 | below the 25th |
| NYU | 9.4% | 1470–1570 | barely inside the mid-50 |
| Georgia Tech | 16.5% | 1370–1550 | comfortably mid-range |
| U Michigan | 17.9% | 1350–1530 | upper half |
| Ohio State | 50.8% | 1310–1480 | above the 75th percentile |
| Alabama | 75.8% | 1130–1410 | above the 75th — presidential scholarship territory |
so what’s the actual likelihood of acceptance? being honest about what a score can and can’t tell you:
- T10 schools (Harvard through Duke): the best AI model scores below the 25th percentile of students these schools actually enroll. when your strongest quantifiable credential is below the bottom quartile at a school admitting 3-7% of applicants, your realistic odds are low single digits at best — the score isn’t disqualifying, but it’s a headwind, and everything would have to come from essays and extracurriculars. (the models’ extracurriculars are “being a data center,” which admissions officers reportedly find one-note.)
- MIT specifically: mid-50 floor of 1530 with a math floor of 780 among admits — and the best AI math section is 685. MIT would see a lopsided transcript: perfect verbal, visibly weak math-with-figures. for a school that is math, that’s the worst possible shape. effectively 0-2%.
- NYU / Georgia Tech / Michigan tier: scores are genuinely competitive here — inside or above the mid-50. likelihood roughly tracks or beats the base admit rate: ~10-25%.
- Ohio State / Alabama tier: above the 75th percentile of enrolled students. near-lock admits, likely with merit money. gemini-3.1-pro is getting a full ride to tuscaloosa.
two honest caveats: (1) mid-50 ranges are for students who submitted scores in the test-optional era, which inflates them — the true bar is a bit lower than these numbers suggest; (2) holistic admissions means scores are maybe a third of the decision. but the direction is unambiguous:
AGI is a 97th-percentile test-taker that still doesn’t clear the flagship-school scoreline. above average is not ivy league — the last 3 percentiles are worth more than the first 97.
and the reason it doesn’t clear the line is specific and kind of funny: it can’t reliably read a graph. the exact skill the SAT’s math section leans on hardest is the exact place every frontier stack is weakest. a 17-year-old with a 1530 beats a trillion-parameter model because she can look at a scatterplot.
what did we learn
- ☑️ frontier models score 1440-1485 — 95th-97th percentile of real SAT takers
- ☑️ reading & writing is saturated (four 800s); figure-based math is the open frontier
- ☑️ Chinese models match the frontier on text — DeepSeek at 93.1% is statistically tied with gemini/opus — but lack of vision keeps them off the full test
- ☑️ contamination is real and measurable: 50-72% accuracy on questions with the passage deleted
- ☑️ models fail by not answering (reasoning loops, refusals) as much as by being wrong
- ☑️ no model would clear the score bar at a T10 school. Ohio State says welcome.
the meta-lesson: benchmark plumbing is the whole game. answer parsing, error handling, official score tables, uncertainty, contamination probes — skip any one of these and your leaderboard is fan fiction. demand raw logs from every eval you read.
methodology notes for the nerds
the datasets are College Board digital SAT practice tests #10 and #11 — 120 questions each, extracted from the official PDFs with a layout-aware parser. answer keys got checked against two independent sources per test (the scoring PDF and the per-question rationale text). where they disagreed, i went and read the question myself.
elicitation is deliberately boring. chain-of-thought, one question per request, fresh context every time, and the model has to end with Answer: X on its own line. the parser also accepts \boxed{} and “the final answer is X”. nothing else. a response with no parseable answer gets recorded as unparseable and scored like a blank bubble — i never fish a letter out of the reasoning, because that’s the exact bug class that produces fake leaderboards.
scoring runs raw section counts through the official conversion table for that specific form (the scale is nonlinear; section floor is 200). confidence intervals come from an item bootstrap, 2,000 resamples through the conversion table. there’s also a coverage gate: no scaled score unless ≥98% of a model’s units returned real responses, and infra errors get retried until the sheet is clean, so nothing on the board is error-contaminated.
the 34 figure questions ship as actual images. text-only models get the text subset, not a fake total. ablations: direct-answer condition, seeded option shuffles, the no-passage probe, and the 426-question AGIEval anchor. all told: ~14,000 graded responses, about $50 in API costs.
the harness, prompts, raw responses, and scoring pipeline live in benchmark/ in the repo. the copyrighted questions aren’t committed — manifests + hashes are, so you can rebuild the exact dataset from the public PDFs and check every number yourself.
originally published january 17, 2026. substantially updated july 2026: harness rebuilt end-to-end (full test incl. math + figures, official conversion tables, bootstrap CIs, contamination probes, robustness checks), all numbers re-measured, and part 2 (bell curve + admissions analysis) added. AI models were harmed in the making of this update — grok-4.3 is still in a reasoning loop somewhere.