01
Standardize
An AI agent assembles the paper, experiment code, stimuli, and released data into an initial EML specification.
About CogGym
Have we built models that think like humans?
AI systems can now solve problems that once seemed out of reach. But high benchmark scores do not tell us whether their judgments, uncertainties, and characteristic mistakes resemble those of people. Measuring human–model alignment asks a complementary question: not only can a model produce the right answer, but does it interpret and respond to the world in human-like ways?
CogGym makes that question measurable. It converts the experimental record of cognitive science into matched evaluations for people and computational models, allowing us to compare average judgments, full response distributions, and the domains in which human and machine cognition converge—or remain fundamentally different.
Why CogGym
A fragmented picture of human–AI alignment
Across cognitive science and AI, studies offer different answers to the question of whether models behave like humans. Some find striking similarities in judgments and errors; others reveal substantial differences. But these findings are difficult to compare because they come from different tasks, models, prompting procedures, response measures, and analysis choices. The result is a fragmented literature rather than a cumulative account of where human and machine cognition align.
A central challenge is that behavioral experiments are diverse and non-standardized. They span text, images, and video; use tools such as Qualtrics, jsPsych, PsychoPy, MATLAB, and custom websites; and encode their instructions, stimuli, randomization, and response scales in incompatible ways. CogGym brings these experiments into a shared representation and evaluation pipeline, making it possible to compare human and model behavior systematically across many domains.
A shared experimental language
EML makes experiments portable across humans and models
The Experiment Markup Language (EML) represents a behavioral experiment as model-agnostic, executable JSON rather than framework-specific code. It records the experimental configuration and randomization, participant instructions, trials, stimuli, response widgets, and human data in a shared structure.
From that single specification, CogGym can render an interactive study for human participants and construct a matched prompt for a model. Both routes preserve the same trial identifiers, response options, and query tags, allowing human and model responses to be aligned directly without bespoke scoring code for every study.
The agentic pipeline
AI accelerates conversion; researchers establish fidelity
CogGym uses an AI agent to produce an initial standardized experiment from the paper, supplementary materials, original code or survey instruments, stimuli, and released data. The platform renders that draft so a researcher can compare it side by side with the original task. The researcher checks the instructions, condition logic, stimuli, trial order, and response interface, then sends discrepancies back to the agent for revision. This loop continues until the researcher judges the conversion faithful; a typical experiment takes about one hour to standardize.
Once validated, the experiment can be rerun with new human participants, evaluated on current or future models, and analyzed against the original human data. The result is living scientific infrastructure rather than a one-time static benchmark.
02
Review and refine
Researchers compare the rendered task with the source experiment and direct revisions until the implementation is faithful.
03
Run and compare
The same EML is administered to people and models, producing responses with shared trial IDs and scoring fields.
Current Paper Snapshot
The current CogGym paper release spans a broad experimental surface, bringing together papers, labs, standardized experiments, and large-scale human judgment data in one comparison framework.
- 100
- Papers
- 30+
- Labs
- 258
- Experiments
- 57K+
- Human participants*
- 2.7M
- Human judgments
Results at a glance
What CogGym reveals about human and machine cognition
Most AI benchmarks ask whether a model can produce the correct answer. CogGym asks a complementary question: when people disagree, make systematic mistakes, or express graded judgments, does a model respond in the same way? Across 258 experiments, the answer is increasingly yes—but not yet at the level of human reliability.
We use two complementary metrics. Model–human R² measures agreement in average responses across experimental conditions, where higher is better. Normalized distributional divergence compares the full response distributions, where lower is better. Human split-half reliability shows how closely independent groups of people reproduce one another and provides the natural reference for both measures.
Read the full paperFormal reasoning improves faster than human alignment
Newer models do increasingly reproduce human mean judgments and response distributions. The pace of that improvement, however, is modest compared with the sharp gains seen on GPQA Diamond, MATH, and HumanEval. A system can become much better at producing correct answers without becoming equally better at predicting how people judge uncertain, social, perceptual, or physical situations.
This distinction is visible in the remaining gap to the human reference. The strongest models reach average model–human R² values of 0.59 for text, 0.58 for image, and 0.43 for video, compared with human split-half reference values of 0.93, 0.95, and 0.92.
Larger open-weight models are more human-like
Scale helps. Across open-weight model families, model–human R² rises with parameter count in text, image, and video tasks, while normalized distributional divergence falls. The relationships are consistent across modalities: the R² correlations are +0.81, +0.75, and +0.81, respectively, and the divergence correlations are −0.83, −0.69, and −0.68.
But the fitted trends also show how much room remains. Extrapolating the observed scaling curves does not bring current model families close to the human reliability lines, especially for video. Bigger models are moving in the right direction, but parameter scaling alone has not closed the cognitive-alignment gap.
Most experiments remain below human consistency
Aggregate scores can hide large differences between experiments, so we also compare the strongest model with human split-half reliability one experiment at a time. For Gemini 3.1 Pro, 94% of experiments fall below the R² parity line: the model is less consistent with the original human pattern than a new group of humans would be.
The same conclusion appears when we compare full response distributions. Model responses generally diverge more from human responses than two human groups diverge from one another. This makes human reliability a useful ceiling rather than treating a perfect numerical score as the only meaningful target.
Alignment varies by topic and modality
The remaining errors are not distributed evenly. Gemini 3.1 Pro aligns relatively well with human judgments in several topic areas, while physical reasoning is substantially weaker when the evidence is presented through images or videos. Modality therefore changes more than the input format; it changes which forms of cognition the model must reproduce.
These differences point toward a more diagnostic use of CogGym. Instead of asking only which model has the highest overall score, we can ask which kinds of human reasoning transfer across modalities, where model behavior departs from people, and which improvements reflect genuine cognitive alignment rather than benchmark-specific competence.
Looking ahead
Toward a cumulative science of human and machine cognition
CogGym is designed to grow alongside both fields. As new models, post-training methods, and interaction paradigms emerge, the benchmark can incorporate new experiments and modalities, rerun matched human–model comparisons, and reveal whether advances generalize across cognitive domains.
The next phase is to move from a static leaderboard toward a living map of cognition: identifying where models converge with people, where they develop qualitatively different strategies, and which gaps matter for real-world collaboration. By making diverse experiments portable and comparable, CogGym can help cognitive scientists and AI researchers build on one another's evidence—and guide the development of systems that understand and work with people more effectively.