ARC-AGI-3: A New Ruler for Measuring the Gap Between AI and Human Intelligence

From ARC-AGI-1 to ARC-AGI-3: Evolving the Measure of General Intelligence

In 2019, François Chollet posed a deceptively simple challenge: can you solve a problem you've never seen before? His answer was ARC — the Abstraction and Reasoning Corpus — a set of visual puzzles requiring pattern induction from minimal examples, with no reliance on pre-trained knowledge. What seemed like a niche benchmark became one of AI's most stubborn obstacles.

What ARC-AGI-3 Actually Tests

In March 2026, the ARC Prize team released ARC-AGI-3, the third generation of this benchmark. The shift is fundamental: from solving static puzzles to learning and adapting inside dynamic environments.

ARC-AGI-1 tested visual rule inference without natural language prompts. ARC-AGI-2 tightened the difficulty further. ARC-AGI-3 requires an agent to enter a completely novel interactive environment — discovering goals on the fly, figuring out what actions matter, then completing tasks within a limited step budget.

The scoring is deliberately rigorous. A perfect 100 means completing every level with the efficiency of the second-best human solution. The formula is:

score = (human_steps / agent_steps)²

If a human takes 10 steps and your agent takes 100, the score is just 1%. The average human test-taker scores around 25–30%. This isn't a forgiving benchmark.

Why This Is Fundamentally Harder Than Puzzle-Solving

ARC-AGI-3 targets continual learning combined with world model construction — precisely where today's LLMs are weakest.

Zero-instruction environments: No natural language tells the agent what the rules are. It must infer the model from exploratory behavior. This severs the LLM's main crutch: language-based reasoning chains.

Sparse feedback: Actions don't yield immediate dense rewards. The agent must sustain long-horizon exploration before discovering whether a strategy works. Standard RL methods like PPO struggle significantly here.

Goal acquisition on the fly: The objective itself is hidden. The agent must first discover what it's trying to do, then do it — a fundamental departure from instruction-following architectures.

Long-horizon planning: Working memory must span multiple steps. The agent needs to distinguish productive action sequences from dead ends, requiring something closer to mental simulation than step-by-step inference.

Current State and Industry Implications

ARC-AGI-3 hit Hacker News front page within hours of launch, sparking heated technical debate. Critics noted the scoring system's asymmetry — using a squared efficiency penalty against the second-best human means even human-average AI performance translates to only 15–30 points.

That asymmetry is intentional. Chollet's position has been consistent: we don't want AI that's merely adequate. We want systems that acquire capabilities rapidly in unfamiliar contexts. ARC-AGI doesn't just measure performance — it measures learning efficiency.

Current frontier models are expected to score below 5% on ARC-AGI-3. Human novices reach 25%+ with minimal practice. This gap exposes a core architectural gap in LLMs: they can pattern-match brilliantly over training distributions, but the perceive-act-learn loop remains genuinely broken.

"As long as there is a gap between AI and human learning, we do not have AGI. ARC-AGI-3 makes that gap measurable." — François Chollet

SFD Lab Perspective

At SFD Lab, we've hit the same wall in practical agent development. Equipping an LLM with tool calls is straightforward. Getting it to genuinely model its environment and self-correct its strategy is not. Our agents consistently over-weight first-round inferences and show little appetite for active exploration when no template exists.

ARC-AGI-3 is essentially a stress test for closed-loop adaptability. For a team building real AI toolchains, this benchmark is a useful forcing function: when we say our agent is "smart enough," what are we actually measuring? Throughput? Instruction-following fluency? Or genuine adaptive intelligence?

We treat ARC-AGI-3 internally as a mirror — not a leaderboard target for tomorrow, but a design philosophy to audit our agents against: Can it discover its goal? Can it learn from failure? Can it rebuild a world model in a novel environment without leaning on prior training?

SFD Editor's Note: This article is based on the ARC Prize website, Hacker News discussion thread #47521150, and related technical analysis. SFD Lab is evaluating integrating ARC-AGI-style tests into our agent evaluation pipeline as the gold standard for "does it actually generalize."