AI Models Are Getting Smarter. New Tests Are Racing to Catch Up

AI developers don't always know what their most advanced systems are capable of.

To find out, systems are subjected to a range of tests designed to tease out their limits.

But due to rapid progress in the field, today ’s systems regularly achieve top scores on many popular tests, including SATs and the U.S. bar exam.

A new set of much more challenging evals has emerged in response, created by companies, nonprofits, and governments.

Epoch AI’s FrontierMath benchmark consists of approximately 300 original math problems, spanning most major branches of the subject.

Half the problems require “graduate level education in math” to solve, while the most challenging 25% of problems come from “the frontier of research of that specific topic” The benchmark is intended to go live in late 2024 or early 2025 .

A third benchmark to watch is designed to simulate real-world machine-learning work.

AI systems ace many existing tests, but they continue to struggle with tasks that would be simple for humans.

Future models that excel on this benchmark may be able to improve upon themselves, exacerbating human researchers’ lack of control over them.

The U.S. and U.K. AI Safety Institutes have begun evaluating cutting-edge models before they are deployed.