Mercor maps AI benchmarks against US jobs and finds most working hours barely covered

AIAI #benchmarks #evaluation #jobs #labor #research

Mercor maps AI benchmarks against US jobs and finds most working hours barely covered

The interactive map lets you click into any job family and see which tasks AI tests actually measure.

Mercor, which sells AI evaluation services, mapped 29 AI benchmarks onto the US Labor Department's O*NET list of 923 occupations and 18,797 tasks. Only 208 occupations have a task covered by at least five benchmark prompts, 6 of 22 job families have none, and Mercor estimates 90% of American working hours go to tasks with little or no benchmark coverage, making it hard to know how well AI does the work people actually do.

Coverage is lopsided: computer and math (29 of 36 occupations) and business and finance (33 of 48) fare well, while healthcare practitioners (12 of 89) and education (9 of 62) lag. Mercor says benchmarks favor the largest labor markets; for example, they heavily cover lawyers' tasks but barely touch those of similarly exposed legal secretaries and court clerks.

Read it at the source

MercorA benchmark map of the US economy Independent ↗ Hacker NewsDiscussion: 90% of US working hours are spent on tasks not covered well by AI benchmarks Independent ↗

Cascadic Analysis 2

UndertowBenchmark scores can overstate real-world reliability

What hidden risk could pull against this, even if the news is good?

Mercor's map says 90% of US working hours go to tasks with little or no benchmark coverage, so a model's headline scores say little about most real work. Mercor sells AI evaluation services, which gives it a stake in the conclusion that more evaluation is needed.

UndertowRole underspecification / the Doorman Problem

What hidden risk could pull against this, even if the news is good?

Mercor's example: benchmarks heavily cover lawyers' tasks but barely touch those of similarly exposed legal secretaries and court clerks. If those jobs are automated on the strength of lawyer-focused scores, no one will have measured how AI handles the exceptions and judgment calls they actually involve.

Rabbit Holes 1

Occupational Information Network

After exploring A benchmark map of the US economy, the job database underneath the map is the Occupational Information Network.

Describes O*NET, the free US database of job definitions and tasks that the map is built on.

Wikipedia · Wade · 5 min

0

Mercor maps AI benchmarks against US jobs and finds most working hours barely covered

Further down the stream