Mercor: Only 208 of 923 Occupations Have Real AI Benchmark Coverage

⏱️ Reading time: 10 min

A chatbot can solve almost any programming question in seconds, but no one knows if one does the real job of an electrician, a nurse, or a safety inspector well. According to a report by Mercor, only 208 of the 923 US occupations have a task measured by AI benchmarks with five or more tests.

📑 En este artículo

1. TL;DR
2. What Mercor’s Benchmark Map Is
3. What Happened
4. Context and Background
5. How Mercor Measures AI Benchmarks
6. Impact and Analysis
7. What’s Next
8. Frequently Asked Questions
   1. What exactly does Mercor’s Benchmark Map measure?
   2. How many of the 923 US occupations are well covered?
   3. Why are legal occupations better evaluated than administrative ones?
   4. What is Eloundou et al.’s AI exposure?
   5. Will Mercor keep updating this benchmark index?
9. References

The aggregate figure is more striking: 90% of the country’s paid work hours fall into tasks that current AI tests barely touch.

TL;DR

• Mercor finds that only 208 of 923 US occupations have AI benchmarks covering 5 or more tasks.
• 90% of the country’s work hours fall into tasks with little or no AI test coverage.
• Twenty-nine benchmarks map 18,797 O*NET tasks grouped into 22 job families, according to the study.
• Six of 22 job families, including healthcare and education, don’t have a single well-covered occupation.
• Coverage tracks salary: Computer and Mathematical occupations reach 29 of 36 measured occupations.

What Mercor’s Benchmark Map Is

Mercor’s Benchmark Map is a public index that cross-references the 923 occupations and 18,797 tasks in the US Department of Labor’s O*NET taxonomy with 29 existing AI benchmarks, to measure what portion of the country’s real work has at least one test evaluating it.

Mercor is known in the industry for running a marketplace that connects experts from different professions with AI labs to generate training and evaluation data. Its own benchmark, APEX, is part of the 29 included in this index, with the advantage that the company has access to prompts not yet published and can map them directly to each O*NET task.

What Happened

The report, published by Mercor and available as an interactive map, cross-references the O*NET taxonomy with 29 AI model evaluation benchmarks. According to the study, of the 923 occupations that make up the US workforce, only 208 have at least one task with five or more prompts from some benchmark mapped to it.

The aggregate figure confirms the problem: nine out of every ten paid work hours in the country correspond to tasks with no benchmark at all or with minimal coverage, according to the same report. Mercor grouped the 923 occupations into 22 job families following O*NET’s official classification, and found that 6 of those 22 families, including healthcare and education, don’t have a single occupation with real coverage.

The map divides those 923 occupations into 22 families that follow O*NET’s standard classification, from Computer and Mathematical to Healthcare Practitioners. Each occupation appears as a cell colored according to the percentage of its tasks with coverage, which makes it possible to see at a glance which sectors concentrate almost all the attention of current AI tests and which remain practically in the dark.

Context and Background

Benchmarks that measure how well a model reasons or codes have multiplied in recent years, but almost all of them were built to measure academic or software engineering capabilities, not the work that sustains the rest of the economy. Mercor says this Benchmark Map is just its first step toward a resource it plans to update continuously.

The idea of measuring how exposed an occupation is to AI isn’t new. In 2023, researchers Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock published a study that labeled thousands of O*NET tasks according to how much a language model could accelerate them. Mercor takes those binary exposure labels and cross-references them with its own coverage map to see whether AI tests actually concentrate where exposure is highest.

The report itself acknowledges that the saturation of AI tests in software engineering isn’t accidental: that’s the kind of task where it’s easiest to automate evaluation, because there are tests, clear success metrics, and public repositories, and where AI labs have a direct interest in measuring their own models’ progress. That’s why Mercor imposed a cap on benchmarks per job family when building its index of 29 tests, so that software engineering and academic research wouldn’t dominate the picture the way they do in previous catalogs.

How Mercor Measures AI Benchmarks

The mechanism behind the map is an automated judge. Mercor uses a language model to read each prompt from the 29 benchmarks evaluated and assign it one or more tasks from O*NET’s list of 18,797; the same prompt can map to several different tasks if it evaluates overlapping skills. An O*NET task counts as covered only if the judge assigned it five or more prompts across all benchmarks combined; a single isolated mention isn’t enough.

To confirm that the automated labeling wasn’t inventing matches, the team compared the judge’s labels with those of human experts on APEX, Mercor’s own test bank, where the company has access to prompts not yet published. That validation is what allows the claim that the measured coverage isn’t statistical noise, but a reasonable approximation of what each test measures.

The 29 benchmarks in the index were chosen using five criteria:

• Public availability: at least 25 public tasks, or direct access via APEX in Mercor’s case.
• Agentic tasks: asking the model to execute a task, not just answer questions about it.
• Economic realism: tasks similar to what a real company would ask of an employee, not academic puzzles.
• Economic diversity: a cap on benchmarks per job family so software and research don’t dominate the index.
• Balanced difficulty: benchmarks that frontier models haven’t saturated yet.

To understand why some occupations have more coverage than others, Mercor ran a task-level logistic regression: the probability of coverage is explained by Eloundou et al.’s AI exposure label and the logarithm of the occupation’s wage bill, calculated by multiplying the mean wage reported by the Bureau of Labor Statistics by the number of workers and a 40-hour week. Standard errors are clustered by occupational family, and the 95% confidence intervals use the delta method.

The result of that regression is a curve, not a simple table: as an occupation’s wage bill rises, the estimated probability that one of its tasks is covered by some benchmark rises consistently, and the gap between high- and low-AI-exposure tasks stays nearly constant across the entire wage range. Being an automatable task according to Eloundou et al. helps, but it helps much less than belonging to a well-paid occupation.

flowchart TD
    A["O*NET Taxonomy: 923 occupations, 18797 tasks"] --> B["LLM judge labels each benchmark prompt with its O*NET tasks"]
    B --> C["Validation against human labels on APEX"]
    C --> D{"Task with 5 or more prompts mapped?"}
    D -->|"Yes"| E["Task covered"]
    D -->|"No"| F["Task not covered"]

Impact and Analysis

The report’s most uncomfortable finding is the pattern behind the total number of covered occupations, more than the number itself. AI benchmarks aren’t distributed randomly across the 923 occupations: the probability that a task has coverage rises with the wage bill of the occupation it belongs to, and it rises even more if that task is already labeled as exposed to AI. The example Mercor gives is stark: lawyers’ tasks, a high-exposure, high-salary occupation, are widely covered, but equally exposed tasks in occupations like legal secretaries or court clerks barely appear in any benchmark.

The table shows only 12 of the 22 families so as not to dilute the pattern, but the trend holds across the rest: the larger a job family’s aggregate payroll, the more likely some benchmark measures it, even when lower-paid occupations share the same level of AI exposure according to Eloundou et al.’s classification.

The report calls this the long tail of middle-income work: occupations like administrative assistants, lab technicians, or project coordinators rarely show up in a benchmarks paper, even though they add up to millions of hours worked per year. Without direct measurement of those tasks, any claim about how good AI is at real work rests on a sample skewed toward code and finance.

💭 Key takeaway: a task’s AI exposure matters less than the occupation’s salary when it comes to deciding whether anyone bothered to measure it.

What’s Next

Mercor says this map is a living resource, not a closed report: it plans to add more tests to the index as benchmarks with greater economic realism appear, and to expand coverage toward the job families that currently sit at zero, such as healthcare and education. The stated goal is for anyone to be able to check, occupation by occupation, how well evaluated AI is against the work that person does every day.

If you evaluate AI models or design benchmarks, open Mercor’s interactive map and filter by your own industry’s job family to see how many of its tasks have any real test behind them.

Frequently Asked Questions

What exactly does Mercor’s Benchmark Map measure?

It measures what portion of the 18,797 tasks O*NET uses to describe work in the United States has at least one AI model test covering it with five or more prompts, grouping the result by the country’s 923 occupations and 22 job families.

How many of the 923 US occupations are well covered?

Only 208 have a task with five or more prompts mapped by Mercor’s index; the rest, about 715 occupations, don’t reach that threshold or have no associated test at all.

Why are legal occupations better evaluated than administrative ones?

Because the Benchmark Map’s coverage tracks the occupation’s wage bill more closely than its real AI exposure: lawyers, with high salaries, are well covered, while legal secretaries and court clerks, with similar exposure according to Eloundou et al., are left with almost no tests.

What is Eloundou et al.’s AI exposure?

It’s a binary label that 2023 study assigned to thousands of O*NET tasks based on how much a large language model could accelerate them; Mercor uses it as a control variable to separate the effect of salary from the effect of real exposure.

Will Mercor keep updating this benchmark index?

Yes: the company describes it as a first step toward a resource it plans to update continuously, adding benchmarks and occupations as tests with sufficient economic realism appear.

References

• Mercor Benchmark Map: the original report with the interactive coverage map by occupation.
• O*NET Resource Center: the official US Department of Labor taxonomy with the 923 occupations and their tasks.
• Eloundou, Manning, Mishkin, and Rock (2023): the study that labeled each O*NET task’s exposure to language models.
• Bureau of Labor Statistics, Occupational Employment and Wage Statistics: the source of the wages used to calculate each occupation’s wage bill.

📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Featured image: Foto de Chris Liverani en Unsplash