What 10,765 Agents Hired Revealed About Resume Keywords and Performance
Across 10,765 agents hired at a Fortune 500 insurance carrier, researchers tested whether resume keywords predict job performance. They parsed 8,181 unique skills from applicant tracking system profiles and tested the 3,597 that had enough data to measure. After Bonferroni correction, zero keywords predicted the production milestone. Thirty were anti-predictive.
Source: "Decision Traces," Saad Bin Shafiq, Nodes, 2026. N=10,765 agents hired 2022 to 2025 at a Fortune 500 insurance carrier, analyzed inside the carrier's VPC. Read it on arXiv.
The test: every keyword, against real outcomes
The dataset held 8,181 unique parsed skills. Researchers tested the 3,597 skills with enough data for analysis, then applied Bonferroni correction across those simultaneous tests.
The result was clean and one-directional. Zero keywords significantly predicted production. Thirty were significantly anti-predictive. Across all tested keywords, 70.2% had an odds ratio below 1, meaning listing them was associated with a lower chance of producing. The median odds ratio was 0.749.
Standard insurance filters did worse than no filter
The team then checked the six categories the industry actually screens on:
| Filter | Produced with it | Produced without it | Odds ratio | Direction |
|---|---|---|---|---|
| Insurance experience | 28.0% | 33.7% | 0.763 | Anti-predictive |
| License | 24.9% | 33.1% | 0.668 | Anti-predictive |
Both held up within individual hiring years, so they are not a cohort artifact. The two signals the industry treats as table stakes were associated with lower production.
The candidates a keyword filter throws away
The carrier had 677 agents with no traditional keywords who also fell in a high behavioral-score band. They produced at 33.7%, above the 26.3% rate for agents who had every keyword present. The keyword filter was selecting in the wrong direction. See what that cost in dollars.
What this does not say
This is one carrier and one role type, insurance sales. Another organization can repeat the decision-trace test, but its keyword results may differ. The finding is a reason to test your own keywords against your own outcomes.
Frequently asked questions
Do resume keywords predict job performance? In this study of 10,765 insurance hires, no. Of 3,597 testable keywords, none predicted production after correcting for multiple comparisons, and 30 were anti-predictive.
What is Bonferroni correction and why does it matter here? It is a conservative adjustment for testing many hypotheses at once. Applied to 3,597 keyword tests, it raises the threshold a result must clear. Zero keywords survived it.
Does having more skills on a resume help? Not in this study. Agents with 0 to 10 parsed skills produced at 36.1%, while agents with 51 or more produced at 22.7%.
Is this finding specific to insurance? The data comes from insurance agent hiring. The same test can be run where screening inputs can be connected to defined outcomes, but the result may differ by role and organization.
Related reading
- Why ATS keywords fail to predict performance
- The $17.7M cost of a single hiring filter
- Decision traces, explained
Want to know which of your screening rules are anti-predictive? Request a Decision Replay.