Algorithmic Hiring: How Bias Gets In and What to Ask a Vendor

Laura applies for four jobs in the same month. The four employers have nothing to do with each other. But all four screen applicants with online games from the same vendor, and all four turn her down within a day. Most likely no person ever reads her application. Laura assumes she just wasn’t the right fit four times in a row, because she has no way of knowing that one vendor’s software was behind all four answers.

Laura is a composite made up for this guide. The pattern behind her story is real, though. A Stanford-led study of more than four million job applications looked at people who applied to four positions screened by the same vendor’s algorithms. It found that 10% of them were marked “do not recommend” for all four. That changes the question an HR leader has to ask, because it isn’t only “is this tool fair in our hiring?” It’s also “are we one of four employers saying no to the same person for the same hidden reason?”

What follows is a plain-language walk-through of how bias gets into hiring software and what the evidence says. It also covers what the law asks of you, and the questions we’d ask a vendor before buying or building a screening tool.

What algorithmic hiring is, stage by stage

Algorithmic hiring means using software to make or shape decisions about who moves forward for a job. An algorithm is just a set of rules the software follows. Some tools follow rules a person wrote, such as “screen out anyone without a degree,” while others learn their rules from past data. Those learned rules are the ones to watch, because nobody wrote them down and nobody can easily read them.

Software can act at almost every stage of hiring. The table shows where it usually sits and where bias can slip in. The last column is our judgment of where the risk sits, not a finding from a single study.

Hiring stage	What the software does	Where bias can enter
Sourcing	Finds people to contact, or decides who sees a job ad	It looks for more people like the ones you hired before
Resume screen	Scores or filters resumes against the job	Schools, gaps, hobbies or word choice can stand in for gender, race or age
Assessment or game	Scores tests or puzzles against a profile of current staff	If the “good” profile came from a narrow group, people unlike that group score lower
Video or voice interview	Scores spoken answers and how they’re delivered	Accent, speech patterns and disability can move the score
Final ranking	Combines all the scores into a shortlist	Small gaps at each step add up, and the shortlist is all a recruiter sees

The point of the table is that a tool doesn’t need to be told to discriminate, since in most cases nobody types “prefer men” into anything. Instead, bias gets in through the data the tool learned from. It also gets in through the signals the tool decides are worth paying attention to.

How bias gets into a hiring algorithm

The easiest way to see this is with an example. Brookings used a hypothetical one in its 2019 paper on bias in algorithmic hiring, and it’s worth walking through.

Say a company trains a resume model on the people it hired and rated highly over the last ten years. The model goes looking for anything those people have in common, and one thing it notices is that many of them played lacrosse. So the model starts giving resumes that mention lacrosse a higher score. Nobody asked it to care about sport, but it found a pattern and used it.

The trouble is that lacrosse isn’t spread evenly across society. As Brookings puts it, the sport may tend to be played by affluent white men. So “played lacrosse” quietly tracks wealth, race and gender, and the model now favors one kind of applicant without ever being told who they are. A signal that stands in for something else like this is called a proxy variable, or just a proxy.

Why removing gender and race fields doesn’t fix it

The obvious fix is to hide the sensitive fields, since a tool can’t use what it can’t see. That sounds right, but it misses how proxies work. The tool was never using the gender field in the first place. It was using everything else on the resume, such as hobbies, schools and ZIP codes. It was also using gaps in work history and the words people chose. Those stay on the resume after you delete the box marked “gender.” So if they track gender or race in your past hiring data, the tool will still learn the same pattern.

That’s why “we don’t collect demographic data” is not an answer to the bias question. It may even make the problem harder to spot. If nobody records who applied, nobody can check later whether one group was turned down more often.

How a 95% accurate tool can still fail a whole group

Vendors often quote one accuracy figure for the whole tool. Brookings showed why that number can hide a lot, and the arithmetic is simple. Say 5% of your applicants come from one small group. A tool could get every other applicant right and every person in that group wrong, and it would still be 95% accurate overall.

So a single headline number tells you how the tool does on average. It tells you nothing about how it does for the people most likely to be shut out. You need results broken down by group to see that.

What the largest study found

The study behind Laura’s story is called “Algorithmic Monocultures in Hiring”, and it was published in May 2026. Rishi Bommasani of Stanford led the work with Sarah Bana, Kathleen Creel, Dan Jurafsky and Percy Liang. The team looked at one vendor that sells game-based assessments to employers. That vendor is pymetrics, which is now part of Harver. Applicants play a set of short online games, and each employer gets its own model trained on how its current staff played. All the figures below come from the paper itself.

The size of the data set is what makes it stand out. It covers December 2018 to December 2022, which is 4,197,168 applications from 3,372,132 people. They applied to 1,746 positions at 156 employers across 11 industries. Most earlier writing on this topic relies on one famous story or on theory, at least in our reading. This study is different because it rests on four years of real hiring decisions.

The four-fifths rule, and why the study checked each job

The researchers tested for unfair impact with a long-standing US rule of thumb called the four-fifths rule. It comes from the federal Uniform Guidelines on Employee Selection Procedures, and it compares how often each group gets through a step. If one group’s pass rate is less than four-fifths (80%) of the top group’s rate, that’s a warning sign of what lawyers call adverse impact. The guidelines also say a smaller gap can still count, so passing the test isn’t a safe harbor.

The Institute for the Future of Work gives a simple example. Say 60% of the men who apply get invited to interview. Four-fifths of 60% is 48%. So if fewer than 48% of the women who apply are invited, the tool is flagged.

The researchers applied this test one job at a time. That choice matters, because when they pooled all the jobs together the tool looked fine. The pooled pass rate was 58.3% for white applicants and 56.8% for Hispanic applicants. It was 53.3% for Asian applicants and 52.5% for Black applicants. Every group cleared the 80% line on those totals. The paper explains the problem with an analogy. If only men were recommended for doctor jobs and only women for nurse jobs, the totals could still look balanced. But nobody applies to the totals. Each person applies to one job at a time, so each job is where the fairness question lives.

What it means for one applicant

The picture changed when the researchers checked job by job. About one in ten positions (10.62%) put Black applicants at a disadvantage, and about one in twenty (5.32%) did the same to Asian applicants. Those positions drew 25.87% of all applications from Black applicants and 14.74% of all applications from Asian applicants. If Black and Asian applicants to those positions had been recommended at the same rate as the most selected group, about 40,800 more of their applications would have been recommended. That’s 11,513 applications from Black applicants and 29,320 from Asian applicants.

The title also mentions a “monoculture,” which means many employers relying on algorithms from the same vendor. If one hiring manager doesn’t like an applicant’s resume, the next one might. But a pymetrics applicant’s game results are stored and reused for any other pymetrics position they apply to within 330 days. Each employer’s model weighs those same results, and similar weightings lead to similar answers. That’s how 10% of people who applied to four of these positions ended up marked “do not recommend” by all four. The paper notes that a “do not recommend” usually means the applicant is rejected without a person looking at them.

The study found one more thing that matters for buyers. The games don’t directly capture demographic information, and the vendor aims to remove bias from each model it builds. Adverse impact showed up anyway. The researchers place this alongside earlier work on proxy discrimination, which is the same kind of problem as the lacrosse example. They don’t name which game signal acts as the proxy, though.

What is settled and what is still argued about

Some points have firm support. Proxies are real, and they explain why removing sensitive fields doesn’t remove the effect. Pooled results can hide problems that show up job by job, and a single accuracy number can also hide failure for a small group. These follow from the arithmetic and from the Stanford data together.

Other points are less settled. The biggest open question is how common this kind of bias is across all hiring tools. The Stanford study looked at one vendor’s tool. It’s a large sample, but it isn’t every product on the market. There’s also a fair question about the comparison, because human screeners have their own biases. In our judgment the useful question is whether the tool does better or worse than your current process for each group, and whether you can prove it.

There is also a counter-view. A 2025 piece from Purdue’s Daniels School of Business asks whether algorithmic bias in hiring is fact or myth. It says a review of the 40 most-cited of about 100 articles found very little evidence that bias is a widespread problem, though it could be. It also tells readers not to assume bias is a fact, and not to assume it’s a myth either. The piece doesn’t link to that review or name it, so it’s hard to check. It’s one view and it’s worth knowing about. But set beside four million real applications checked job by job, it’s the weaker evidence.

The vendor has a view too. According to The Register’s report on the study, pymetrics researchers published a paper in 2022 which found their algorithm would not run afoul of EEOC standards. The EEOC is the US agency that enforces laws against job discrimination. Harver didn’t immediately respond to The Register’s request for comment on the Stanford findings. The Stanford paper explains the gap. The vendor’s own study reported only results pooled across all applications, and pooled results are exactly where the per-job problems disappear.

Vendor claims in general are hard to compare. A team led by Manish Raghavan studied what 18 vendors of pre-employment assessments say publicly about bias. What each vendor says it tests for varies a lot, and so does how it tests. That’s a good reason to ask the questions further down rather than rely on the marketing page.

Is algorithmic hiring legal? What the law asks of employers

Using software to screen candidates is legal in most places, but the kind of outcome the Stanford study describes can break the law. US anti-discrimination law covers the result of a hiring process, not only the intent behind it. If your tool turns down one group far more often and you can’t justify it, the fact that a vendor built it doesn’t move the problem off your desk. This is general background and not legal advice. Check the detail with your own counsel.

Two sets of rules go further and are worth knowing by name.

New York City Local Law 144. If you use an automated employment decision tool for jobs in New York City, the city’s Department of Consumer and Worker Protection says the tool needs an independent bias audit within one year before you use it. You also have to make a summary of the audit results public, and you have to tell candidates that the tool is being used. We cover the detail in our guide to NYC Local Law 144. The Stanford paper adds a warning here. The city’s example of how to work out impact ratios appears to merge all the data together, which blurs the line between positions and even between employers.

The EU AI Act. The Act treats AI used to recruit or select candidates as high-risk, under its Annex III. High-risk systems carry extra duties for the companies that build and use them, but the dates when those duties apply have been under revision. So check the current text on EUR-Lex rather than relying on a date you read in a blog, including this one.

Both laws point the same way. They expect you to know what your tool does to each group and to be able to show it. If you’re putting governance around AI in HR more broadly, our post on AI governance in HR is a good next read.

Questions to ask a vendor before you buy (or a builder before you build)

We build HR agents, so we sit on the builder’s side of this conversation. These are the questions we’d want answered before any screening tool touched a real applicant. They work just as well for a vendor’s sales team as for your own build team.

1. What exactly does your bias audit measure? Ask which test was used, which groups were checked and who ran it. “We audit for bias” without detail tells you very little.
2. Are results reported per job and per group, or pooled? The Stanford study shows why pooled results can look clean when single jobs are not.
3. What inputs does the tool use? Ask for the list, and then ask which of those inputs could act as a proxy for gender or race. Ask the same about age and disability.
4. What was the tool trained on? If it learned from past hires, ask whose past hires. A profile built from a narrow group will favor that group.
5. How many other employers use the same vendor’s models? This is the monoculture question. If many employers use them, the applicants you reject may also be getting rejected elsewhere for the same reason.
6. Who sees the rejections? Find out whether a person ever looks at the applicants the tool turns down, or only at the ones it passes.
7. What does a human actually review, and when? “Human in the loop” can mean anything from reading every decision to glancing at a dashboard once a quarter.
8. Can we run our own check? Ask whether you can get the pass rates by group for your own jobs, so you aren’t relying only on the vendor’s numbers.

My advice is to give question six the most weight. A tool that only shows recruiters the people it passed makes its mistakes invisible, because nobody can spot a pattern in applications they never see.

Our 10X Framework has a lens for this called “Trust to earn.” It asks how much trust a use case has to earn before people can rely on it. Resume screening has a lot of trust to earn, because it shapes a decision about a named person. In our judgment it also has more to earn because the people it affects never see the tool and can’t push back on it. The framework’s rule is that AI may rank or summarize, but a person decides. That doesn’t mean you shouldn’t use a screening tool, but it does mean the questions above have to be answered first. It also helps to know what your own hiring data looks like before anyone trains on it, which our HR data readiness post walks through.

Before you sign, talk it through

If a screening tool is on your desk right now, the questions above are a good place to start. If you’d rather work through them with someone who builds these systems, book a 30-minute call. We’ll look at what the tool does, where the risk sits for your roles and what you’d need to check before it goes live.

You can also see where screening fits among other hiring use cases on the HR Agent Map’s talent acquisition page. The full HR Agent Map shows every area of HR where we think agents can help, and how much trust each one has to earn.

FAQ

Does algorithmic hiring reduce bias?

It can, but it doesn’t happen on its own. A tool applies the same rules to everyone, which removes some of the mood and gut feel of human screening. But if those rules were learned from biased past hiring, the tool repeats that bias at scale. The Stanford study found adverse impact even in a tool whose games don’t directly capture demographic information, from a vendor that aims to remove bias from every model.

Is algorithmic hiring legal?

Yes in most places, but you’re still responsible for the outcome. US anti-discrimination law covers the results of your hiring process. New York City requires an independent bias audit within one year before these tools are used. The EU AI Act treats recruitment AI as high-risk.

What is the four-fifths rule?

It’s a rule of thumb for spotting unfair impact. If one group passes a hiring step at less than 80% of the rate of the top group, that’s a warning sign. If 60% of men are invited to interview, fewer than 48% of women being invited would be flagged.

Will removing names and demographic fields from resumes fix bias?

Not by itself, because other details on a resume can stand in for the fields you removed. Schools, hobbies and ZIP codes can all do this. So can gaps in work history. A tool can still learn the same pattern from those proxies.