Credit: Unsplash/CC0 Public Domain
When Kate Grazko, a graduate student at the University of Washington, was searching for a research internship last year, she noticed recruiters posting online that they had used OpenAI’s ChatGPT and other artificial intelligence tools to summarize resumes and rank applicants. Automated screening has been common in hiring for decades. But Grazko, a doctoral student at the Paul G. Allen School of Computer Science & Engineering at the University of Washington, is researching how generative AI can replicate and amplify real-world biases, such as prejudice against people with disabilities. She wondered how such a system could rank resumes that suggested someone was disabled.
In a new study, researchers from the University of Washington found that ChatGPT consistently ranked resumes with disability-related honors or certifications (such as the “Tom Wilson Disability Leadership Award”) lower than resumes without those honors or certifications. When asked to explain the rankings, the system spit out biased perceptions of people with disabilities. For example, it claimed that resumes with Autism Leadership Awards “are less focused on leadership roles,” alluding to the stereotype that autistic people are not good leaders.
But when the researchers customized the tool with written instructions to avoid being ableist, this bias was reduced for all but one of the disabilities they tested. Five of the six implied disabilities improved – hearing impairment, visual impairment, cerebral palsy, autism, and a general “disability” – but only three were ranked higher than resumes that didn’t mention a disability.
The research team presented their findings on June 5 at the 2024 ACM Conference on Fairness, Accountability, and Transparency in Rio de Janeiro.
“AI-based resume ranking is becoming more common, but there’s still not much research on whether it’s safe and effective,” said Grazko, the study’s lead author. “For job seekers with disabilities, the question always arises of whether they should include their disability qualifications when submitting a resume. We think people with disabilities would take that into consideration even if a human were reviewing it.”
The researchers used a roughly 10-page curriculum vitae (CV) published by one of the study authors. The team then created six enhanced CVs, each of which implied a different disability by including four disability-related qualifications: scholarships, awards, seats on diversity, equity, and inclusion (DEI) panels, and membership in a student organization.
The researchers then used ChatGPT’s GPT-4 model to rank these enhanced resumes against original versions of actual “student researcher” job advertisements from a large US-based software company. They ran each comparison 10 times. Over 60 trials, the system ranked the enhanced resumes (which were identical except for the implied disability) first in only a quarter of cases.
“In a fair world, a strong resume should always rank first,” said Jennifer Mankoff, a professor at the University of Washington’s Allen School of Business and lead author of the study. “I can’t think of any profession where, for example, someone who has been recognized for their leadership abilities shouldn’t be ranked higher than someone with the same background who hasn’t been recognized for it.”
When the researchers asked GPT-4 to explain its rankings, they found explicit and implicit ableism in its responses, noting, for example, that candidates with depression “place more emphasis on DEI and personal challenges,” which “detracts from the core technical and research-oriented aspects of the role.”
“Some GPT descriptions color an entire resume based on disability, arguing that DEI or disability engagement can detract from other parts of the resume,” Grazko said. “For example, the depression resume comparison illusory the concept of ‘challenges,’ but doesn’t mention ‘challenges’ at all, which can lead to certain stereotypes popping up.”
Given this, the researchers were interested in whether the system could be trained to be less biased. They took advantage of the GPTs Editor tool, which allowed users to customize GPT-4 with written instructions (no code required). They instructed the chatbot to not exhibit ableist bias and instead behave in accordance with disability justice and DEI principles.
They ran the experiment again, this time with a newly trained chatbot. Overall, the system ranked the enhanced CV higher than the control CV 37 times out of 60. However, for some disorders, the improvement was minimal or nonexistent: the autism CV took first place only 3 times out of 10, and the depression CV only 2 times (unchanged from the original GPT-4 results).
“When using AI for these real-world tasks, people need to be aware of the biases in the system,” Grazko said, “otherwise recruiters using ChatGPT wouldn’t be able to make these corrections or realize that biases may remain even with instructions.”
The researchers note that some organizations, such as ourability.com and inclusively.com, are working to improve outcomes for job seekers with disabilities, who face bias regardless of whether AI is used in hiring. They also emphasize that more research is needed to document and remediate bias in AI. This could include testing other systems, such as Google’s Gemini and Meta’s Llama, incorporating other disabilities, studying the intersection of systems’ bias against disabilities with other attributes such as gender and race, exploring whether further customization can make bias more consistent across disabilities, and seeing whether a base version of GPT-4 can be made less biased.
“Studying and documenting these biases is critical,” Mankoff said. “We hope to learn from and contribute to the broader conversation about ensuring technology is implemented and deployed in a fair and equitable way, not just for disabilities but for other underrepresented identities as well.”
Additional co-authors are Yusuf Mohammed, an undergraduate student at the UW Allen School, Venkatesh Potluri, a doctoral student at the UW Allen School, and Ben Kosa, who completed the study as an undergraduate student at the UW Allen School and will be entering a doctoral program at the University of Wisconsin-Madison.
Further information: Kate Glazko et al., “Identifying and Remediating Disability Bias in GPT-Based Resume Screening,” 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024. DOI: 10.1145/3630106.3658933
Courtesy of University of Washington
Citation: ChatGPT is biased against resumes with credentials that imply disabilities, but it can be improved (June 22, 2024) Retrieved June 22, 2024 from https://techxplore.com/news/2024-06-chatgpt-biased-resumes-credentials-imply.html
This document is subject to copyright. It may not be reproduced without written permission, except for fair dealing for the purposes of personal study or research. The content is provided for informational purposes only.