Connect with us

Tools & Platforms

Workday, Amazon AI employment bias claims add to growing concerns about the tech’s hiring discrimination

Published

on


Despite AI hiring tools’ best efforts to streamline hiring processes for a growing pool of applicants, the technology meant to open doors for a wider array of prospective employees may actually be perpetuating decades-long patterns of discrimination.

AI hiring tools have become ubiquitous, with 492 of the Fortune 500 companies using applicant tracking systems to streamline recruitment and hiring in 2024, according to job application platform Jobscan. While these tools can help employers screen more job candidates and help identify relevant experience, human resources and legal experts warn improper training and implementation of hiring technologies can proliferate biases.

Research offers stark evidence of AI’s hiring discrimination. The University of Washington Information School published a study last year finding that in AI-assisted resume screenings across nine occupations using 500 applications, the technology favored white-associated names in 85.1% of cases and female associated names in only 11.1% of cases. In some settings, Black male participants were disadvantaged compared to their white male counterparts in up to 100% of cases.

“You kind of just get this positive feedback loop of, we’re training biased models on more and more biased data,” Kyra Wilson, a doctoral student at the University of Washington Information School and the study’s lead author, told Fortune. “We don’t really know kind of where the upper limit of that is yet, of how bad it is going to get before these models just stop working altogether.”

Some workers are claiming to see evidence of this discrimination outside of just experimental settings. Last month, five plaintiffs, all over the age of 40, claimed in a collective action lawsuit that workplace management software firm Workday has discriminatory job applicant screening technology. Plaintiff Derek Mobley alleged in an initial lawsuit last year the company’s algorithms caused him to be rejected from more than 100 jobs over seven years on account of his race, age, and disabilities.

Workday denied the discrimination claims and said in a statement to Fortune the lawsuit is “without merit.” Last month the company announced it received two third-party accreditations for its “commitment to developing AI responsibly and transparently.”

“Workday’s AI recruiting tools do not make hiring decisions, and our customers maintain full control and human oversight of their hiring process,” the company said. “Our AI capabilities look only at the qualifications listed in a candidate’s job application and compare them with the qualifications the employer has identified as needed for the job. They are not trained to use—or even identify—protected characteristics like race, age, or disability.”

It’s not just hiring tools with which workers are taking issue. A letter sent to Amazon executives, including CEO Andy Jassy, on behalf of 200 employees with disabilities claimed the company flouted the Americans with Disabilities Act. Amazon allegedly had employees make decisions on accommodations based on AI processes that don’t abide by ADA standards, The Guardian reported this week. Amazon told Fortune its AI does not make any final decisions around employee accommodations.

“We understand the importance of responsible AI use, and follow robust guidelines and review processes to ensure we build AI integrations thoughtfully and fairly,” a spokesperson told Fortune in a statement.

How could AI hiring tools be discriminatory?

Just as with any AI application, the technology is only as smart as the information it’s being fed. Most AI hiring tools work by screening resumes or resume screening evaluating interview questions, according to Elaine Pulakos, CEO of talent assessment developer PDRI by Pearson. They’re trained with a company’s existing model of assessing candidates, meaning if the models are fed existing data from a company—such as demographics breakdowns showing a preference for male candidates or Ivy League universities—it is likely to perpetuate hiring biases that can lead to “oddball results” Pulakos said.

“If you don’t have information assurance around the data that you’re training the AI on, and you’re not checking to make sure that the AI doesn’t go off the rails and start hallucinating, doing weird things along the way, you’re going to you’re going to get weird stuff going on,” she told Fortune. “It’s just the nature of the beast.”

Much of AI’s biases come from human biases, and therefore, according to Washington University law professor Pauline Kim, AI’s hiring discrimination exists as a result of human hiring discrimination, which is still prevalent today. A landmark 2023 Northwestern University meta-analysis of 90 studies across six countries found persistent and pervasive biases, including that employers called back white applicants on average 36% more than Black applicants and 24% more than Latino applicants with identical resumes.

The rapid scaling of AI in the workplace can fan these flames of discrimination, according to Victor Schwartz, associate director of technical product management of remote work job search platform Bold.

“It’s a lot easier to build a fair AI system and then scale it to the equivalent work of 1,000 HR people, than it is to train 1,000 HR people to be fair,” Schwartz told Fortune. “Then again, it’s a lot easier to make it very discriminatory, than it is to train 1,000 people to be discriminatory.”

“You’re flattening the natural curve that you would get just across a large number of people,” he added. “So there’s an opportunity there. There’s also a risk.”

How HR and legal experts are combatting AI hiring biases

While employees are protected from workplace discrimination through the Equal Employment Opportunity Commission and Title VII of the Civil Rights Act of 1964, “there aren’t really any formal regulations about employment discrimination in AI,” said law professor Kim. 

Existing law prohibits against both intentional and disparate impact discrimination, which refers to discrimination that occurs as a result of a neutral appearing policy, even if it’s not intended.

“If an employer builds an AI tool and has no intent to discriminate, but it turns out that overwhelmingly the applicants that are screened out of the pool are over the age of 40, that would be something that has a disparate impact on older workers,” Kim said.

Though disparate impact theory is well-established by the law, Kim said, President Donald Trump has made clear his hostility for this form of discrimination by seeking to eliminate it through an executive order in April.

“What it means is agencies like the EEOC will not be pursuing or trying to pursue cases that would involve disparate impact, or trying to understand how these technologies might be having a discrete impact,” Kim said. “They are really pulling back from that effort to understand and to try to educate employers about these risks.”

The White House did not immediately respond to Fortune’s request for comment.

With little indication of federal-level efforts to address AI employment discrimination, politicians on the local level have attempted to address the technology’s potential for prejudice, including a New York City ordinance banning employers and agencies from using “automated employment decision tools” unless the tool has passed a bias audit within a year of its use. 

Melanie Ronen, an employment lawyer and partner at Stradley Ronon Stevens & Young, LLP, told Fortune other state and local laws have focused on increasing transparency on when AI is being used in the hiring process, “including the opportunity [for prospective employees] to opt out of the use of AI in certain circumstances.”

The firms behind AI hiring and workplace assessments, such as PDRI and Bold, have said they’ve taken it upon themselves to mitigate bias in the technology, with PDRI CEO Pulakos advocating for human raters to evaluate AI tools ahead of their implementation.

Bold technical product management director Schwartz argued that while guardrails, audits, and transparency should be key in ensuring AI is able to conduct fair hiring practices, the technology also had the potential to diversify a company’s workforce if applied appropriately. He cited research indicating women tend to apply to fewer jobs than men, doing so only when they meet all qualifications. If AI on the job candidate’s side can streamline the application process, it could remove hurdles for those less likely to apply to certain positions.

“By removing that barrier to entry with these auto-apply tools, or expert-apply tools, we’re able to kind of level the playing field a little bit,” Schwartz said.



Source link

Tools & Platforms

NSU expands cybersecurity, AI programs to meet growing job demand

Published

on


As cybersecurity threats and artificial intelligence continue reshaping the job market, Northeastern State University is stepping up its efforts to prepare students for these in-demand fields.

With programs targeting both K-12 engagement and college-level degrees, NSU is positioning itself as a key player in Oklahoma’s tech talent pipeline.

Cybersecurity: Training the Next Generation

NSU is working to meet the rising need for cybersecurity professionals by launching educational initiatives for students at multiple levels. Dr. Stacey White, the university’s cybersecurity program coordinator, says young people are especially suited for these roles because of their comfort with technology.

That’s why NSU is hosting cybersecurity camps and has built hands-on facilities like a cybersecurity lab to introduce students to real-world applications.

“When I first started in technology and the cyber world, it was usernames and passwords,” Dr. White said. “Today, it’s much more intricate than that.”

The Scope of the Problem

Cybercrime is a growing threat that shows no signs of slowing down. According to Dr. White, everyone should have a basic understanding of cybersecurity, but the greatest need lies in training new professionals who can keep up with evolving threats.

Currently, there are nearly 450,000 open cybersecurity jobs nationwide — including almost 4,200 in Oklahoma alone.

New AI Degree Launching This Fall

This fall, NSU is introducing a new degree in Artificial Intelligence and Data Analytics. Dr. Janet Buzzard, dean of the College of Business and Technology, says the program combines technical knowledge with business insight — a skill set that employers across many industries are seeking.

“All of our graduates in our College of Business and Technology need that skill set of artificial intelligence,” Dr. Buzzard said. “Not just the one major and degree that we’re promoting here.”

The new degree is designed to respond to student interest and market demand, offering versatile career paths in fields such as finance, logistics, and technology development.

Encouraging Early Engagement

Dr. Buzzard adds that exposing students to artificial intelligence and cybersecurity early in their academic careers helps them see these paths as viable and exciting career options.

This is one of the reasons NSU Broken Arrow is hosting a cybersecurity camp for middle school-aged students today and June 8. Campers will learn from industry professionals and experienced educators about the importance of cybersecurity, effective communication in a rapidly evolving digital world and foundational concepts in coding and encoding. 

NSU’s efforts to modernize its programs come at a crucial time, with both AI and cybersecurity jobs seeing major growth. For students and professionals alike, the university is building opportunities that align with the future of work.





Source link

Continue Reading

Tools & Platforms

Lecturer Says AI Has Made Her Workload Skyrocket, Fears Cheating

Published

on


This as-told-to essay is based on a transcribed conversation with Risa Morimoto, a senior lecturer in economics at SOAS University of London, in England. The following has been edited for length and clarity.

Students always cheat.

I’ve been a lecturer for 18 years, and I’ve dealt with cheating throughout that time, but with AI tools becoming widely available in recent years, I’ve experienced a significant change.

There are definitely positive aspects to AI. It’s much easier to get access to information and students can use these tools to improve their writing, spelling, and grammar, so there are fewer badly written essays.

However, I believe some of my students have been using AI to generate essay content that pulls information from the internet, instead of using material from my classes to complete their assignments.

AI is supposed to help us work efficiently, but my workload has skyrocketed because of it. I have to spend lots of time figuring out whether the work students are handing in was really written by them.

I’ve decided to take dramatic action, changing the way I assess students to encourage them to be more creative and rely less on AI. The world is changing, so universities can’t stand still.

Cheating has become harder to detect because of AI

I’ve worked at SOAS University of London since 2012. My teaching focus is ecological economics.

Initially, my teaching style was exam-based, but I found that students were anxious about one-off exams, and their results wouldn’t always correspond to their performance.

I eventually pivoted to a focus on essays. Students chose their topic and consolidated theories into an essay. It worked well — until AI came along.

Cheating used to be easier to spot. I’d maybe catch one or two students cheating by copying huge chunks of text from internet sources, leading to a plagiarism case. Even two or three years ago, detecting inappropriate AI use was easier due to signs like robotic writing styles.

Now, with more sophisticated AI technologies, it’s harder to detect, and I believe the scale of cheating has increased.

I’ll read 100 essays and some of them will be very similar using identical case examples, that I’ve never taught.

These examples are typically referenced on the internet, which makes me think the students are using an AI tool that is incorporating them. Some of the essays will cite 20 pieces of literature, but not a single one will be something from the reading list I set.

While students can use examples from internet sources in their work, I’m concerned that some students have just used AI to generate the essay content without reading or engaging with the original source.

I started using AI detection tools to assess work, but I’m aware this technology has limitations.

AI tools are easy to access for students who feel pressured by the amount of work they have to do. University fees are increasing, and a lot of students work part-time jobs, so it makes sense to me that they want to use these tools to complete work more quickly.

There’s no obvious way to judge misconduct

During the first lecture of my module, I’ll tell students they can use AI to check grammar or summarize the literature to better understand it, but they can’t use it to generate responses to their assignments.

SOAS has guidance for AI use among students, which sets similar principles about not using AI to generate essays.

Over the past year, I’ve sat on an academic misconduct panel at the university, dealing with students who’ve been flagged for inappropriate AI use across departments.

I’ve seen students refer to these guidelines and say that they only used AI to support their learning and not to write their responses.

It can be hard to make decisions because you can’t be 100% sure from reading the essay whether it’s AI-generated or not. It’s also hard to draw a line between cheating and using AI to support learning.

Next year, I’m going to dramatically change my assignment format

My colleagues and I speak about the negative and positive aspects of AI, and we’re aware that we still have a lot to learn about the technology ourselves.

The university is encouraging lecturers to change their teaching and assessment practices. At the department level, we often discuss how to improve things.

I send my two young children to a school with an alternative, progressive education system, rather than a mainstream British state school. Seeing how my kids are educated has inspired me to try two alternative assessment methods this coming academic year. I had to go through a formal process with the university to get them approved.

I’ll ask my students to choose a topic and produce a summary of what they learned in the class about it. Second, they’ll create a blog, so they can translate what they’ve understood of the highly technical terms into a more communicable format.

My aim is to make sure the assignments are directly tied to what we’ve learned in class and make assessments more personal and creative.

The old assessment model, which involves memorizing facts and regurgitating them in exams, isn’t useful anymore. ChatGPT can easily give you a beautiful summary of information like this. Instead, educators need to help students with soft skills, communication, and out-of-the-box thinking.

In a statement to BI, a SOAS spokesperson said students are guided to use AI in ways that “uphold academic integrity.” They said the university encouraged students to pursue work that is harder for AI to replicate and have “robust mechanisms” in place for investigating AI misuse. “The use of AI is constantly evolving, and we are regularly reviewing and updating our policies to respond to these changes,” the spokesperson added.

Do you have a story to share about AI in education? Contact this reporter at ccheong@businessinsider.com.





Source link

Continue Reading

Tools & Platforms

Searching for boundaries in the AI jungle

Published

on


Stamatis Gatirdakis, co-founder and president of the Ethikon Institute, still remembers the first time he used ChatGPT. It was the fall of 2022 and a fellow student in the Netherlands sent him the link to try it out. “It made a lot of mistakes back then, but I saw how it was improving at an incredible rate. From the very first tests, I felt that it would change the world,” he tells Kathimerini. Of course, he also identified some issues, mainly legal and ethical, that could arise early on, and last year, realizing that there was no private entity that dealt exclusively with the ethical dimension of artificial intelligence, he decided to take the initiative.

He initially turned to his friends, young lawyers like him, engineers and programmers with similar concerns. “In the early days, we would meet after work, discussing ideas about what we could do,” recalls Maria Voukelatou, executive director at Ethikon and lawyer specialized in technology law and IP matters. Her master’s degree, which she earned in the Netherlands in 2019, was on the ethics and regulatory aspects of new technologies. “At that time, the European Union’s white paper on artificial intelligence had just been released, which was a first, hesitant step. But even though technology is changing rapidly, the basic ethical dilemmas and how we legislate remain constant. The issue is managing to balance innovation with citizen protection,” she explains.

Together with three other Greeks (Apostolos Spanos, Michael Manis and Nikos Vadivoulis), they made up the institute’s founding team, and sought out colleagues abroad with experience in these issues. Thus, Ethikon was created – a nonprofit company that does not provide legal services, but implements educational, research and social awareness actions on artificial intelligence.

Stamatis Gatirdakis, co-founder and president of the Ethikon Institute.

Copyrights

One of the first issues they addressed was copyrights. “In order not to stop the progress of technology, exceptions were initially made so that these models of productive artificial intelligence could use online content for educational purposes, without citing the source or compensating the creators,” explains Gatirdakis, adding that this resulted in copyrights being sidelined. “The battle between creators and the big tech giants has been lost. But because companies don’t want them against them, they have started making commercial agreements, whereby every time their data is used to produce answers, they receive percentages on a calculated model.”

Beyond compensation, another key question arises: Who is ultimately the creator of a work produced through artificial intelligence? “There are already conflicting court decisions. In the US, they argue that artificial intelligence cannot produce an ‘original’ work and that the work belongs to the search engine companies,” says Voukelatou. A typical example is the comic book, ‘Zarya of the Dawn,’ authored by artist and artificial intelligence (AI) consultant Kris Kashtanova, with images generated through the AI platform Midjourney. The US Copyright Office rejected the copyright application for the images in her book when it learned that they were created exclusively by artificial intelligence. On the contrary, in China, in corresponding cases, they ruled that because the user gives the exact instructions, he or she is the creator.

Personal data

Another crucial issue is the protection of personal data. “When we upload notes or files, what happens to all this content? Does the algorithm learn from them? Does it use them elsewhere? Presumably not, but there are still no safeguards. There is no case law, nor a clear regulatory framework,” says Voukelatou, who mentions the loopholes that companies exploit to overcome obstacles with personal data. “Like the application that transforms your image into a cartoon by the famous Studio Ghibli. Millions of users gave consent for their image to be processed and so this data entered the systems and trained the models. If a similar image is subsequently produced, it no longer belongs to the person who first uploaded it. And this part is legally unregulated.”

The problem, they explain, is that the development of these technologies is mainly taking place in the United States and China, which means that Europe remains on the sidelines of a meaningful discussion. The EU regulation on artificial intelligence (AI Act), first presented in the summer of 2024, is the first serious attempt to set a regulatory framework. Members of Ethikon participated in the consultation of the regulation and specifically focused on the categorization of artificial intelligence applications based on the level of risk. “We supported with examples the prohibition of practices such as ‘social scoring’ adopted by China, where citizens are evaluated in real time through surveillance cameras. This approach was incorporated and the regulation explicitly prohibits such practices,” says Gatirdakis, who participated in the consultation.

“The final text sets obligations and rules. It also provides for strict fines depending on turnover. However, we are in a transition period and we are all waiting for further guidelines from the European Union. It is assumed that it will be fully implemented in the summer of 2026. However, there are already delays in the timetable and in the establishment of the supervisory authorities,” the two experts said.

searching-for-boundaries-in-the-ai-jungle2
Maria Voukelatou, executive director at Ethikon and lawyer specialized in technology law and IP matters.

The team’s activities

Beyond consultation, the Ethikon team is already developing a series of actions to raise awareness among users, whether they are business executives or students growing up with artificial intelligence. The team’s executives created a comic inspired by the Antikythera Mechanism that explains in a simple way the possibilities but also the dangers of this new technology. They also developed a generative AI engine based exclusively on sources from scientific libraries – however, its use is expensive and they are currently limiting it to pilot educational actions. They recently organized a conference in collaboration with the Laskaridis Foundation and published an academic article on March 29 exploring the legal framework for strengthening of copyright.

In the article, titled “Who Owns the Output? Bridging Law and Technology in LLMs Attribution,” they analyze, among other things, the specific tools and techniques that allow the detection of content generated by artificial intelligence and its connection to the data used to train the model or the user who created it. “For example, a digital signature can be embedded in texts, images or videos generated by AI, invisible to the user, but recognizable with specific tools,” they explain.

The Ethikon team has already begun writing a second – more technical – academic article, while closely monitoring technological developments internationally. “In 2026, we believe that we will be much more concerned with the energy and environmental footprint of artificial intelligence,” says Gatirdakis. “Training and operating models requires enormous computing power, resulting in excessively high energy and water consumption for cooling data centers. The concern is not only technical or academic – it touches the core of the ethical development of artificial intelligence. How do we balance innovation with sustainability.” At the same time, he explains, serious issues of truth management and security have already arisen. “We are entering a period where we will not be able to easily distinguish whether what we see or hear is real or fabricated,” he continues. 

In some countries, the adoption of technology is happening at breakneck speed. In the United Arab Emirates, an artificial intelligence system has been developed that drafts laws and monitors the implementation of laws. At the same time, OpenAI announced a partnership with the iPhone designer to launch a new device that integrates artificial intelligence with voice, visual and personal interaction in late 2026. “A new era seems to be approaching, in which artificial intelligence will be present not only on our screens but also in the natural environment.” 





Source link

Continue Reading

Trending