Connect with us

Ethics & Policy

A simple twist fooled AI—and revealed a dangerous flaw in medical ethics

Published

on


A study by investigators at the Icahn School of Medicine at Mount Sinai, in collaboration with colleagues from Rabin Medical Center in Israel and other collaborators, suggests that even the most advanced artificial intelligence (AI) models can make surprisingly simple mistakes when faced with complex medical ethics scenarios.

The findings, which raise important questions about how and when to rely on large language models (LLMs), such as ChatGPT, in health care settings, were reported in the July 22 online issue of NPJ Digital Medicine[10.1038/s41746-025-01792-y].

The research team was inspired by Daniel Kahneman’s book “Thinking, Fast and Slow,” which contrasts fast, intuitive reactions with slower, analytical reasoning. It has been observed that large language models (LLMs) falter when classic lateral-thinking puzzles receive subtle tweaks. Building on this insight, the study tested how well AI systems shift between these two modes when confronted with well-known ethical dilemmas that had been deliberately tweaked.

“AI can be very powerful and efficient, but our study showed that it may default to the most familiar or intuitive answer, even when that response overlooks critical details,” says co-senior author Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai. “In everyday situations, that kind of thinking might go unnoticed. But in health care, where decisions often carry serious ethical and clinical implications, missing those nuances can have real consequences for patients.”

To explore this tendency, the research team tested several commercially available LLMs using a combination of creative lateral thinking puzzles and slightly modified well-known medical ethics cases. In one example, they adapted the classic “Surgeon’s Dilemma,” a widely cited 1970s puzzle that highlights implicit gender bias. In the original version, a boy is injured in a car accident with his father and rushed to the hospital, where the surgeon exclaims, “I can’t operate on this boy — he’s my son!” The twist is that the surgeon is his mother, though many people don’t consider that possibility due to gender bias. In the researchers’ modified version, they explicitly stated that the boy’s father was the surgeon, removing the ambiguity. Even so, some AI models still responded that the surgeon must be the boy’s mother. The error reveals how LLMs can cling to familiar patterns, even when contradicted by new information.

In another example to test whether LLMs rely on familiar patterns, the researchers drew from a classic ethical dilemma in which religious parents refuse a life-saving blood transfusion for their child. Even when the researchers altered the scenario to state that the parents had already consented, many models still recommended overriding a refusal that no longer existed.

“Our findings don’t suggest that AI has no place in medical practice, but they do highlight the need for thoughtful human oversight, especially in situations that require ethical sensitivity, nuanced judgment, or emotional intelligence,” says co-senior corresponding author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and Chief AI Officer of the Mount Sinai Health System. “Naturally, these tools can be incredibly helpful, but they’re not infallible. Physicians and patients alike should understand that AI is best used as a complement to enhance clinical expertise, not a substitute for it, particularly when navigating complex or high-stakes decisions. Ultimately, the goal is to build more reliable and ethically sound ways to integrate AI into patient care.”

“Simple tweaks to familiar cases exposed blind spots that clinicians can’t afford,” says lead author Shelly Soffer, MD, a Fellow at the Institute of Hematology, Davidoff Cancer Center, Rabin Medical Center. “It underscores why human oversight must stay central when we deploy AI in patient care.”

Next, the research team plans to expand their work by testing a wider range of clinical examples. They’re also developing an “AI assurance lab” to systematically evaluate how well different models handle real-world medical complexity.

The paper is titled “Pitfalls of Large Language Models in Medical Ethics Reasoning.”

The study’s authors, as listed in the journal, are Shelly Soffer, MD; Vera Sorin, MD; Girish N. Nadkarni, MD, MPH; and Eyal Klang, MD.

About Mount Sinai’s Windreich Department of AI and Human Health

Led by Girish N. Nadkarni, MD, MPH — an international authority on the safe, effective, and ethical use of AI in health care — Mount Sinai’s Windreich Department of AI and Human Health is the first of its kind at a U.S. medical school, pioneering transformative advancements at the intersection of artificial intelligence and human health.

The Department is committed to leveraging AI in a responsible, effective, ethical, and safe manner to transform research, clinical care, education, and operations. By bringing together world-class AI expertise, cutting-edge infrastructure, and unparalleled computational power, the department is advancing breakthroughs in multi-scale, multimodal data integration while streamlining pathways for rapid testing and translation into practice.

The Department benefits from dynamic collaborations across Mount Sinai, including with the Hasso Plattner Institute for Digital Health at Mount Sinai — a partnership between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System — which complements its mission by advancing data-driven approaches to improve patient care and health outcomes.

At the heart of this innovation is the renowned Icahn School of Medicine at Mount Sinai, which serves as a central hub for learning and collaboration. This unique integration enables dynamic partnerships across institutes, academic departments, hospitals, and outpatient centers, driving progress in disease prevention, improving treatments for complex illnesses, and elevating quality of life on a global scale.

In 2024, the Department’s innovative NutriScan AI application, developed by the Mount Sinai Health System Clinical Data Science team in partnership with Department faculty, earned Mount Sinai Health System the prestigious Hearst Health Prize. NutriScan is designed to facilitate faster identification and treatment of malnutrition in hospitalized patients. This machine learning tool improves malnutrition diagnosis rates and resource utilization, demonstrating the impactful application of AI in health care.

* Mount Sinai Health System member hospitals: The Mount Sinai Hospital; Mount Sinai Brooklyn; Mount Sinai Morningside; Mount Sinai Queens; Mount Sinai South Nassau; Mount Sinai West; and New York Eye and Ear Infirmary of Mount Sinai



Source link

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Ethics & Policy

5 ways companies are incorporating AI ethics – myupnow.com

Published

on



5 ways companies are incorporating AI ethics  myupnow.com



Source link

Continue Reading

Ethics & Policy

Letters: Two-party system | International affairs | AI ethics | Bringing music education to kids

Published

on

By


Two-party system

For the time being, and for the foreseeable future, we live in a two-party system. That means that Democrats are the only political party that can check the power of Trump, MAGA and Republicans who choose to bow to a fascist regime. It also means that Democrats have to win in the 2026 midterm elections and the 2028 general election.

This is a tall order given all the woes that currently beset the party: no clear leader, lousy messaging, an inability to connect with young people and, perhaps most importantly to recognize with the recent observance of Labor Day, the loss of working class voters including low-income and low-propensity voters.

Yet this could also be an opportunity. To paraphrase NASA’s Gene Krantz during the Apollo 13 crisis in 1970, “This could be our (Democrats) finest hour.” Labor Day can serve as a reminder to us that working people have the power to drastically alter the political environment. We have seen this time and again in our country’s history: think of the conditions that led to the New Deal, the civil rights movement, and the war on poverty. 

As Bishop William J. Barber from the Poor People’s campaign has noted, the combination of working people, moral leaders, and strong allies coming together can “reconstruct democracy”.

– Ward Kanowsky

International affairs

National security is of utmost importance; foreign aid is how we secure it.

National security and foreign aid are often seen as tangential entities. National security conjures images of large, marching militaries or closed, concrete borders. Foreign aid is seen as a nonprofit undertaking, one carried out by large organizations like UNICEF or smaller local enterprises.

These vivid images are not completely stereotypical, but they don’t paint the whole picture. As an intern at the Borgen Project, I learnt a very vital dogma: foreign aid secures national security.

There are pronounced correlations that prove that focusing on non-combat, diplomatic strategies can alleviate poverty in developing countries while securing America’s borders. 

The most dangerous countries in the world are also the poorest. Families who cannot afford expensive education send their kids to religious schools, which, while providing an avenue for education, can also be a breeding ground for extremist ideology. 

In the late 1980s, Charlie Wilson pleaded for Congress to build schools in Afghanistan after their war with the Soviets. The consequences of his failed plea can be seen in the rise of extremism in Afghanistan in the following years.

The solution to this cause is best summarized by former secretary of defense Chuck Hagel:

“America’s role in the world should reflect the hope and promise of our country, and possibilities for all mankind, tempered with a wisdom that has been the hallmark of our national character. That means pursuing a principled and engaged realism that employs diplomatic, economic, and security tools as well as our values to advance our security and our prosperity.”

— Atheeth Ravikrishnan

Teen’s nonprofit brings music education to kids

As a high school student, I’m proud to share the work of Youthtones, a nonprofit I started with a team of teen volunteers to bring music education to kids in the Bay Area. Our mission is simple: connect young musicians with children to provide free or affordable music lessons.

Through YouthTones, our team helps students develop not only musical skills, but also confidence, creativity, and a sense of community. What makes this program special is that it’s entirely run by teens — our volunteers aren’t just teaching music, they’re mentoring and inspiring the next generation of young musicians.

Watching the students grow, overcome challenges, and find joy in music has been incredibly rewarding. Many families in our area don’t have easy access to music lessons, and YouthTones helps fill that gap.

I hope our story inspires others to recognize the power of youth leadership and the impact a group of motivated teens can have in their community. Music has the power to bring people together, and our team at YouthTones is dedicated to making that power accessible to every child who wants to learn.

— Henna Lam 

AI ethics

When I began studying artificial intelligence as a college student, I learned how AI could be a tool for social good, helping us understand climate change, improve public health and reduce waste through smart automation. I still see that potential. But the way we are building AI today is taking us further from that vision.

Like many students entering tech, I first saw AI as innovation. I was taught to celebrate breakthroughs in machine learning, natural language processing and automation. But it did not take long before I started questioning what was missing from those conversations.

The environmental costs of large scale AI models are enormous. A 2023 MIT report found that training a single large language model could emit over 626 thousand pounds of carbon dioxide, equal to five cars over their lifetimes. These models run in data centers that consume massive electricity and water, often in areas already strained by climate change.

These facts are not minor. They are just ignored. Something we also overlook is the labor behind AI. Thousands of underpaid workers in countries like Kenya, the Philippines and Venezuela label toxic content so others can have so called safe systems. Their trauma goes unseen.

In school, we barely talked about climate or workers. That needs to change.

AI can support climate action, but not if it causes harm or worsens inequality. We cannot build sustainable solutions on extractive foundations.

I still believe in AI. But belief is not enough. If we do not build ethically now, we may not get a second chance.

– Aadya Madgula

Most Popular



Source link

Continue Reading

Ethics & Policy

OpenAI Merges Teams to Boost ChatGPT Ethics and Cut Biases

Published

on

By


In a move that underscores the evolving priorities within artificial intelligence development, OpenAI has announced a significant reorganization of its Model Behavior team, the group responsible for crafting the conversational styles and ethical guardrails of models like ChatGPT. According to an internal memo obtained by TechCrunch, this compact unit of about 14 researchers is being folded into the larger Post Training team, which focuses on refining AI models after their initial training phases. The shift, effective immediately, sees the team’s leader, Lilian Weng, transitioning to a new role within the company, while the group now reports to Max Schwarzer, head of Post Training.

This restructuring comes amid growing scrutiny over how AI systems interact with users, particularly in balancing helpfulness with honesty. The Model Behavior team has been instrumental in addressing issues like sycophancy—where models excessively affirm user opinions—and mitigating political biases in responses. Insiders suggest the integration aims to streamline these efforts, embedding personality shaping directly into the core refinement process rather than treating it as a separate silo.

Strategic Alignment in AI Development

OpenAI’s decision reflects broader industry trends toward more cohesive AI development pipelines, where behavioral tuning is not an afterthought but a foundational element. Recent user feedback on GPT-5, as highlighted in posts on X (formerly Twitter), has pointed to overly formal or detached interactions, prompting tweaks to make ChatGPT feel “warmer and friendlier” without veering into unwarranted flattery. For instance, OpenAI’s own announcements on the platform in August 2025 detailed the introduction of new chat personalities like Cynic, Robot, Listener, and Nerd, available as opt-in options in settings.

These changes build on earlier experiments, such as A/B testing different personality styles noted by users on X as far back as April 2025. Publications like WebProNews report that the reorganization is partly driven by GPT-5 feedback, emphasizing reductions in sycophantic tendencies and enhancements in engagement through advanced reasoning and safety features.

Implications for Ethical AI and User Experience

The merger could accelerate OpenAI’s ability to iterate on model behaviors, potentially leading to more context-aware interactions that better align with ethical standards. As detailed in a BitcoinWorld analysis, this realignment is crucial for influencing user experience and ethical frameworks, especially in sectors like cryptocurrency and blockchain where AI’s role is expanding. The team’s past work on models since GPT-4 has reduced harmful outputs by significant margins, with one X post claiming a 78% drop in certain biases, though such figures remain unverified by OpenAI.

Critics, however, worry that consolidating teams might dilute specialized focus on nuanced issues like bias management. Industry observers on X have debated the “sycophancy trap,” where tuning for truthfulness risks alienating casual users who prefer comforting responses, creating a game-theory dilemma for developers.

Leadership Shifts and Future Directions

Lilian Weng’s departure from the team leadership marks a notable transition; her expertise in AI safety has been pivotal, and her new project remains undisclosed. OpenAI spokesperson confirmed to StartupNews.fyi that the move is designed to foster closer collaboration, positioning the company to lead in human-AI dialogue evolution.

Looking ahead, this reorganization signals OpenAI’s bet on integrated teams to handle the complexities of next-generation AI. With GPT-5 already incorporating subtle warmth adjustments based on internal tests, as per OpenAI’s X updates, the focus is on genuine, professional engagement that avoids pitfalls like ungrounded praise. For industry insiders, this could mean faster deployment of features that make AI feel more human-like, while upholding values of honesty and utility.

Broader Industry Ripple Effects

The changes at OpenAI are likely to influence competitors, as the quest for balanced AI personalities intensifies. Reports from NewsBytes and Bitget News emphasize how this restructuring enhances post-training interactions, potentially setting new benchmarks for AI ethics. User sentiment on X, including discussions of model selectors and capacity limits, suggests ongoing refinements will be key to retaining loyalty.

Ultimately, as OpenAI navigates these internal shifts, the emphasis on personality could redefine how we perceive and interact with AI, blending technical prowess with empathetic design in ways that resonate across applications from everyday queries to complex problem-solving.



Source link

Continue Reading

Trending