Ethics & Policy
A simple twist fooled AI—and revealed a dangerous flaw in medical ethics
A study by investigators at the Icahn School of Medicine at Mount Sinai, in collaboration with colleagues from Rabin Medical Center in Israel and other collaborators, suggests that even the most advanced artificial intelligence (AI) models can make surprisingly simple mistakes when faced with complex medical ethics scenarios.
The findings, which raise important questions about how and when to rely on large language models (LLMs), such as ChatGPT, in health care settings, were reported in the July 22 online issue of NPJ Digital Medicine[10.1038/s41746-025-01792-y].
The research team was inspired by Daniel Kahneman’s book “Thinking, Fast and Slow,” which contrasts fast, intuitive reactions with slower, analytical reasoning. It has been observed that large language models (LLMs) falter when classic lateral-thinking puzzles receive subtle tweaks. Building on this insight, the study tested how well AI systems shift between these two modes when confronted with well-known ethical dilemmas that had been deliberately tweaked.
“AI can be very powerful and efficient, but our study showed that it may default to the most familiar or intuitive answer, even when that response overlooks critical details,” says co-senior author Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai. “In everyday situations, that kind of thinking might go unnoticed. But in health care, where decisions often carry serious ethical and clinical implications, missing those nuances can have real consequences for patients.”
To explore this tendency, the research team tested several commercially available LLMs using a combination of creative lateral thinking puzzles and slightly modified well-known medical ethics cases. In one example, they adapted the classic “Surgeon’s Dilemma,” a widely cited 1970s puzzle that highlights implicit gender bias. In the original version, a boy is injured in a car accident with his father and rushed to the hospital, where the surgeon exclaims, “I can’t operate on this boy — he’s my son!” The twist is that the surgeon is his mother, though many people don’t consider that possibility due to gender bias. In the researchers’ modified version, they explicitly stated that the boy’s father was the surgeon, removing the ambiguity. Even so, some AI models still responded that the surgeon must be the boy’s mother. The error reveals how LLMs can cling to familiar patterns, even when contradicted by new information.
In another example to test whether LLMs rely on familiar patterns, the researchers drew from a classic ethical dilemma in which religious parents refuse a life-saving blood transfusion for their child. Even when the researchers altered the scenario to state that the parents had already consented, many models still recommended overriding a refusal that no longer existed.
“Our findings don’t suggest that AI has no place in medical practice, but they do highlight the need for thoughtful human oversight, especially in situations that require ethical sensitivity, nuanced judgment, or emotional intelligence,” says co-senior corresponding author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and Chief AI Officer of the Mount Sinai Health System. “Naturally, these tools can be incredibly helpful, but they’re not infallible. Physicians and patients alike should understand that AI is best used as a complement to enhance clinical expertise, not a substitute for it, particularly when navigating complex or high-stakes decisions. Ultimately, the goal is to build more reliable and ethically sound ways to integrate AI into patient care.”
“Simple tweaks to familiar cases exposed blind spots that clinicians can’t afford,” says lead author Shelly Soffer, MD, a Fellow at the Institute of Hematology, Davidoff Cancer Center, Rabin Medical Center. “It underscores why human oversight must stay central when we deploy AI in patient care.”
Next, the research team plans to expand their work by testing a wider range of clinical examples. They’re also developing an “AI assurance lab” to systematically evaluate how well different models handle real-world medical complexity.
The paper is titled “Pitfalls of Large Language Models in Medical Ethics Reasoning.”
The study’s authors, as listed in the journal, are Shelly Soffer, MD; Vera Sorin, MD; Girish N. Nadkarni, MD, MPH; and Eyal Klang, MD.
About Mount Sinai’s Windreich Department of AI and Human Health
Led by Girish N. Nadkarni, MD, MPH — an international authority on the safe, effective, and ethical use of AI in health care — Mount Sinai’s Windreich Department of AI and Human Health is the first of its kind at a U.S. medical school, pioneering transformative advancements at the intersection of artificial intelligence and human health.
The Department is committed to leveraging AI in a responsible, effective, ethical, and safe manner to transform research, clinical care, education, and operations. By bringing together world-class AI expertise, cutting-edge infrastructure, and unparalleled computational power, the department is advancing breakthroughs in multi-scale, multimodal data integration while streamlining pathways for rapid testing and translation into practice.
The Department benefits from dynamic collaborations across Mount Sinai, including with the Hasso Plattner Institute for Digital Health at Mount Sinai — a partnership between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System — which complements its mission by advancing data-driven approaches to improve patient care and health outcomes.
At the heart of this innovation is the renowned Icahn School of Medicine at Mount Sinai, which serves as a central hub for learning and collaboration. This unique integration enables dynamic partnerships across institutes, academic departments, hospitals, and outpatient centers, driving progress in disease prevention, improving treatments for complex illnesses, and elevating quality of life on a global scale.
In 2024, the Department’s innovative NutriScan AI application, developed by the Mount Sinai Health System Clinical Data Science team in partnership with Department faculty, earned Mount Sinai Health System the prestigious Hearst Health Prize. NutriScan is designed to facilitate faster identification and treatment of malnutrition in hospitalized patients. This machine learning tool improves malnutrition diagnosis rates and resource utilization, demonstrating the impactful application of AI in health care.
* Mount Sinai Health System member hospitals: The Mount Sinai Hospital; Mount Sinai Brooklyn; Mount Sinai Morningside; Mount Sinai Queens; Mount Sinai South Nassau; Mount Sinai West; and New York Eye and Ear Infirmary of Mount Sinai
Welcome back to Neural Notes, a weekly column where I look at how AI is affecting Australia. In this edition: a challenger appears for sovereign Australian AI, and it actually wants to pay creatives.
Sovereign Australia AI is a newly launched Sydney venture aiming to build foundational large language models that are entirely Australian. Founded by Simon Kriss and Dr Troy Neilson, the company’s models are said to be trained on Australian data, run on local infrastructure, and governed by domestic privacy standards.
Setting itself apart from offshore giants like OpenAI and Meta, the company plans to build the system for under $100 million, utilising 256 Nvidia Blackwell B200 GPUs. This is the largest domestic AI hardware deployment to date.
The startup also says it has no interest in going up against the likes of OpenAI and Perplexity.
“We are not trying to directly compete with ChatGPT or other global models, because we don’t need to. Instead, we are creating our own foundational models that will serve as viable alternatives that better capture the Australian voice,” Neilson said to SmartCompany.
There are many public and private organisations in Australia that will greatly benefit from a truly sovereign AI solution, but do not want to sacrifice real-world performance. Our Ginan and Australis models will fill that gap.”
Its other point of differentiation is the plan for ethically sourced datasets. In fact, Sovereign Australia AI says that over $10 million is earmarked for licensing and compensating copyright holders whose work contributes to training. Though it admits it will probably require more cash.
“We want to set that benchmark high. Users should be able to understand what datasets the AI they use is trained on. They should be able to be aware of how the data has been curated,” Kriss told SmartCompany.
“They should expect that copyright holders whose data was used to make the model more capable are compensated. That’s what we will bring to the table.”
The price of ethics
As we have seen in recent weeks, copyright has been at the forefront of the generative AI conversation, with both tech founders and lobbyists arguing for relaxed rules and ‘fair use’ exemptions in the name of innovation
Kriss sees an urgent need to value Australian creativity not just as a resource but as an ethical benchmark.
AI development, he says, must avoid the “Wild West, lawless and lacking empathy” mentality that defined its early years and pursue a path that actively engages and protects local content makers.
There’s also a shift away from Silicon Valley’s ‘move fast and litigate later’ philosophy.
Neilson told SmartCompany he has watched international platforms defer creator payment until legal action forced their hand, pointing out the “asking for forgiveness instead of seeking permission” playbook is now coming with a hefty price tag.
Related Article Block Placeholder
Article ID: 322045
Moving forward together with content creators, he suggests, is not only right for Australia but essential if they want to build lasting trust and capability.
But compensation sits awkwardly with technical realities. The company is openly exploring whether a public library-like model, using meta tagging and attribution to route payments, could meaningfully support creators.
Kriss frames this not only as a technical necessity but as a principle: paying for content actually consumed is the backbone of sustainable AI training. The team acknowledges “synthesis is a tough nut to crack,” but for Neilson, that’s a discussion the sector needs rather than something to defer to lawsuits and policy cycles.
As AI industry figures urge Australia to model its approach on the US’ “fair use,” creators and advocates warn this risks legitimising mass scraping and leaving local culture unpaid and unprotected.
This legal ambiguity is rapidly becoming a global standard. Anthropic’s proposed US$1.5 billion book piracy settlement, a case promising $3,000 per affected title, is now on hold as US courts question both the payout and the precedent.
Judges caution that dismissing other direct copyright claims does not resolve the lawfulness of training AI on copyrighted material, leaving creators and platforms worldwide in limbo.
And in recent weeks, another US judge dismissed a copyright infringement lawsuit brought against Meta by 13 authors, including comedian Sarah Silverman. It was the second claim of this nature to be dismissed by the court in San Francisco at the time.
When funding and talent collide with ambition
The presence of multiple teams working in this area, such as Maincode’s Matilda model, suggests that Australia’s sovereign AI movement is well underway.
Neilson welcomes this competition and doesn’t see fragmentation as a potential risk.
“We applaud anyone in the space working on sovereign AI. We’re proud to be part of the initial ground swell standing up AI capability in Australia, and we look forward to the amazing things that the Maincode team will build,” Neilson said.
“The worst parties are the ones where you’re in the room by yourself.”
Behind the scenes, the budget calculations remain complicated. Sovereign Australia AI’s planned $100 million investment sits well below what analysts believe is required for competitive, world-class infrastructure. Industry bodies have called for $2–4 billion to ensure genuine sovereign capability.
While Neilson maintains that local talent and expertise are up to the challenge, persistent skills gaps and global talent poaching mean only coordinated investment can bridge the distance from prototype to deployment.
Transparency and Australian investment
According to Sovereign Australia AI, transparency is a platform feature.
“We can’t control all commercial conversations, but I think it would benefit everyone if these deals were disclosed,” Kriss said.
“Somewhat selfishly, it would benefit us, as users would understand how much of what we charge is going back to creators to make them whole.”
Neilson also welcomes the idea of independent audits.
Related Article Block Placeholder
Article ID: 280226
“How we use that data, may be commercial in confidence, but the raw data, absolutely.”
“This is critical for several reasons. Firstly, it helps eliminate the black box nature of LLMs, where a lack of understanding of the underlying data impedes understanding of outputs.
Secondly, we want to provide the owners of the data the opportunity to opt out of our models if they choose. We need to tag all data to empower this process and we need to have that process audited so every Australian can be proud of the work we do.”
The team also says its Australian and ethical focus is fundamental.
“We’re sovereign down to our corporate structure, our employees, our supply chain, and where we house our hardware. We cannot and will not change our tune just because someone is willing to write a larger cheque,” Kriss said.
He also said the startup would refuse foreign investment “without giving it a second thought”.
“For us, this is not about finding any money … it is all about finding the RIGHT money. If an investor said they would not support paying creatives, we would walk away from the deal. We need to do this right for Australia.”
Finally, the founders challenge conventional notions of what sovereignty and security mean for digital Australia.
Kriss proposed genuine sovereignty can’t be reduced to protection against threats or state interests alone.
“Security goes beyond guns, bombs and intelligence. It goes to a musician being secure in being able to pay their bills, to a reporter being confident that their work isn’t being unfairly ripped off. To divorce being fair from sovereignty is downright unAustralian.”
Can principle meet practicality in Australia’s sovereign AI experiment?
The aims of Sovereign Australia AI are an ambitious counterpoint to the global status quo. Whether these promises will prove practical or affordable for all Australian creators is still an open question.
And this will be all the more difficult to achieve without targeting a global market. Sovereign Australia AI has remained firm about building local for local.
The founders have indicated no plans to chase global scale or go head-to-head with US tech giants.
“No, Australia is our market, and we need this to maintain our voice on the world stage,” Neilson said.
“Yet we hope that other sovereign nations around the globe see the amazing work that we’re doing and seek us out. We have the capability and smarts to help other nations.
“This would enable us to scale commercially beyond Australia, without jeopardising our sovereignty.”
As for paying creators, the startup is still considering different options.
It says that a sustainable model may be easier to structure with large organisations, which, as Neilson puts it, “have well-developed systems in place for content licensing and attribution”.
But for individual artists and writers, he acknowledges, the solution could “look to systems like those used by YouTube” to engage at scale.
“We are not saying we have all the answers, but we are open to working it out for the benefit of all Australians.”
- 11 minutes ago
- News
- Duration 6:32
As artificial intelligence becomes more commonplace, London Morning will be talking about its impact on our lives. Mark Daley, the chief artificial intelligence officer at Western University, will join host Andrew Brown once a month. Their first conversation focused on AI and education.
A new ethics-driven Australian AI venture, Sovereign Australia AI, has launched with an investment in commercial Australian AI infrastructure which the founders says is designed to keep our nation in control of its digital future. The company says it is committed to setting a new benchmark for transparency and ethics in artificial intelligence, with $10 million set aside to compensate copyright holders whose data is used to train its initial models. Sovereign Australia AI users will know how its models are built, what data they are trained on, and that they reflect Australian values.
Sovereign Australia AI was founded by Troy Neilson and Simon Kriss. Troy combines more than 20 years of startup and commercialisation experience with a PhD in AI and Machine Learning and has overseen over 100 AI-driven solutions across diverse industries. Simon is an AI strategist who has advised large corporates, government bodies and the United Nations on aligning AI technology with commercial and policy objectives. Sovereign Australia AI has also added Annette Kimmet AM, former CEO of the Victorian Gambling and Casino Control Commission to its board.
The company has placed Australia’s largest-ever order for sovereign AI capacity — 256 of the latest NVidia Blackwell B200 GPUs, to power model development at a scale previously unseen in Australia, creating a sovereign capability to create Large Language Models (LLMs) that are of a size that is on par with many foreign frontier models. The hardware will be hosted in secure, Australian-based data centres operated by NEXTDC, ensuring data sovereignty and compliance with Australian privacy and security standards.
Sovereign Australia AI says it has earmarked a minimum of $10 million to source copyrighted materials needed for its models. The company’s mission is to create AI models that are Australian owned, Australian controlled, and trained with ethically sourced datasets. Its upcoming Ginan and Australis models are designed to reflect the culture, language and principles of this nation, and to provide an alternative to offshore systems that may embed foreign values and biases.
To maximise transparency, Sovereign Australia AI says it will openly provide visibility of the data used to train its models. They will also open source the Ginan research model for free public use. This open and proactive stance sets a new benchmark for what is meant by the term ‘ethical AI’.
Sovereign Australia AI has also signed a memorandum of understanding with the Australian Computer Society (ACS). Under the agreement, Sovereign Australia AI will develop a bespoke AI capability for ACS, Australia’s peak professional body for the ICT sector.
“ACS is excited to collaborate with Sovereign Australia AI to help ensure Australia’s technology workforce has access to sovereign, ethical AI built with our nation’s values at its core,” said ACS CEO Josh Griggs.
Co-founder and CEO Simon Kriss said the launch marks a critical step towards securing Australia’s digital future.
“We are already seeing how AI is shaping the way people think, work and engage with information. If the foundational AI models Australians rely on are built offshore, we risk losing control over how our national values are represented,” he said. “Sovereign Australia AI will ensure we have a home-grown alternative that is ethical, transparent, and built for trust. We believe, as an industry, we must define what we mean when we say ‘ethical AI’ — simply saying you are ethical is not enough.
“We want to set that benchmark high — users should be able to understand what datasets the AI they use is trained on. They should be able to be aware of how the data has been curated. They should expect that copyright holders whose data was used to make the model more capable are compensated. That’s what we will bring to the table.”
Co-founder and CTO Troy Neilson said the focus is on sovereignty and trust rather than competition.
“Building sovereign AI capabilities here at home is a significant technological challenge, but we have the expertise in Australia to get the job done,” he said. “We are not trying to directly compete with ChatGPT or other global models, because we don’t need to. Instead, we are creating our own foundational models that will serve as viable alternatives that better capture the Australian voice.
“There are many public and private organisations in Australia who will greatly benefit from a truly sovereign AI solution, but do not want to sacrifice real-world performance. Our Ginan and Australis models will fill that gap with a capable, ethical alternative.”
Originally published
here.
-
Business2 weeks agoThe Guardian view on Trump and the Fed: independence is no substitute for accountability | Editorial
-
Tools & Platforms4 weeks ago
Building Trust in Military AI Starts with Opening the Black Box – War on the Rocks
-
Ethics & Policy1 month agoSDAIA Supports Saudi Arabia’s Leadership in Shaping Global AI Ethics, Policy, and Research – وكالة الأنباء السعودية
-
Events & Conferences4 months agoJourney to 1000 models: Scaling Instagram’s recommendation system
-
Jobs & Careers2 months agoMumbai-based Perplexity Alternative Has 60k+ Users Without Funding
-
Education2 months agoVEX Robotics launches AI-powered classroom robotics system
-
Podcasts & Talks2 months agoHappy 4th of July! 🎆 Made with Veo 3 in Gemini
-
Education2 months agoMacron says UK and France have duty to tackle illegal migration ‘with humanity, solidarity and firmness’ – UK politics live | Politics
-
Funding & Business2 months agoKayak and Expedia race to build AI travel agents that turn social posts into itineraries
-
Podcasts & Talks2 months agoOpenAI 🤝 @teamganassi