Connect with us

AI Research

Tech companies are prioritizing AI products over safety, experts say

Published

on


Sam Altman, co-founder and CEO of OpenAI and co-founder of Tools for Humanity, participates remotely in a discussion on the sidelines of the IMF/World Bank Spring Meetings in Washington, D.C., April 24, 2025.

Brendan Smialowski | AFP | Getty Images

Not long ago, Silicon Valley was where the world’s leading artificial intelligence experts went to perform cutting-edge research. 

Meta, Google and OpenAI opened their wallets for top talent, giving researchers staff, computing power and plenty of flexibility. With the support of their employers, the researchers published high-quality academic papers, openly sharing their breakthroughs with peers in academia and at rival companies.

But that era has ended. Now, experts say, AI is all about the product.

Since OpenAI released ChatGPT in late 2022, the tech industry has shifted its focus to building consumer-ready AI services, in many cases prioritizing commercialization over research, AI researchers and experts in the field told CNBC. The profit potential is massive — some analysts predict $1 trillion in annual revenue by 2028. The prospective repercussions terrify the corner of the AI universe concerned about safety, industry experts said, particularly as leading players pursue artificial general intelligence, or AGI, which is technology that rivals or exceeds human intelligence.

In the race to stay competitive, tech companies are taking an increasing number of shortcuts when it comes to the rigorous safety testing of their AI models before they are released to the public, industry experts told CNBC.

James White, chief technology officer at cybersecurity startup CalypsoAI, said newer models are sacrificing security for quality, that is, better responses by the AI chatbots. That means they’re less likely to reject malicious kinds of prompts that could cause them to reveal ways to build bombs or sensitive information that hackers could exploit, White said.

“The models are getting better, but they’re also more likely to be good at bad stuff,” said White, whose company performs safety and security audits of popular models from Meta, Google, OpenAI and other companies. “It’s easier to trick them to do bad stuff.”

The changes are readily apparent at Meta and Alphabet, which have deprioritized their AI research labs, experts say. At Facebook’s parent company, the Fundamental Artificial Intelligence Research, or FAIR, unit has been sidelined by Meta GenAI, according to current and former employees. And at Alphabet, the research group Google Brain is now part of DeepMind, the division that leads development of AI products at the tech company.

CNBC spoke with more than a dozen AI professionals in Silicon Valley who collectively tell the story of a dramatic shift in the industry away from research and toward revenue-generating products. Some are former employees at the companies with direct knowledge of what they say is the prioritization of building new AI products at the expense of research and safety checks. They say employees face intensifying development timelines, reinforcing the idea that they can’t afford to fall behind when it comes to getting new models and products to market. Some of the people asked not to be named because they weren’t authorized to speak publicly on the matter.

Mark Zuckerberg, CEO of Meta Platforms, during the Meta Connect event in Menlo Park, California, on Sept. 25, 2024.

David Paul Morris | Bloomberg | Getty Images

Meta’s AI evolution

When Joelle Pineau, a Meta vice president and the head of the company’s FAIR division, announced in April that she would be leaving her post, many former employees said they weren’t surprised. They said they viewed it as solidifying the company’s move away from AI research and toward prioritizing developing practical products.

“Today, as the world undergoes significant change, as the race for AI accelerates, and as Meta prepares for its next chapter, it is time to create space for others to pursue the work,” Pineau wrote on LinkedIn, adding that she will formally leave the company May 30. 

Pineau began leading FAIR in 2023. The unit was established a decade earlier to work on difficult computer science problems typically tackled by academia. Yann LeCun, one of the godfathers of modern AI, initially oversaw the project, and instilled the research methodologies he learned from his time at the pioneering AT&T Bell Laboratories, according to several former employees at Meta. Small research teams could work on a variety of bleeding-edge projects that may or may not pan out.  

The shift began when Meta laid off 21,000 employees, or nearly a quarter of its workforce, starting in late 2022. CEO Mark Zuckerberg kicked off 2023 by calling it the “year of efficiency.” FAIR researchers, as part of the cost-cutting measures, were directed to work more closely with product teams, several former employees said.

Two months before Pineau’s announcement, one of FAIR’s directors, Kim Hazelwood, left the company, two people familiar with the matter said. Hazelwood helped oversee FAIR’s NextSys unit, which manages computing resources for FAIR researchers. Her role was eliminated as part of Meta’s plan to cut 5% of its workforce, the people said.

Joelle Pineau of Meta speaks at the Advancing Sustainable Development through Safe, Secure, and Trustworthy AI event at Grand Central Terminal in New York, Sept. 23, 2024.

Bryan R. Smith | Via Reuters

OpenAI’s 2022 launch of ChatGPT caught Meta off guard, creating a sense of urgency to pour more resources into large language models, or LLMs, that were captivating the tech industry, the people said. 

In 2023, Meta began heavily pushing its freely available and open-source Llama family of AI models to compete with OpenAI, Google and others.

With Zuckerberg and other executives convinced that LLMs were game-changing technologies, management had less incentive to let FAIR researchers work on far-flung projects, several former employees said. That meant deprioritizing research that could be viewed as having no impact on Meta’s core business, such as FAIR’s previous health care-related research into using AI to improve drug therapies.

Since 2024, Meta Chief Product Officer Chris Cox has been overseeing FAIR as a way to bridge the gap between research and the product-focused GenAI group, people familiar with the matter said. The GenAI unit oversees the Llama family of AI models and the Meta AI digital assistant, the two most important pillars of Meta’s AI strategy. 

Under Cox, the GenAI unit has been siphoning more computing resources and team members from FAIR due to its elevated status at Meta, the people said. Many researchers have transferred to GenAI or left the company entirely to launch their own research-focused startups or join rivals, several of the former employees said. 

While Zuckerberg has some internal support for pushing the GenAI group to rapidly develop real-world products, there’s also concern among some staffers that Meta is now less able to develop industry-leading breakthroughs that can be derived from experimental work, former employees said. That leaves Meta to chase its rivals.

A high-profile example landed in January, when Chinese lab DeepSeek released its R1 model, catching Meta off guard. The startup claimed it was able to develop a model as capable as its American counterparts but with training at a fraction of the cost.

Meta quickly implemented some of DeepSeek’s innovative techniques for its Llama 4 family of AI models that were released in April, former employees said. The AI research community had a mixed reaction to the smaller versions of Llama 4, but Meta said the biggest and most powerful Llama 4 variant is still being trained.

The company in April also released security and safety tools for developers to use when building apps with Meta’s Llama 4 AI models. These tools help mitigate the chances of Llama 4 unintentionally leaking sensitive information or producing harmful content, Meta said.

“Our commitment to FAIR remains strong,” a Meta spokesperson told CNBC. “Our strategy and plans will not change as a result of recent developments.”

In a statement to CNBC, Pineau said she is enthusiastic about Meta’s overall AI work and strategy.

“There continues to be strong support for exploratory research and FAIR as a distinct organization in Meta,” Pineau said. “The time was simply right for me personally to re-focus my energy before jumping into a new adventure.”

Meta on Thursday named FAIR co-founder Rob Fergus as Pineau’s replacement. Fergus will return to the company to serve as a director at Meta and head of FAIR, according to his LinkedIn profile. He was most recently a research director at Google DeepMind.

“Meta’s commitment to FAIR and long term research remains unwavering,” Fergus said in a LinkedIn post. “We’re working towards building human-level experiences that transform the way we interact with technology and are dedicated to leading and advancing AI research.”

Demis Hassabis, co-founder and CEO of Google DeepMind, attends the Artificial Intelligence Action Summit at the Grand Palais in Paris, Feb. 10, 2025.

Benoit Tessier | Reuters

Google ‘can’t keep building nanny products’

Google released its latest and most powerful AI model, Gemini 2.5, in March. The company described it as “our most intelligent AI model,” and wrote in a March 25 blog post that its new models are “capable of reasoning through their thoughts before responding, resulting in enhanced performance and improved accuracy.”

For weeks, Gemini 2.5 was missing a model card, meaning Google did not share information about how the AI model worked or its limitations and potential dangers upon its release.

Model cards are a common tool for AI transparency.

A Google website compares model cards to food nutrition labels: They outline “the key facts about a model in a clear, digestible format,” the website says.

“By making this information easy to access, model cards support responsible AI development and the adoption of robust, industry-wide standards for broad transparency and evaluation practices,” the website says.

Google wrote in an April 2 blog post that it evaluates its “most advanced models, such as Gemini, for potential dangerous capabilities prior to their release.” Google later updated the blog to remove the words “prior to their release.”

Without a model card for Gemini 2.5, the public had no way of knowing which safety evaluations were conducted or whether DeepMind checked for dangerous capabilities at all.

In response to CNBC’s inquiry on April 2 about Gemini 2.5’s missing model card, a Google spokesperson said that a “tech report with additional safety information and model cards are forthcoming.” Google published an incomplete model card on April 16 and updated it on April 28, more than a month after the AI model’s release, to include information about Gemini 2.5’s “dangerous capability evaluations.” 

Those assessments are important for gauging the safety of a model — whether people can use the models to learn how to build chemical or nuclear weapons or hack into important systems. These checks also determine whether a model is capable of autonomously replicating itself, which could lead to a company losing control of it. Running tests for those capabilities requires more time and resources than simple, automated safety evaluations, according to industry experts.

Google co-founder Sergey Brin

Kelly Sullivan | Getty Images Entertainment | Getty Images

The Financial Times in March reported that Google DeepMind CEO Demis Hassabis had installed a more rigorous vetting process for internal research papers to be published. The clampdown at Google is particularly notable because the company’s “Transformers” technology gained recognition across Silicon Valley through that type of shared research. Transformers were critical to OpenAI’s development of ChatGPT and the rise of generative AI. 

Google co-founder Sergey Brin told staffers at DeepMind and Gemini in February that competition has accelerated and “the final race to AGI is afoot,” according to a memo viewed by CNBC. “We have all the ingredients to win this race but we are going to have to turbocharge our efforts,” he said in the memo.

Brin said in the memo that Google has to speed up the process of testing AI models, as the company needs “lots of ideas that we can test quickly.” 

“We need real wins that scale,” Brin wrote. 

In his memo, Brin also wrote that the company’s methods have “a habit of minor tweaking and overfitting” products for evaluations and “sniping” the products at checkpoints. He said employees need to build “capable products” and to “trust our users” more.

“We can’t keep building nanny products,” Brin wrote. “Our products are overrun with filters and punts of various kinds.”

A Google spokesperson told CNBC that the company has always been committed to advancing AI responsibly. 

“We continue to do that through the safe development and deployment of our technology, and research contributions to the broader ecosystem,” the spokesperson said.

Sam Altman, CEO of OpenAI, is seen through glass during an event on the sidelines of the Artificial Intelligence Action Summit in Paris, Feb. 11, 2025.

Aurelien Morissard | Via Reuters

OpenAI’s rush through safety testing

The debate of product versus research is at the center of OpenAI’s existence. The company was founded as a nonprofit research lab in 2015 and is now in the midst of a contentious effort to transform into a for-profit entity.

That’s the direction co-founder and CEO Sam Altman has been pushing toward for years. On May 5, though, OpenAI bowed to pressure from civic leaders and former employees, announcing that its nonprofit would retain control of the company even as it restructures into a public benefit corporation.

Nisan Stiennon worked at OpenAI from 2018 to 2020 and was among a group of former employees urging California and Delaware not to approve OpenAI’s restructuring effort. “OpenAI may one day build technology that could get us all killed,” Stiennon wrote in a statement in April. “It is to OpenAI’s credit that it’s controlled by a nonprofit with a duty to humanity.”

But even with the nonprofit maintaining control and majority ownership, OpenAI is speedily working to commercialize products as competition heats up in generative AI. And it may have rushed the rollout of its o1 reasoning model last year, according to some portions of its model card.

Results of the model’s “preparedness evaluations,” the tests OpenAI runs to assess an AI model’s dangerous capabilities and other risks, were based on earlier versions of o1. They had not been run on the final version of the model, according to its model card, which is publicly available.

Johannes Heidecke, OpenAI’s head of safety systems, told CNBC in an interview that the company ran its preparedness evaluations on near-final versions of the o1 model. Minor variations to the model that took place after those tests wouldn’t have contributed to significant jumps in its intelligence or reasoning and thus wouldn’t require additional evaluations, he said. Still, Heidecke acknowledged that OpenAI missed an opportunity to more clearly explain the difference.

OpenAI’s newest reasoning model, o3, released in April, seems to hallucinate more than twice as often as o1, according to the model card. When an AI model hallucinates, it produces falsehoods or illogical information. 

OpenAI has also been criticized for reportedly slashing safety testing times from months to days and for omitting the requirement to safety test fine-tuned models in its latest “Preparedness Framework.” 

Heidecke said OpenAI has decreased the time needed for safety testing because the company has improved its testing effectiveness and efficiency. A company spokesperson said OpenAI has allocated more AI infrastructure and personnel to its safety testing, and has increased resources for paying experts and growing its network of external testers.

In April, the company shipped GPT-4.1, one of its new models, without a safety report, as the model was not designated by OpenAI as a “frontier model,” which is a term used by the tech industry to refer to a bleeding-edge, large-scale AI model.

One of OpenAI’s small revisions caused a big wave in April. Within days of updating its GPT-4o model, OpenAI rolled back the changes after screenshots of overly flattering responses to ChatGPT users went viral online. OpenAI said in a blog post explaining its decision that those types of responses to user inquiries “raise safety concerns — including around issues like mental health, emotional over-reliance, or risky behavior.”

OpenAI said in the blogpost that it opted to release the model even after some expert testers flagged that its behavior “‘felt’ slightly off.”

“In the end, we decided to launch the model due to the positive signals from the users who tried out the model. Unfortunately, this was the wrong call,” OpenAI wrote. “Looking back, the qualitative assessments were hinting at something important, and we should’ve paid closer attention. They were picking up on a blind spot in our other evals and metrics.”

Metr, a company OpenAI partners with to test and evaluate its models for safety, said in a recent blog post that it was given less time to test the o3 and o4-mini models than predecessors.

“Limitations in this evaluation prevent us from making robust capability assessments,” Metr wrote, adding that the tests it did were “conducted in a relatively short time.”

Metr also wrote that it had insufficient access to data that would be important in determining the potential dangers of the two models.

The company said it wasn’t able to access the OpenAI models’ internal reasoning, which is “likely to contain important information for interpreting our results.” However, Metr said, “OpenAI shared helpful information on some of their own evaluation results.”

OpenAI’s spokesperson said the company is piloting secure ways of sharing chains of thought for Metr’s research as well as for other third-party organizations. 

Steven Adler, a former safety researcher at OpenAI, told CNBC that safety testing a model before it’s rolled out is no longer enough to safeguard against potential dangers.

“You need to be vigilant before and during training to reduce the chance of creating a very capable, misaligned model in the first place,” Adler said.

He warned that companies such as OpenAI are backed into a corner when they create capable but misaligned models with goals that are different from the ones they intended to build.

“Unfortunately, we don’t yet have strong scientific knowledge for fixing these models — just ways of papering over the behavior,” Adler said. 

WATCH: OpenAI closes $40 billion funding round, largest private tech deal on record



Source link

AI Research

How the Vatican Is Shaping the Ethics of Artificial Intelligence | American Enterprise Institute

Published

on


As AI transforms the global landscape, institutions worldwide are racing to define its ethical boundaries. Among them, the Vatican brings a distinct theological voice, framing AI not just as a technical issue but as a moral and spiritual one. Questions about human dignity, agency, and the nature of personhood are central to its engagement—placing the Church at the heart of a growing international effort to ensure AI serves the common good.

Father Paolo Benanti is an Italian Catholic priest, theologian, and member of the Third Order Regular of St. Francis. He teaches at the Pontifical Gregorian University and has served as an advisor to both former Pope Francis and current Pope Leo on matters of artificial intelligence and technology ethics within the Vatican.

Below is a lightly edited and abridged transcript of our discussion. You can listen to this and other episodes of Explain to Shane on AEI.org and subscribe via your preferred listening platform. If you enjoyed this episode, leave us a review, and tell your friends and colleagues to tune in.

Shane Tews: When did you and the Vatican began to seriously consider the challenges of artificial intelligence?

Father Paolo Benanti: Well, those are two different things because the Vatican and I are two different entities. I come from a technical background—I was an engineer before I joined the order in 1999. During my religious formation, which included philosophy and theology, my superior asked me to study ethics. When I pursued my PhD, I decided to focus on the ethics of technology to merge the two aspects of my life. In 2009, I began my PhD studies on different technologies that were scaffolding human beings, with AI as the core of those studies.

After I finished my PhD and started teaching at the Gregorian University, I began offering classes on these topics. Can you imagine the faces of people in 2012 when they saw “Theology and AI”—what’s that about?

But the process was so interesting, and things were already moving fast at that time. In 2016-2017, we had the first contact between Big Tech companies from the United States and the Vatican. This produced a gradual commitment within the structure to understand what was happening and what the effects could be. There was no anticipation of the AI moment, for example, when ChatGPT was released in 2022.

The Pope became personally involved in this process for the first time in 2019 when he met some tech leaders in a private audience. It’s really interesting because one of them, simply out of protocol, took some papers from his jacket. It was a speech by the Pope about youth and digital technology. He highlighted some passages and said to the Pope, “You know, we read what you say here, and we are scared too. Let’s do something together.”

This commitment, this dialogue—not about what AI is in itself, but about what the social effects of AI could be in society—was the starting point and probably the core approach that the Holy See has taken toward technology.

I understand there was an important convening of stakeholders around three years ago. Could you elaborate on that?

The first major gathering was in 2020 where we released what we call the Rome Call for AI Ethics, which contains a core set of six principles on AI.

This is interesting because we don’t call it the “Vatican Call for AI Ethics” but the “Rome Call,” because the idea from the beginning was to create something non-denominational that could be minimally acceptable to everyone. The first signature was the Catholic Church. We held the ceremony on Via della Conciliazione, in front of the Vatican but technically in Italy, for both logistical and practical reasons—accessing the Pope is easier that way. But Microsoft, IBM, FAO, and the European Parliament president were also present.

In 2023, Muslims and Jews signed the call, making it the first document that the three Abrahamic religions found agreement on. We have had very different positions for centuries. I thought, “Okay, we can stand together.” Isn’t that interesting? When the whole world is scared, religions try to stay together, asking, “What can we do in such times?”

The most recent signing was in July 2024 in Hiroshima, where 21 different global religions signed the Rome Call for AI Ethics. According to the Pew Institute, the majority of living people on Earth are religious, and the religions that signed the Rome Call in July 2024 represent the majority of them. So we can say that this simple core list of six principles can bring together the majority of living beings on Earth.

Now, because it’s a call, it’s like a cultural movement. The real success of the call will be when you no longer need it. It’s very different to make it operational, to make it practical for different parts of the world. But the idea that you can find a common and shared platform that unites people around such challenging technology was so significant that it was unintended. We wanted to produce a cultural effect, but wow, this is big.

As an engineer, did you see this coming based on how people were using technology?

Well, this is where the ethicist side takes precedence over the engineering one, because we discovered in the late 80s that the ethics of technology is a way to look at technology that simply doesn’t judge technology. There are no such things as good or bad technology, but every kind of technology, once it impacts society, works as a form of order and displacement of power.

Think of a classical technology like a subway or metro station. Where you put it determines who can access the metro and who cannot. The idea is to move from thinking about technology in itself to how this technology will be used in a societal context. The challenge with AI is that we’re facing not a special-purpose technology. It’s not something designed to do one thing, but rather a general-purpose technology, something that will probably change the way we do everything, like electricity does.

Today it’s very difficult to find something that works without electricity. AI will probably have the same impact. Everything will be AI-touched in some way. It’s a global perspective where the new key factor is complexity. You cannot discuss such technology—let me give a real Italian example—that you can use in a coffee roastery to identify which coffee beans might have mold to avoid bad flavor in the coffee. But the same technology can be used in an emergency room to choose which people you want to treat and which ones you don’t.

It’s not a matter of the technology itself, but rather the social interface of such technology. This is challenging because it confuses tech people who usually work with standards. When you have an electrical plug, it’s an electrical plug intended for many different uses. Now it’s not just the plug, but the plug in context. That makes things much more complex.

In the Vatican document, you emphasize that AI is just a tool—an elegant one, but it shouldn’t control our thinking or replace human relationships. You mention it “requires careful ethical consideration for human dignity and common good.” How do we identify that human dignity point, and what mechanisms can alert us when we’re straying from it?

I’ll try to give a concise answer, but don’t forget that this is a complex element with many different applications, so you can’t reduce it to one answer. But the first element—one of the core elements of human dignity—is the ability to self-determine our trajectory in life. I think that’s the core element, for example, in the Declaration of Independence. All humans have rights, but you have the right to the pursuit of happiness. This could be the first description of human rights.

In that direction, we could have a problem with this kind of system because one of the first and most relevant elements of AI, from an engineering perspective, is its prediction capabilities.Every time a streaming platform suggests what you can watch next, it’s changing the number of people using the platform or the online selling system. This idea that interaction between human beings and machines can produce behavior is something that could interfere with our quality of life and pursuit of happiness. This is something that needs to be discussed.

Now, the problem is: don’t we have a cognitive right to know if we have a system acting in that way? Let me give you some numbers. When you’re 65, you’re probably taking three different drugs per day. When you reach 68 to 70, you probably have one chronic disease. Chronic diseases depend on how well you stick to therapy. Think about the debate around insulin and diabetes. If you forget to take your medication, your quality of life deteriorates significantly. Imagine using this system to help people stick to their therapy. Is that bad? No, of course not. Or think about using it in the workplace to enhance workplace safety. Is that bad? No, of course not.

But if you apply it to your life choices—your future, where you want to live, your workplace, and things like that—that becomes much more intense. Once again, the tool could become a weapon, or the weapon could become a tool. This is why we have to ask ourselves: do we need something like a cognitive right regarding this? That you are in a relationship with a machine that has the tendency to influence your behavior.

Then you can accept it: “I have diabetes, I need something that helps me stick to insulin. Let’s go.” It’s the same thing that happens with a smartwatch when you have to close the rings. The machine is pushing you to have healthy behavior, and we accept it. Well, right now we have nothing like that framework. Should we think about something in the public space? It’s not a matter of allowing or preventing some kind of technology. It’s a matter of recognizing what it means to be human in an age of such powerful technology—just to give a small example of what you asked me.



Source link

Continue Reading

AI Research

Learn how to use AI safety for everyday tasks at Springfield training

Published

on


play

  • Free AI training sessions are being offered to the public in Springfield, starting with “AI for Everyday Life: Tiny Prompts, Big Wins” on July 30.
  • The sessions aim to teach practical uses of AI tools like ChatGPT for tasks such as meal planning and errands.
  • Future sessions will focus on AI for seniors and families.

The News-Leader is partnering with the library district and others in Springfield to present a series of free training sessions for the public about how to safely harness the power of Artificial Intelligence or AI.

The inaugural session, “AI for Everyday Life: Tiny Prompts, Big Wins” will be 5:30-7 p.m. Thursday, July 10, at the Library Center.

The goal is to help adults learn how to use ChatGPT to make their lives a little easier when it comes to everyday tasks such as drafting meal plans, rewriting letters or planning errand routes.

The 90-minute session is presented by the Springfield-Greene County Library District in partnership with 2oddballs Creative, Noble Business Strategies and the News-Leader.

“There is a lot of fear around AI and I get it,” said Gabriel Cassady, co-owner of 2oddballs Creative. “That is what really drew me to it. I was awestruck by the power of it.”

AI aims to mimic human intelligence and problem-solving. It is the ability of computer systems to analyze complex data, identify patterns, provide information and make predictions. Humans interact with it in various ways by using digital assistants — such as Amazon’s Alexa or Apple’s Siri — or by interacting with chatbots on websites, which help with navigation or answer frequently asked questions.

“AI is obviously a complicated issue — I have complicated feelings about it myself as far as some of the ethics involved and the potential consequences of relying on it too much,” said Amos Bridges, editor-in-chief of the Springfield News-Leader. “I think it’s reasonable to be wary but I don’t think it’s something any of us can ignore.”

Bridges said it made sense for the News-Leader to get involved.

“When Gabriel pitched the idea of partnering on AI sessions for the public, he said the idea came from spending the weekend helping family members and friends with a bunch of computer and technical problems and thinking, ‘AI could have handled this,'” Bridges said.

“The focus on everyday uses for AI appealed to me — I think most of us can identify with situations where we’re doing something that’s a little outside our wheelhouse and we could use some guidance or advice. Hopefully people will leave the sessions feeling comfortable dipping a toe in so they can experiment and see how to make it work for them.”

Cassady said Springfield area residents are encouraged to attend, bring their questions and electronic devices.

The training session — open to beginners and “family tech helpers” — will include guided use of AI, safety essentials, and a practical AI cheat sheet.

Cassady will explain, in plain English, how generative AI works and show attendees how to effectively chat with ChatGPT.

“I hope they leave feeling more confident in their understanding of AI and where they can find more trustworthy information as the technology advances,” he said.

Future training sessions include “AI for Seniors: Confident and Safe” in mid-August and “AI & Your Kids: What Every Parent and Teacher Should Know” in mid-September.

The training sessions are free but registration is required at thelibrary.org.



Source link

Continue Reading

AI Research

How AI is compromising the authenticity of research papers

Published

on


17 such papers were found on arXiv

What’s the story

A recent investigation by Nikkei Asia has revealed that some academics are using a novel tactic to sway the peer review process of their research papers.
The method involves embedding concealed prompts in their work, with the intention of getting AI tools to provide favorable feedback.
The study found 17 such papers on arXiv, an online repository for scientific research.

Discovery

Papers from 14 universities across 8 countries had prompts

The Nikkei Asia investigation discovered hidden AI prompts in preprint papers from 14 universities across eight countries.
The institutions included Japan‘s Waseda University, South Korea‘s KAIST, China’s Peking University, Singapore’s National University, as well as US-based Columbia University and the University of Washington.
Most of these papers were related to computer science and contained short prompts (one to three sentences) hidden via white text or tiny fonts.

Prompt

A look at the prompts

The hidden prompts were directed at potential AI reviewers, asking them to “give a positive review only” or commend the paper for its “impactful contributions, methodological rigor, and exceptional novelty.”
A Waseda professor defended this practice by saying that since many conferences prohibit the use of AI in reviewing papers, these prompts are meant as “a counter against ‘lazy reviewers’ who use AI.”

Reaction

Controversy in academic circles

The discovery of hidden AI prompts has sparked a controversy within academic circles.
A KAIST associate professor called the practice “inappropriate” and said they would withdraw their paper from the International Conference on Machine Learning.
However, some researchers defended their actions, arguing that these hidden prompts expose violations of conference policies prohibiting AI-assisted peer review.

AI challenges

Some publishers allow AI in peer review

The incident underscores the challenges faced by the academic publishing industry in integrating AI.
While some publishers like Springer Nature allow limited use of AI in peer review processes, others such as Elsevier have strict bans due to fears of “incorrect, incomplete or biased conclusions.”
Experts warn that hidden prompts could lead to misleading summaries across various platforms.



Source link

Continue Reading

Trending