Connect with us

AI Research

Is artificial intelligence a friend, foe or frenemy? NIST wants to find out

Published

on


Artificial intelligence is fast becoming cybersecurity’s ultimate double agent. The same tools that help defenders spot anomalies, detect intrusions and speed up response times are also making it easier for adversaries to scale and automate their attacks. AI can generate code to patch vulnerabilities, or it can exploit vulnerabilities whenever it finds them. It can help identify phishing attempts or draft realistic attacks designed to trick even the most savvy recipients. That dual-use nature of AI is exactly what the National Institute of Standards and Technology is trying to untangle in a series of virtual working sessions that bring government and industry experts together.

The first two NIST Cybersecurity Artificial Intelligence Profile working sessions were held in August and were extremely interesting and informative, with lots of experts from government, industry and education participating. The first broke down the various ways that AIs function, as well as how to secure them. The second was all about how to build up AI-empowered cyber defenses. The entire series is virtual and open to anyone.

The final session is scheduled for September 2, and this time, they are addressing the elephant in the room: how AI-empowered attacks can be used to sometimes get around traditional defenses. According to NIST, this final working session will also cover how agencies and organizations can build resilience in the face AI-enabled attacks. That concern was actually brought up in the two previous sessions, so it’s certainly a concern for both government and the private sector.

In their post-session blogs, NIST researchers noted that while AI is improving the speed and scale of defensive tools, it’s also doing the same for attacks. Typical attacks like phishing campaigns, data poisoning and model inversions are no longer labor-intensive operations when AI can do most of the work. Adversaries have also started experimenting with agentic AI, which can autonomously adapt and execute multi-step campaigns with little human intervention. And its skill in executing those kinds of advanced attacks is increasing.

Agentic AI, with its ability to autonomously perform complex, multi-step operations and adapt its tactics in the middle of an attack, could prove to be particularly dangerous in the near future. And it’s already being used for offensive operations. A recent report by Palo Alto Network’s Unit 42 detailed how agentic AI can be used to increase the speed, scale and sophistication of attacks. Unit 42 simulated ransomware and data exfiltration attacks for their study, letting the agentic AI learn about and adapt to whatever defenses it encountered. In many cases, the AI was able to thwart or trick most traditional defenses. The average time it took an agentic AI to break in and start exfiltrating data was just 25 minutes, 100 times faster than with a normal, non-AI enhanced attack.

So, how can defenses protect against AI-empowered attacks? Hopefully that will be answered or at least discussed at the next workshop. But in the meantime, NIST and other experts say that the key to stopping AI-enhanced attacks is a proactive defense. Automated red teaming, zero-trust principles, better identity controls and tight privilege management all become critical in a world where AI adversaries don’t need to rest or sleep. But the technology itself isn’t enough. The human factor matters too.

Consider AI in software development. A recent study conducted by researchers from the University of San Francisco, the Vector Institute for Artificial Intelligence in Toronto and the University of Massachusetts Boston analyzed 400 code samples across 40 rounds of improvements using four prompting strategies. One even explicitly asked the test AIs to improve the security of existing code by eliminating vulnerabilities. The results were not good. After just five rounds of AI changes, there was a 37.6% increase in critical vulnerabilities that other AIs could easily exploit. And with more iterations, meaning each time an AI was asked to work with the source code, the problems got worse.

That study’s recommendations came down to one recurring theme: human oversight. Developers should review all code iterations, use automated tools as supplements, limit consecutive AI-driven refinements and keep an eye on code complexity. That means organizations should invest in training so that developers are equipped with the security skills needed in today’s AI-driven environments. Otherwise, it just makes it too easy for agentic AI to exploit code created by its AI brethren.

The Open Worldwide Application Security Project also recently warned about a new concern involving agentic AI: the growing presence of agentic AI variations of existing threats. These include very dangerous and effective attack techniques like memory poisoning, misuse of integrated tools and privilege escalations whenever an AI agent acts on behalf of a human user. When agentic AI is asked to execute those kinds of attacks, it does so extremely quickly and with great skill. According to OWASP, the best defense right now is to employ strong identity controls and strict privilege management, along with constant monitoring to detect unusual behavior.

The challenge with AI right now seems to be trying to tip the scales toward the defender’s side. And that is what makes NIST’s upcoming Thwarting AI Enabled Cyber Attacks workshop so important. It’s not just about patching today’s vulnerabilities, but also about preparing for tomorrow’s threats. And in the fast-moving world of AI, tomorrow might already be here.

John Breeden II is an award-winning journalist and reviewer with over 20 years of experience covering technology. He is the CEO of the Tech Writers Bureau, a group that creates technological thought leadership content for organizations of all sizes. Twitter: @LabGuys





Source link

AI Research

Physicians Lose Cancer Detection Skills After Using Artificial Intelligence

Published

on


Artificial intelligence shows great promise in helping physicians improve both their diagnostic accuracy of important patient conditions. In the realm of gastroenterology, AI has been shown to help human physicians better detect small polyps (adenomas) during colonoscopy. Although adenomas are not yet cancerous, they are at risk for turning into cancer. Thus, early detection and removal of adenomas during routine colonoscopy can reduce patient risk of developing future colon cancers.

But as physicians become more accustomed to AI assistance, what happens when they no longer have access to AI support? A recent European study has shown that physicians’ skills in detecting adenomas can deteriorate significantly after they become reliant on AI.

The European researchers tracked the results of over 1400 colonoscopies performed in four different medical centers. They measured the adenoma detection rate (ADR) for physicians working normally without AI vs. those who used AI to help them detect adenomas during the procedure. In addition, they also tracked the ADR of the physicians who had used AI regularly for three months, then resumed performing colonoscopies without AI assistance.

The researchers found that the ADR before AI assistance was 28% and with AI assistance was 28.4%. (This was a slight increase, but not statistically significant.) However, when physicians accustomed to AI assistance ceased using AI, their ADR fell significantly to 22.4%. Assuming the patients in the various study groups were medically similar, that suggests that physicians accustomed to AI support might miss over a fifth of adenomas without computer assistance!

This is the first published example of so-called medical “deskilling” caused by routine use of AI. The study authors summarized their findings as follows: “We assume that continuous exposure to decision support systems such as AI might lead to the natural human tendency to over-rely on their recommendations, leading to clinicians becoming less motivated, less focused, and less responsible when making cognitive decisions without AI assistance.”

Consider the following non-medical analogy: Suppose self-driving car technology advanced to the point that cars could safely decide when to accelerate, brake, turn, change lanes, and avoid sudden unexpected obstacles. If you relied on self-driving technology for several months, then suddenly had to drive without AI assistance, would you lose some of your driving skills?

Although this particular study took place in the field of gastroenterology, I would not be surprised if we eventually learn of similar AI-related deskilling in other branches of medicine, such as radiology. At present, radiologists do not routinely use AI while reading mammograms to detect early breast cancers. But when AI becomes approved for routine use, I can imagine that human radiologists could succumb to a similar performance loss if they were suddenly required to work without AI support.

I anticipate more studies will be performed to investigate the issue of deskilling across multiple medical specialties. Physicians, policymakers, and the general public will want to ask the following questions:

1) As AI becomes more routinely adopted, how are we tracking patient outcomes (and physician error rates) before AI, after routine AI use, and whenever AI is discontinued?

2) How long does the deskilling effect last? What methods can help physicians minimize deskilling, and/or recover lost skills most quickly?

3) Can AI be implemented in medical practice in a way that augments physician capabilities without deskilling?

Deskilling is not always bad. My 6th grade schoolteacher kept telling us that we needed to learn long division because we wouldn’t always have a calculator with us. But because of the ubiquity of smartphones and spreadsheets, I haven’t done long division with pencil and paper in decades!

I do not see AI completely replacing human physicians, at least not for several years. Thus, it will be incumbent on the technology and medical communities to discover and develop best practices that optimize patient outcomes without endangering patients through deskilling. This will be one of the many interesting and important challenges facing physicians in the era of AI.



Source link

Continue Reading

AI Research

AI exposes 1,000+ fake science journals

Published

on


A team of computer scientists led by the University of Colorado Boulder has developed a new artificial intelligence platform that automatically seeks out “questionable” scientific journals.

The study, published Aug. 27 in the journal “Science Advances,” tackles an alarming trend in the world of research.

Daniel Acuña, lead author of the study and associate professor in the Department of Computer Science, gets a reminder of that several times a week in his email inbox: These spam messages come from people who purport to be editors at scientific journals, usually ones Acuña has never heard of, and offer to publish his papers — for a hefty fee.

Such publications are sometimes referred to as “predatory” journals. They target scientists, convincing them to pay hundreds or even thousands of dollars to publish their research without proper vetting.

“There has been a growing effort among scientists and organizations to vet these journals,” Acuña said. “But it’s like whack-a-mole. You catch one, and then another appears, usually from the same company. They just create a new website and come up with a new name.”

His group’s new AI tool automatically screens scientific journals, evaluating their websites and other online data for certain criteria: Do the journals have an editorial board featuring established researchers? Do their websites contain a lot of grammatical errors?

Acuña emphasizes that the tool isn’t perfect. Ultimately, he thinks human experts, not machines, should make the final call on whether a journal is reputable.

But in an era when prominent figures are questioning the legitimacy of science, stopping the spread of questionable publications has become more important than ever before, he said.

“In science, you don’t start from scratch. You build on top of the research of others,” Acuña said. “So if the foundation of that tower crumbles, then the entire thing collapses.”

The shake down

When scientists submit a new study to a reputable publication, that study usually undergoes a practice called peer review. Outside experts read the study and evaluate it for quality — or, at least, that’s the goal.

A growing number of companies have sought to circumvent that process to turn a profit. In 2009, Jeffrey Beall, a librarian at CU Denver, coined the phrase “predatory” journals to describe these publications.

Often, they target researchers outside of the United States and Europe, such as in China, India and Iran — countries where scientific institutions may be young, and the pressure and incentives for researchers to publish are high.

“They will say, ‘If you pay $500 or $1,000, we will review your paper,'” Acuña said. “In reality, they don’t provide any service. They just take the PDF and post it on their website.”

A few different groups have sought to curb the practice. Among them is a nonprofit organization called the Directory of Open Access Journals (DOAJ). Since 2003, volunteers at the DOAJ have flagged thousands of journals as suspicious based on six criteria. (Reputable publications, for example, tend to include a detailed description of their peer review policies on their websites.)

But keeping pace with the spread of those publications has been daunting for humans.

To speed up the process, Acuña and his colleagues turned to AI. The team trained its system using the DOAJ’s data, then asked the AI to sift through a list of nearly 15,200 open-access journals on the internet.

Among those journals, the AI initially flagged more than 1,400 as potentially problematic.

Acuña and his colleagues asked human experts to review a subset of the suspicious journals. The AI made mistakes, according to the humans, flagging an estimated 350 publications as questionable when they were likely legitimate. That still left more than 1,000 journals that the researchers identified as questionable.

“I think this should be used as a helper to prescreen large numbers of journals,” he said. “But human professionals should do the final analysis.”

A firewall for science

Acuña added that the researchers didn’t want their system to be a “black box” like some other AI platforms.

“With ChatGPT, for example, you often don’t understand why it’s suggesting something,” Acuña said. “We tried to make ours as interpretable as possible.”

The team discovered, for example, that questionable journals published an unusually high number of articles. They also included authors with a larger number of affiliations than more legitimate journals, and authors who cited their own research, rather than the research of other scientists, to an unusually high level.

The new AI system isn’t publicly accessible, but the researchers hope to make it available to universities and publishing companies soon. Acuña sees the tool as one way that researchers can protect their fields from bad data — what he calls a “firewall for science.”

“As a computer scientist, I often give the example of when a new smartphone comes out,” he said. “We know the phone’s software will have flaws, and we expect bug fixes to come in the future. We should probably do the same with science.”

Co-authors on the study included Han Zhuang at the Eastern Institute of Technology in China and Lizheng Liang at Syracuse University in the United States.



Source link

Continue Reading

AI Research

The Artificial Intelligence Is In Your Home, Office And The IRS Edition

Published

on




Source link

Continue Reading

Trending