To gain the U.S. government’s trust, advanced AI systems must be engineered from the outset with reliable components offering explainability and transparency, senior federal and industry officials said Friday.

“This [topic] is something I think about a lot,” the CIA’s chief AI officer Lakshmi Raman noted at the Billington Cybersecurity Summit. “And in our [community], it’s about how artificial intelligence can assist and be an intelligence amplifier with the human during the process, keeping their eyes on everything that’s happening and ensuring that, at the end, they’re able to help.”

During a panel, Raman and other current and former government officials underscored the importance of guardrails and oversight — particularly as the U.S. military and IC adopt the technology for an ever-increasing range of operations, and experts predict major breakthroughs will emerge in certain areas within the next few years.

“Trust is such a critical dimension for intelligence,” said Sean Batir, a National Geospatial-Intelligence Agency alum and AWS principal tech lead for frontier AI, quantum and robotics.

Frontier AI refers to next-generation systems, also dubbed foundation models, that are considered among the most powerful and complex technologies currently in development. These likely disruptive capabilities hold potential to unlock discoveries that could be immensely helpful or catastrophically harmful to humanity. 

Departments across the government have been expanding their use of AI and machine learning over the past several years, but defense and national security agencies were some of the earliest adopters. Recently, in July, questions started swirling after the Pentagon’s Chief Digital and AI Office (CDAO) revealed new, separate deals with xAI, Google, Anthropic and OpenAI to accelerate the enterprise- and military-wide deployment of frontier AI. 

“ChatGPT, our flagship product, has upwards of 800 million users every day. So one-tenth of the world is using ChatGPT in various forms,” said Joseph Larson, vice president and head of government business at OpenAI. “At an individual level, AI is there. The more challenging question [for] my job is, with government [use], what does AI adoption look like at an institutional level?”

Larson previously served from 2022 to 2024 as the Pentagon’s first-ever deputy chief digital and AI officer for algorithmic warfare. 

“When we talk about institutions, what does that require above and beyond just access to the technology and the foundation models? It requires, really, a partnership. And that partnership extends to questions around infrastructure, around data, and I think, key, around security,” he said. “And what are the security implications for AI as it moves from just something that you communicate with, that informs maybe a workflow, to something that’s part of an agentic system that’s actually operating in your environment and that has its own controls and authorities? So, those institutional challenges are really the ones that are driving our work within the government today.”

Both OpenAI and Anthropic have reportedly disclosed recent efforts to implement new guardrails because their models appear to be approaching high-risk levels for potentially helping produce certain weapons.

On the panel, Anthropic Chief Information Security Officer Jason Clinton noted that “trust is something that is built up over time.”

“In order to do that, there is a human — and there is a supervisory role — for these models. The one thing that those models will never be able to do is to bring humanity to the equation, right? We will. We will always need to bring our perspective, our values, our institutional wisdom, to what we’re asking the models to be doing,” Clinton said.

He and the other panelists spotlighted multiple risks and threats posed by emerging frontier AI applications. For instance, prompt injections are a type of cyberattack that happen when malicious users craft inputs to an AI system to trick the model into performing unintended or dangerous actions, such as revealing sensitive data or generating unsafe material.

“I’m very optimistic that we will solve some of the more fundamental guardrail problems — like prompt injection — within three-ish years, I guess,” Clinton said. “And the models will be getting smarter, so I suspect the ways that we interact with them will evolve towards more like having a virtual coworker beside you, who you interact with and who learns and adapts … and sort of grows with you in your environment.”

The panelists also discussed the potential power of cutting-edge AI to help reduce vulnerabilities in software by automatically finding and fixing bugs in code and zero-day exploits.

“DARPA just ran a competition at DefCon [hacking conference] that demonstrated the future possibilities there,” said Dr. Kathleen Fisher, director of that agency’s Information Innovation Office.

For the event, officials pulled 54 million lines of code across 20 different repositories that were recommended by critical infrastructure operators who use them to do their daily business. 

“The teams that ran the competition planted 70 systemic vulnerabilities that were patterned after real vulnerabilities that people have struggled with. The teams found 54 of those systemic vulnerabilities, and they patched 43 of them. More importantly, or at the same time, they found 18 zero-days, and they patched 11 of those. It took about 45 minutes to find and fix vulnerability at a cost of $152. Think about what that might mean in the future — like this is the worst that technology is ever going to be,” Fisher said. “Think about what that might mean in the context of things like Volt Typhoon, and Salt Typhoon, and ransomware that is currently plaguing our hospitals. When a hospital gets affected by a ransomware attack — when it shuts down for any period of time — that puts people’s lives at risk.”

Building on that, Microsoft Federal Chief Technology Officer Jason Payne added: “This is the worst version of the technology we will ever use. I also feel like we have the lowest amount of trust in technology, right? And I think if we all use it more, if we experience it more, we’ll sort of understand what it is and what it’s capable of.”

He continued: “Security, governance and explainability are key themes that we’re looking for to kind of build that trust. And at the end of the day, I think government agencies are looking for organizations that are transparent with their AI systems.”

The unemployment rate has ticked up in recent months and layoffs have been in the headlines all year. There are a plethora of reasons behind that labor market shift, but one that’s been cited directly by some companies like Salesforce, Indeed and Glassdoor is artificial intelligence adoption.

While it’s clear that AI is rapidly changing many aspects of our economic landscape, it remains to be seen what the technology’s ultimate impact on the labor market will actually be.

Jeff Dixon is a sociology professor at College of the Holy Cross. He wrote a piece for The Conversation titled, “The US really is unlike other rich countries when it comes to job insecurity — and AI could make it even more ‘exceptional.’” He joined “Marketplace” host Kimberly Adams to discuss his concerns with AI’s long-term effects on the stability of the American worker.

To hear their conversation, use the audio player above.