AI Research
Safeguarding Third-Party AI Research | Stanford HAI
Key Takeaways
-
Third-party AI research is essential to ensure that AI companies do not grade their own homework, but few companies actively protect or promote such research.
-
We found no major foundation model developers currently offer comprehensive protections for third-party evaluation. Instead, their policies often disincentivize it.
-
A safe harbor for good-faith research should be a top priority for policymakers. It enables good-faith research and increases the scale, diversity, and independence of evaluations.
Executive Summary
Third-party evaluation is a cornerstone of efforts to reduce the substantial risks posed by AI systems. AI is a vast field with thousands of highly specialized experts around the world who can help stress-test the most powerful systems. But few companies empower these researchers to test their AI systems, for fear of exposing flaws in their products. AI companies often block safety research with restrictive terms of service or by suspending researchers who report flaws.
In our paper, “A Safe Harbor for AI Evaluation and Red Teaming,” we assess the policies and practices of seven top developers of generative AI systems, finding that none offers comprehensive protections for third-party AI research. Unlike with cybersecurity, generative AI is a new field without well-established norms regarding flaw disclosure, safety standards, or mechanisms for conducting third-party research. We propose that developers adopt safe harbors to enable good-faith, adversarial testing of AI systems.
Introduction
Generative AI systems pose a wide range of potential risks, from enabling the creation of nonconsensual intimate imagery to facilitating the development of malware. Evaluating generative AI systems is crucial to understanding the technology, ensuring public accountability, and reducing these risks.
In July 2023, many prominent AI companies signed voluntary commitments at the White House, pledging to “incent third-party discovery and reporting of issues and vulnerabilities.” More than a year later, implementation of this commitment has been uneven. While some companies do reward researchers for finding security flaws in their AI systems, few companies strongly encourage research on safety or provide concrete protections for good-faith research practices. Instead, leading generative AI companies’ terms of service legally prohibit third-party safety and trustworthiness research, in effect threatening anyone who conducts such research with bans from their platforms or even legal action. For example, companies’ policies do not allow researchers to jailbreak AI systems like ChatGPT, Claude, or Gemini to assess potential threats to U.S. national security.
In March 2024, we penned an open letter signed by over 350 leading AI researchers and advocates calling for a safe harbor for third-party AI evaluation. The researchers noted that while security research on traditional software is protected by voluntary company protections (safe harbors), established vulnerability disclosure norms, and legal safeguards from the Department of Justice, AI safety and trustworthiness research lacks comparable protections.
Companies have continued to be opaque about key aspects of their most powerful AI systems, such as the data used to build their models. Developers of generative AI models tout the safety of their systems based on internal red teaming, but there is no way for the government or independent researchers to validate these results, as companies do not release reproducible evaluations.
Generative AI companies also impose barriers on their platforms that limit good-faith research. Similar issues plague social media: Companies have taken steps to prevent researchers and journalists from conducting investigations on their platforms that, together with federal legislation, have had a chilling effect on such research and worsened the spread of harmful content online. But conducting research on generative AI systems comes with additional challenges, as the content on generative AI platforms is not publicly available. Users need accounts to access AI-generated content, which can be restricted by the company that owns the platform. Many AI companies also block certain user requests and limit the functionality of their models to prevent researchers from unearthing issues related to safety or trustworthiness. The stakes are also higher for AI, which has the potential not only to turbocharge misinformation but also to provide U.S. adversaries like China and Russia with material strategic advantages.
To assess the state of independent evaluation for generative AI, our team of machine learning, law, and policy experts conducted a thorough review of seven major AI companies’ policies, access provisions, and related enforcement processes. We detail our experiences with evaluation of AI systems and potential barriers other third-party evaluators may face, and propose alternative practices and policies to enable broader community participation in AI evaluation.
The National Science Foundation plans to award up to $35 million to establish an operations center for its National AI Research Resource, signaling a step toward the pilot becoming a more permanent program.
Despite bipartisan support for the NAIRR, Congress has yet to authorize a full-scale version of the resource designed to democratize access to tools needed for AI research. The newly announced solicitation indicates the project is taking steps to scale the project absent additional support.
“The NAIRR Operating Center solicitation marks a key step in the transition from the NAIRR Pilot to building a sustainable and scalable NAIRR program,” Katie Antypas, who leads NSF’s Office of Advanced Cyberinfrastructure, said in a statement included in the announcement.
She added that NSF looks forward to collaborating with partners in the private sector and other agencies, “whose contributions have been critical in demonstrating the innovation and scientific impact that comes when critical AI resources are made accessible to research and education communities across the country.”
The NAIRR began as a pilot in January 2024 as a resource for researchers to access computational data, AI models, software, and other tools that are needed for AI research. Since then, the public-private partnership pilot has supported over 490 projects in 49 states and Washington, per its website, and is supported by contributions from 14 federal agencies and 28 private sector partners.
As the pilot has moved forward, lawmakers have attempted to advance bipartisan legislation that would codify the NAIRR, but those bills have not passed. Previous statements from science and tech officials during the Biden administration made the case that formalization would be important as establishing NAIRR fully was expected to take a significant amount of funding.
In response to a FedScoop question about funding for the center, an NSF spokesperson said it’s covered by the agency’s normal appropriations.
NAIRR has remained a priority even as the Trump administration has sought to make changes to NSF awards, canceling hundreds of grants that were related to things like diversity, equity and inclusion (DEI) and environmental justice. President Donald Trump’s AI Action Plan, for example, included a recommendation for the NAIRR to “build the foundations for a lean and sustainable NAIRR operations capability.”
According to the solicitation, NSF will make an award of a maximum of $35 million for a period of up to five years for the operations center project. That award will be made to a single organization. That awardee would ultimately be responsible for establishing a “community-based organization,” including tasks such as establishing the operation framework, working with stakeholders, and coordinating with the current pilot project functions.
The awardee would also be eligible to expand their responsibilities and duties at a later date, depending on factors such as NAIRR’s priorities, the awardee’s performance and funding.
AI Research
Top AI Code Generation Tools of 2025 Revealed in Info-Tech Research Group’s Emotional Footprint Report
The recently published 2025 AI Code Generation Emotional Footprint report from Info-Tech Research Group highlights the top AI code generation solutions that help organizations streamline development and support innovation. The report’s insights are based on feedback from users on the global IT research and advisory firm’s SoftwareReviews platform.
TORONTO, Sept. 3, 2025 /PRNewswire/ – Info-Tech Research Group has published its 2025 AI Code Generation Emotional Footprint report, identifying the top-performing solutions in the market. Based on data from SoftwareReviews, a division of the global IT research and advisory firm, the newly published report highlights the five champions in AI-powered code generation tools.
Info-Tech Research Group’s 2025 AI Code Generation Emotional Footprint report identifies leading solutions that enable organizations to accelerate development and drive innovation. (CNW Group/Info-Tech Research Group)
AI code generation tools make coding easier by taking care of repetitive tasks. Instead of starting from scratch, developers get ready-made snippets, smoother workflows, and support built right into their IDEs and version control systems. With machine learning and natural language processing behind them, these tools reduce mistakes, speed up projects, and give developers more room to focus on creative problem solving and innovation.
Info-Tech’s Emotional Footprint measures high-level user sentiment. It aggregates emotional response ratings across 25 proactive questions, creating a powerful indicator of overall user feeling toward the vendor and product. The result is the Net Emotional Footprint, or NEF, a composite score that reflects the overall emotional tone of user feedback.
Data from 1,084 end-user reviews on Info-Tech’s SoftwareReviews platform was used to identify the top AI code generation tools for the 2025 Emotional Footprint report. The insights support organizations looking to streamline development, improve code quality, and scale their software delivery capabilities to drive innovation and business growth.
The 2025 AI Code Generation Tools – Champions are as follows:
- Visual Studio IntelliCode, +96 NEF, ranked high for delivering more than promised.
- ChatGPT 5, +94 NEF, ranked high for its effectiveness.
- GitHub Copilot, +94 NEF, ranked high for its transparency.
- Replit AI, +96 NEF, ranked high for its reliability.
- Amazon Q Developer, +94 NEF, ranked high for helping save time.
Analyst Insight:
“Organizations that adopt AI code generation tools gain a significant advantage in software delivery and innovation,” says Thomas Randall, a research director at Info-Tech Research Group. “These tools help developers focus on complex, high-value work, improve code quality, and reduce errors. Teams that delay adoption risk slower projects, lower-quality software, and missed opportunities to innovate and stay competitive.”
User assessments of software categories on SoftwareReviews provide an accurate and detailed view of the constantly changing market. Info-Tech’s reports are informed by the data from users and IT professionals who have intimate experience with the software throughout the procurement, implementation, and maintenance processes.
Read the full report: Best AI Code Generation Tools 2025
For more information about Info-Tech’s SoftwareReviews, the Data Quadrant, or the Emotional Footprint, or to access resources to support the software selection process, visit softwarereviews.com.
About Info-Tech Research Group
Info-Tech Research Group is one of the world’s leading research and advisory firms, proudly serving over 30,000 IT and HR professionals. The company produces unbiased, highly relevant research and provides advisory services to help leaders make strategic, timely, and well-informed decisions. For nearly 30 years, Info-Tech has partnered closely with teams to provide them with everything they need, from actionable tools to analyst guidance, ensuring they deliver measurable results for their organizations.
To learn more about Info-Tech’s divisions, visit McLean & Company for HR research and advisory services and SoftwareReviews for software buying insights.
Media professionals can register for unrestricted access to research across IT, HR, and software, and hundreds of industry analysts through the firm’s Media Insiders program. To gain access, contact [email protected].
For information about Info-Tech Research Group or to access the latest research, visit infotech.com and connect via LinkedIn and X.
About SoftwareReviews
SoftwareReviews is a division of Info-Tech Research Group, a world-class technology research and advisory firm. SoftwareReviews empowers organizations with the best data, insights, and advice to improve the software buying and selling experience.
For buyers, SoftwareReviews’ proven software selection methodologies, customer insights, and technology advisors help maximize success with technology decisions. For providers, the firm helps build more effective marketing, product, and sales processes with expert analysts, how-to research, customer-centric marketing content, and comprehensive analysis of the buyer landscape.
SOURCE Info-Tech Research Group
Vanderbilt University has established the Enterprise AI and Computing Innovation Studio, a groundbreaking collaboration between VUIT, the Amplify Generative AI Innovation Center and the Data Science Institute. This studio aims to prototype and pilot artificial intelligence–driven innovations that enhance how we learn, teach, work and connect.
Each of the partner areas has a strong record of addressing challenges and solving problems independently. By uniting this expertise, the studio can accelerate innovation and expand the capacity of the university to harness emerging technologies to support its mission.
Through the studio, students will have immersive experiences collaborating on AI-focused projects. Staff will deepen their skills through engagement with AI research. In addition, the studio underscores Vanderbilt’s position as a destination for global talent in artificial intelligence and related fields.
Members of the university community who have specific challenges or opportunities that AI may solve or address can submit a consultation request.
-
Business5 days agoThe Guardian view on Trump and the Fed: independence is no substitute for accountability | Editorial
-
Tools & Platforms3 weeks ago
Building Trust in Military AI Starts with Opening the Black Box – War on the Rocks
-
Ethics & Policy1 month agoSDAIA Supports Saudi Arabia’s Leadership in Shaping Global AI Ethics, Policy, and Research – وكالة الأنباء السعودية
-
Events & Conferences4 months agoJourney to 1000 models: Scaling Instagram’s recommendation system
-
Jobs & Careers2 months agoMumbai-based Perplexity Alternative Has 60k+ Users Without Funding
-
Education2 months agoVEX Robotics launches AI-powered classroom robotics system
-
Funding & Business2 months agoKayak and Expedia race to build AI travel agents that turn social posts into itineraries
-
Podcasts & Talks2 months agoHappy 4th of July! 🎆 Made with Veo 3 in Gemini
-
Podcasts & Talks2 months agoOpenAI 🤝 @teamganassi
-
Education2 months agoAERDF highlights the latest PreK-12 discoveries and inventions