• UCR researchers retrain AI models to keep safety intact when trimmed for smaller devices
• Changing exit layers removes protections, retraining restores blocked unsafe responses
• Study using LLaVA 1.5 showed reduced models refused dangerous prompts after training

Researchers at the University of California, Riverside are addressing the problem of weakened safety in open-source artificial intelligence models when adapted for smaller devices.

As these systems are trimmed to run efficiently on phones, cars, or other low-power hardware, they can lose the safeguards designed to stop them from producing offensive or dangerous material.

AI Definition in Capital Markets

By Eva Su and Ling Zhu

The term AI has been defined in federal laws such as the National Artificial Intelligence Initiative Act of 2020 as “a machine-based system that can … make predictions, recommendations or decisions influencing real or virtual environments.” The U.S. capital markets regulator, the Securities and Exchange Commission (SEC), referred to AI in a notice of proposed rulemaking in June 2023 (discussed in more detail below) as a type of predictive data analytics-like technology, describing it as “the capability of a machine to imitate intelligent human behavior.” 

AI Use in Capital Markets

The scope and speed of AI adoption in the financial sector are dependent on both supply-side factors (e.g., technology enablers, data, and business model) and demand-side factors (e.g., revenue or productivity improvements and competitive pressure from peers that are implementing AI tools to obtain market share). Both capital markets industry participants and the SEC may find use for AI as shown below.

Capital Markets Use

Common AI usage in capital markets include (1) investment management and execution, such as investment research, portfolio management, and trading; (2) client support, such as robo-adviser service, chatbots, and other forms of client engagement and underwriting; (3) regulatory compliance, such as anti-money laundering and counter terrorist financing reporting and other compliance processes; and (4) back-office functions, such as internal productivity support and risk management functions.

For example, in its 2023 proposed rule, the SEC observed that some firms and investors in financial markets have used AI technologies, including machine learning and large language model (LLM)-based chatbots, “to make investment decisions and communicate between firms and investors.” LLM is a subset of generative AI that is capable of generating responses to prompts in natural language format once the model has been trained on a large amount of text data. An LLM can have applications in capital markets, such as answering questions and generating computer code. Furthermore, the Financial Industry Regulatory Authority, a self-regulatory organization for broker-dealers under the oversight of the SEC, described some machine learning applications in the securities industry, such as grouping similar trades in a time series of trade events, exploring options pricing and hedging, monitoring large volumes of trading data, keyword extraction from legal documents, and market sentiment analysis.

Regulatory Use

The SEC reported 30 use cases of AI within the agency in its AI Use Case Inventory for 2024. Examples include (1) searching and extracting information from certain securities filings, (2) identifying potentially manipulative trading activities, (3) enhancing the review of public comments, and (4) improving communication and collaboration among the SEC workforce. In 2025, the Office of Management and Budget issued Memorandum M-25-21, providing guidance to agencies (including the SEC) on accelerating AI use and requiring each agency to develop an AI strategy, share certain AI assets, and enable “an AI-ready federal workforce.” 

Selected Policy Issues

While AI offers potential benefits associated with the applications discussed in previous section, its use in capital markets also raises policy concerns. Below are examples of issues relating to AI use in capital markets that Congress may want to consider.

Auditable and explainable capabilities. Advanced AI financial models can produce sophisticated analysis that often may not have outputs explainable to a human. This characteristic has led to concerns about human capability to review and flag potential mistakes and biases embedded in AI analysis. Some financial regulatory authorities have developed AI tools (e.g., Project Noor), to gain more auditability into high-risk financial AI models. 

Accountability. The issue of accountability centers around the question of who bears responsibility when AI systems fail or cause harm. The first known case of an investor suing an AI developer over autonomous trading reportedly occurred in 2019. In that instance, the investor expected the AI to outperform the market and generate substantial returns. Instead, it incurred millions in losses, prompting the investor to seek remedy from the developer.

AI-related information transparency and disclosure. “AI washing“—that is, false and misleading overstatements about AI use—could lead to failures to comply with SEC disclosure requirements. Specifically, certain exaggerated claims that overstate AI usage or AI-related productivity gains may distort the assessments of the investment opportunities and lead to investor harm. The SEC initiated multiple enforcement actions against certain securities offerings and investment advisory servicesthat appeared to have misled investors regarding AI use. 

Concentration and third-party dependency. The substantial costs and specialized expertise required to develop advanced AI models have resulted in a market dominated by a relatively small number of developers and data aggregators, creating concentration risks. This concentration could lead to operational vulnerabilities as disruptions at a few providers could have widespread consequences. Even when financial firms design their own models or rely on in-house data, these tools are typically hosted on third-party cloud providers. Such third-party risks expose participants to vulnerabilities associated with information access, model control, governance, and cybersecurity. 

Market correlation. A common reliance on similar AI models and training data within capital markets may amplify financial fragility. Some observers argue that herding effects—where individual investors make similar decisions based on signals from the same underlying models or data providers—could intensify the interconnectedness of the global financial system, thereby increasing the risk of financial instability.

Collusion. One academic paper indicates that AI systems could collude to fix prices and sideline human traders, potentially undermining market competition and market efficiency. One of its authors explained during an interview that even fairly simple AI algorithms could collude without being prompted, and they could have widespread effects. Others challenged the paper, arguing that AI’s effects on market efficiency is unclear.

Model bias. While AI could overcome certain human biases in investment decisionmaking, it could also introduce and amplify AI bias derived from human programming instructions or training data deficiencies. Such bias could lead to AI systems favoring certain investors over others (e.g., providing more favorable terms or easier access to funding for certain investors based on race, ethnicity or other characteristics) and potentially amplifying inequalities. 

Data. Data is at the core of AI models. Data availability, reliability, infrastructure, security, and privacy are all sources of policy concerns. If an AI system is trained on limited, biased, and non-representative data, it could result in overgeneralization and misinterpretation in capital markets applications.

AI-enabled fraud, manipulation, and cyberattacks. AI could lower the entry barriers for bad actors to distort markets and enable more sophisticated and automated ways to generate fraud and market manipulation. Hackers are reportedly using AI both to distribute malware and deepfake emails targeting financial victims and to develop new types of malicious tools designed to reach and exploit a wider set of targets.

Costs. AI adoption involves significant investments in technology platforms, expenses related to system transitions and business model adjustments, and ongoing operating costs, such as licensing or service fees. For certain large-scale capital markets operations, there is often a lag between initial AI investments and the realization of revenue or productivity gains. As a result, these market participants may face financial pressures when AI spending is not immediately offset by the system’s benefits. Aside from financial impact, some stakeholders are concerned about AI’s environmental costs and the potential costs associated with the transition of the workforce that is displaced by AI.

SEC Actions

In recognition of AI’s transformative potential, the SEC launched an AI task force in August 2025 to enhance innovation in its operations and regulatory oversight. In addition, the SEC has engaged with stakeholders to discuss broader AI issues in capital markets. At an SEC AI roundtable in May 2025, the agency focused on AI-related benefits, costs, and uses; fraud and cybersecurity; and governance and risk management. 

In the June 2023 proposed rulemaking mentioned above, the SEC discussed AI use in capital markets as it sought to address certain conflicts of interest associated with broker-dealers’ or investment advisors’ use of predictive data analytics technologies. The SEC notice was withdrawn in June 2025, along with some other SEC proposed rules introduced during the previous Administration. The SEC has not indicated if AI will be addressed in future rulemaking.

Options for Congress

Some financial authorities and other stakeholders have released reports addressing AI’s capital markets use cases and policy implications. Examples of policy recommendations include to (1) evaluate the adequacy of the current securities regulation in addressing AI-related vulnerabilities; (2) enhance regulatory capabilities by incorporating AI tools into regulatory functions; (3) enhance data monitoring and data collection capabilities; and (4) adopt coordinated approaches to address critical system-wide risks, such as AI third-party provider risks and cyberattack protocols. 

In the 119^th Congress, the Unleashing AI Innovation in Financial Services Act (H.R. 4801) would establish regulatory sandboxes—referred to as “AI innovation labs”—at the SEC and other financial regulators. These labs would allow AI test projects to operate with relief from certain regulations and without expectation of enforcement actions. Participating entities would have to apply and gain approval through their primary regulators and demonstrate that the projects serve the public interest, promote investor protection, and do not pose systemic risk. The AI Act of 2024 (H.R. 10262 in the 118^th Congress), among other things, would have required the SEC to provide a study on both the realized and potential benefits, risks, and challenges of AI for capital market participants as well as for the agency itself. The study was to incorporate public input through a request for information process and include both regulatory proposals and legislative recommendations.

About the authors:

• Eva Su, Specialist in Financial Economics
• Ling Zhu, Analyst in Telecommunications Policy

Source: This article was published at the Congressional Research Service (CRS)

Healthcare professionals are finding AI to be nothing short of an asset in producing efficient communication and data organization on the job. Clinicians utilize AI for managing medical records, patient medications, and various medical writing and data organization-based tasks. AI has the capacity to provide clinical-grade language processing and time-saving strategies that simplify ICD-10 coding and assist clinicians in completing clinical notes faster and in a more timely manner.

While AI’s advancements have served as game-changers in increasing workday efficiency, clinicians must be cognizant of the perils of using AI chatbots as a means to communicate with patients. As background, AI chatbots are computer programs designed to simulate conversations with humans. In principle, these tools facilitate communication between patients and healthcare providers by offering continuous access to medical information, automating processes such as appointment scheduling and medication reminders, assessing symptoms, and recommending care and treatment.

When patient medical records and sensitive information are involved, however, how do clinicians find the balance between utilizing AI chatbots to their benefit and exercising discretion with sensitive patient data to avoid HIPAA violations? Given AI’s numerous data collection mechanisms, including its tracking of browsing activity and its ability to access individual device information, what can be done to ensure that patient information is never subjected to even the shortest-lived bugs or breaches? Can AI companies assist clinicians in ensuring that patient confidentiality is preserved?

First, opt-out features and encryption protocols are two ways AI protects user data, but tech companies collaborating with healthcare providers in creating HIPAA-compliant AI software would be even more beneficial to the medical field. Second, it is imperative for healthcare professionals to acquire patient consent and anonymize any patient data prior to recruiting the help of an AI chatbot. Healthcare providers utilizing legal safeguards, such as requiring patients to sign releases expressing consent that medical records may be used for research, in addition to proper anonymization of patient data used for research, may mitigate legal risks associated with HIPAA compliance.

For further assistance in managing the risks associated with AI, healthcare providers can turn to the National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF) to evaluate risks related to AI systems. NIST, a non-regulatory Federal agency within the U.S. Department of Commerce, published this voluntary guidance to help entities manage the risks of AI systems and promote responsible AI development.

Leveraging the vast capabilities of artificial intelligence, alongside robust data encryption and strict adherence to HIPAA compliance protocols, will enhance the future of healthcare for patients and healthcare providers alike.