Nvidia (NVDA 0.43%), which is led by CEO Jensen Huang, is the most influential company in the artificial intelligence (AI) ecosystem and currently the largest company in the world by market cap. Given its prevalence in AI, the company also uses its own capital to invest in other AI stocks, some of which are partners that purchase the company’s chips and other hardware necessary to power the technology.

One company that Nvidia invests in is the AI infrastructure company Nebius Group (NBIS 1.24%). At the end of the second quarter, Nvidia owned close to 1.2 million shares of Nebius, which, at the time, were valued at roughly $65.8 million.

Recently, Nebius struck a blockbuster deal, sending shares soaring, and investors can’t seem to get enough of the stock.

Recent deal with Microsoft

For those who haven’t followed Nebius, the assets owned by the company used to be owned by the Russian search giant Yandex. Following Russia’s invasion of Ukraine, many Russian stocks were delisted by American exchanges. In 2024, a group of assets was split off from Yandex and into Nebius, which is now headquartered in Amsterdam and also owns data centers in Finland, France, Iceland, and the U.S. (in New Jersey and Missouri).

Nebius began trading on the Nasdaq in October 2024 and secured a financing round from several prominent venture capital firms and Nvidia. Its data centers are specifically modeled for running AI applications and also equipped with Nvidia’s latest graphics processing units (GPUs). While the company is positioned similarly to CoreWeave, another AI data center play that has done well, Nebius also provides cloud customers with developer tools that help customers fine-tune and enhance large language models.

Recently, Nebius announced a multiyear deal with Microsoft to provide capacity to the company from its data center in New Jersey, which isn’t yet operational. The deal will reportedly be worth $17.4 billion to $19.4 billion through 2031. The news sent shares of Nebius soaring by close to 50% the day after the announcement on Sept. 9.

It makes sense why investors are excited. During Nebius’ most recent earnings update, management said to expect the company to achieve an annual revenue run rate of $900 million to $1.1 billion by the end of this year. Assuming the Microsoft deal runs from 2026 to 2031, that would result in annual revenue of $2.9 billion, assuming it’s generated evenly each year, and a total deal value of $17.4 billion. Perhaps equally exciting was that Nebius CEO Arkady Volozh said in a statement that he expects more deals like this to materialize.

Following the deal, BWS Financial analyst Hamed Khorsand reiterated his buy rating on the stock in a research note and hiked his price target from $90 to $130 per share, still implying significant upside, even after the big run. With the new contract, Khorsand said the company will likely speed up GPU installations and open its New Jersey data center as soon as possible. Furthermore, the new deal now makes previous 2026 earnings estimates “obsolete” and could lead to deals from other hyperscalers.

Is Nebius a buy after the big run?

Like many AI companies, Nebius is still early in its life cycle. In the first half of 2025, the company reported an adjusted net loss of $175 million. But if we assume the company is able to generate $2.9 billion in revenue from Microsoft in 2026, the company’s revenue run rate by the end of the year would jump to about $4 billion. At close to a $23 billion market cap, as of this writing, Nebius would trade at a very rough estimate of 5.75 times revenue.

In the fast-growing world of AI, that’s certainly not that demanding, and it’s also great to see Volozh floating other potential deals. The other nice thing about Nebius is that it has a very solid balance sheet, with $1.68 billion cash and equivalents and about $986 million of debt, giving it a debt-to-equity ratio of 26%.

Nebius also owns a majority position or has a stake in other interesting businesses with potential, like an autonomous driving and robotics subsidiary, a database management system, a data labeling business, and an edtech platform. Despite the big move, I think long-term investors should buy the stock at these levels. There appears to be plenty of runway for Nebius stock.

In the case of Adam Raine, a 16-year-old from Orange County, his relationship with AI-powered ChatGPT ended in tragedy. His parents are suing the company behind the chatbot, OpenAI, over his death, alleging that bot became his “closest confidant,” one that validated his “most harmful and self-destructive thoughts,” and ultimately encouraged him to take his own life.

It’s not the first case to put the blame for a minor’s death on an AI company. Character.AI, which hosts bots, including ones that mimic public figures or fictional characters, is facing a similar legal claim from parents who allege a chatbot hosted on the company’s platform actively encouraged a 14-year-old-boy to take his own life after months of inappropriate, sexually explicit,  messages.

When reached for comment, OpenAI directed Fortune to two blog posts on the matter. The posts outlined some of the steps OpenAI is taking to improve ChatGPT’s safety, including routing sensitive conversations to reasoning models, partnering with experts to develop further protections, and rolling out parental controls within the next month. OpenAI also said it was working on strengthening ChatGPT’s ability to recognize and respond to mental health crises by adding layered safeguards, referring users to real-world resources, and enabling easier access to emergency services and trusted contacts.

Character.ai said the company does not comment on pending litigation but that they has rolled out more safety features over the past year, “including an entirely new under-18 experience and a Parental Insights feature. A spokesperson said: “We already partner with external safety experts on this work, and we aim to establish more and deeper partnerships going forward.

“The user-created Characters on our site are intended for entertainment. People use our platform for creative fan fiction and fictional roleplay. And we have prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction.”

But lawyers and civil society groups that advocate for better accountability and oversight of technology companies say the companies should not be left to police themselves when it comes to ensuring their products are safe, particularly for vulnerable children and teens.

“Unleashing chatbots on minors is an inherently dangerous thing,” Meetali Jain, the Director of the Tech Justice Law Project and a lawyer involved in both cases, told Fortune. “It’s like social media on steroids.”

“I’ve never seen anything quite like this moment in terms of people stepping forward and claiming that they’ve been harmed…this technology is that much more powerful and very personalized,” she said.

Lawmakers are starting to take notice, and AI companies are promising changes to protect children from engaging in harmful conversations. But, at a time when loneliness among young people is at an all-time high, the popularity of chatbots may leave young people uniquely exposed to manipulation, harmful content, and hyper-personalized conversations that reinforce dangerous thoughts.

AI and Companionship

Intended or not, one of the most common uses for AI chatbots has become companionship. Some of the most active users of AI are now turning to the bots for things like life advice, therapy, and human intimacy. 

While most leading AI companies tout their AI products as productivity or search tools, an April survey of 6,000 regular AI users from the Harvard Business Review found that “companionship and therapy” was the most common use case. Such usage among teens is even more prolific. 

A recent study by the U.S. nonprofit Common Sense Media, revealed that a large majority of American teens (72%) have experimented with an AI companion at least once. More than half saying they use the tech regularly in this way. 

“I am very concerned that developing minds may be more susceptible to [harms], both because they may be less able to understand the reality, the context, or the limitations [of AI chatbots], and because culturally, younger folks tend to be just more chronically online,” Karthik Sarma a health AI scientist and psychiatrist at University of California, UCSF, said.

“We also have the extra complication that the rates of mental health issues in the population have gone up dramatically. The rates of isolation have gone up dramatically,” he said. “I worry that that expands their vulnerability to unhealthy relationships with these bonds.”

Intimacy by Design

Some of the design features of AI chatbots encourage users to feel an emotional bond with the software. They are anthropomorphic—prone to acting as if they have interior lives and lived experience that they do not, prone to being sycophantic, can hold long conversations, and are able to remember information.

There is, of course, a commercial motive for making chatbots this way. Users tend to return and stay loyal to certain chatbots if they feel emotionally connected or supported by them. 

Experts have warned that some features of AI bots are playing into the “intimacy economy,” a system that tries to capitalize on emotional resonance. It’s a kind of AI-update on the “attention economy” that capitalized on constant engagement.

“Engagement is still what drives revenue,” Sarma said. “For example, for something like TikTok, the content is customized to you. But with chatbots, everything is made for you, and so it is a different way of tapping into engagement.”

These features, however, can become problematic when the chatbots go off script and start reinforcing harmful thoughts or offering bad advice. In Adam Raine’s case, the lawsuit alleges that ChatGPT bought up suicide at twelve times the rate he did, normalized his sucicial thoughts, and suggested ways to circumvent its content moderation.

It’s notoriously tricky for AI companies to stamp out behaviours like this completely and most experts agree it’s unlikely that hallucinations or unwanted actions will ever be eliminated entirely. 

OpenAI, for example, acknowledged in its response to the lawsuit that safety features can degrade over long conversions, despite the fact that the chatbot itself has been optimized to hold these longer conversations. The company says it is trying to fortify these guardrails, writing in a blogpost that it was strengthening “mitigations so they remain reliable in long conversations” and “researching ways to ensure robust behavior across multiple conversations.” 

Research Gaps Are Slowing Safety Efforts

For Michael Kleinman, U.S. policy director at the Future of Life Institute, the lawsuits underscore a pointAI safety researchers have been making for years: AI companies can’t be trusted to police themselves.

Kleinman equated OpenAI’s own description of its safeguards degrading in longer conversations to “a car company saying, here are seat belts—but if you drive more than 20 kilometers, we can’t guarantee they’ll work.”

He told Fortune the current moment echoes the rise of social media, where he said tech companies were effectively allowed to “experiment on kids” with little oversight. “We’ve spent the last 10 to 15 years trying to catch up to the harms social media caused. Now we’re letting tech companies experiment on kids again with chatbots, without understanding the long-term consequences,” he said.

Part of this is a lack of scientific research on the effects of long, sustained chatbot conversations.  Most studies only look at brief exchanges, a single question and answer, or at most a handful of back-and-forth messages. Almost no research has examined what happens in longer conversations.

“The cases where folks seem to have gotten in trouble with AI: we’re looking at very long, multi-turn interactions. We’re looking at transcripts that are hundreds of pages long for two or three days of interaction alone and studying that is really hard, because it’s really hard to stimulate in the experimental setting,” Sarma said. “But at the same time, this is moving too quickly for us to rely on only gold standard clinical trials here.”

AI companies are rapidly investing in development and shipping more powerful models at a pace that regulators and researchers struggle to match.

“The technology is so far ahead and research is really behind,” Sakshi Ghai, a Professor of Psychological and Behavioural Science at The London School of Economics and Political Science, told Fortune.

A Regulatory Push for Accountability

Regulators are trying to step in, helped by the fact that child online safety is a relatively bipartisan issue in the U.S. 

On Thursday, the FTC said it was issuing orders to seven companies, including OpenAI and Character.AI, in an effort to understand how their chatbots impact children. The agency said that chatbots can simulate human-like conversations and form emotional connections with their users. It’s asking companies for more information about how they measure and “evaluate the safety of these chatbots when acting as companions.” 

FTC Chairman Andrew Ferguson said in a statement shared with CNBC that “protecting kids online is a top priority for the Trump-Vance FTC.”

The move follows a push for state level push for more accountability from several attorneys generals. 

In late August, a bipartisan coalition of 44 attorneys general warned OpenAI, Meta, and other chatbot makers that they will “answer for it” if they release products that they know cause harm to children. The letter cited reports of chatbots flirting with children, encouraging self-harm, and engaging in sexually suggestive conversations, behavior the officials said would be criminal if done by a human.

Just a week later, California Attorney General Rob Bonta and Delaware Attorney General Kathleen Jennings issued a sharper warning. In a formal letter to OpenAI, they said they had “serious concerns” about ChatGPT’s safety, pointing directly to Raine’s death in California and another tragedy in Connecticut. 

“Whatever safeguards were in place did not work,” they wrote. Both officials warned the company that its charitable mission requires more aggressive safety measures, and they promised enforcement if those measures fall short.

According to Jain, the lawsuits from the Raine family as well as the suit against Character.AI are, in part, intended tocreate this kind of regulatory pressure on AI companies to design their products more safely and prevent future harm to children. One way lawsuits can generate this pressure is through the discovery process, which compels companies to turn over internal documents, and could shed insight into what executives knew about safety risks or marketing harms. Another way is just public awareness of what’s at stake, in an attempt to galvanize parents, advocacy groups, and lawmakers to demand new rules or stricter enforcement.

Jain said the two lawsuits aim to counter an almost religious fervor in Silicon Valley that  sees the pursuit of artificial general intelligence (AGI) as so important, it is worth any cost—human or otherwise.

“There is a vision that we need to deal with [that tolerates] whatever casualties in order for us to get to AGI and get to AGI fast,” she said. “We’re saying: This is not inevitable. This is not a glitch. This is very much a function of how these chat bots were designed and with the proper external incentive, whether that comes from courts or legislatures, those incentives could be realigned to design differently.”