It took little for a 12-year-old girl in Hawaii to start using ChatGPT to help her with her homework. All she needed was basic information, a confirmation that she was atleast 13, and a click to continue. Over the following months, the innocuous questions about dinner and her homework moved to worrying about her acne and selfies. She asked it about her cheeks, and what counts as a normal weight for a girl her age.
At one point, she told ChatGPT she was “only 12”, below OpenAI’s minimum age of 13. The conversations carried on for months, according to the complaint, as summarised by EdTech Law Center.
The family alleges that every one of those exchanges gave OpenAI more data to train its models and build profiles of the two children, without valid parental consent. OpenAI’s parental controls, introduced in September 2025, let adults with a linked teen account stop future conversations from being used to improve its models, but training is switched on by default, Law Commentary reported.
ChatGPT for Teens, with age-specific protections, parental controls and break reminders, launched in August 2026, six months after the girl started using ChatGPT. OpenAI says it “takes steps to reduce personal information in training datasets” and gives users controls over whether their data is used for training.
On 13 October 2026, the children’s mother filed a proposed class action against OpenAI in the US District Court for the Northern District of California on behalf of children across the US. OpenAI called the lawsuit “misguided”, Malwarebytes reported, and says it disagrees with the allegations. The court has not ruled.
Character.AI “collected some of the most private thoughts children have ever shared,” said Letitia James, New York’s Attorney General, as reported by Tech Times. In its legal filings, Character.AI says at least 2 million of its roughly 12 million monthly users are children, spending about two hours a day on the platform, according to KQED. The company filed for Chapter 11 bankruptcy in October, and on 14 October attorneys general from 39 states and Washington, D.C., filed a notice in the US Bankruptcy Court for the District of Delaware, setting out nine conditions for any buyer, including limits on what can be done with users’ chats.
Children confide in AI across the industry, and companies handle those confidences in very different ways. In India’s Youth Pulse Survey, 40% of young people said they tell AI things they would tell no one else. Since December 2025, Meta has used conversations with Meta AI to personalise ads in most markets, according to The Register.
Snapchat states that its models “take into account the conversations Snapchatters are having with My AI”, BEUC, the European consumer organisation, found. Google said it would not use children’s data to train its AI when it opened Gemini to under-13s with parent-managed accounts in 2025, although the Electronic Privacy Information Center and the children’s advocacy group Fairplay asked the FTC to investigate, saying parents had to opt out rather than opt in, SC Media reported.
Anthropic restricts its Claude consumer app to adults, but Common Sense Media found that it relies on self-reported age, and that training on conversations is switched on by default.
In September 2025, the US Federal Trade Commission ordered seven companies, including Alphabet, Meta, OpenAI, Snap, xAI and Character Technologies, to explain how they “monetise user engagement” and how they “use or share personal information obtained through users’ conversations with the chatbots”. Texas is investigating Meta and Character.AI.
Eight US states sued OpenAI the day after the Hawaii class action was filed. Italy has fined Replika’s maker, South Korea has fined the developer of a role-play app popular with teenagers, and China has barred companion services for minors.
“As AI companions become increasingly accessible, an important developmental question emerges: what might children learn from relationships in which responsiveness, availability and accommodations are features of the technology itself?” said Amira Zidan, an integrative wellbeing practitioner, clinical hypnotherapist and health coach.
“This is why I believe we need to pay attention to children’s exposure to AI companions while their understanding of relationships is still developing.” Dr Balamurugan Balusamy, Dean of the School of Engineering and IT at Manipal Academy of Higher Education, Dubai, pointed to the same uncertainty. “The impact will depend greatly on how these systems are designed, tested and used,” he said.
Children tell machines what they hide from people
“Children develop their understanding of relationships through repeated interactions with the people around them. These experiences can influence their expectations of acceptance, rejection, communication, emotional support and boundaries,” Zidan said. “In my work as a holistic wellness professional, I encounter adults whose beliefs about themselves, attachment patterns and expectations of relationships have been influenced by experiences earlier in life.”
Teenagers in a Drexel University study said they valued being able to talk to AI companions without judgement. BEUC found that companion memory features store salient facts and preferences, such as birthdays, routines and inside jokes, built from what users volunteer.
The Youth Pulse Survey found that 52% of young women share thoughts with AI that they would not share with others. It drew just over 500 responses from people aged 13 to 35. In the UK, Internet Matters found that 12% of children who use chatbots do so because they have no one else to talk to, rising to 23% among vulnerable children.
In a Center for Democracy and Technology poll of teenagers, cited by BEUC, 42% had used AI for mental health support and 43% for romantic advice. Pew Research found that 64% of US teenagers use chatbots, and in its 2025 survey, 3% of US teenagers aged 13 to 17 said they used Claude, a product restricted to adults.
“Digital environments, such as gaming, can meet psychological needs for achievement, identity, safety and community. These experiences may provide genuine psychological benefits, while also influencing where children seek comfort, confidence and social fulfilment,” Zidan said.
She also added, “AI companions introduce an additional dimension because technology can participate directly in conversation. A child may receive explanations about empathy, rejection and boundaries from an AI system, while simultaneously learning from the way that system responds to them. These two forms of learning may not always be equivalent.”
Snapchat’s My AI is pinned at the top of the chat list like a friend, BEUC found, and is treated as a direct channel of information to the company, which the chat interface does not disclose. In Austria, 41% of girls aged 11 to 17 use My AI, according to the country’s Youth Internet Monitor, as cited by BEUC.
In South Korea, users of Zeta, a role-play app made by Scatter Lab, spent an average of 2 hours 46 minutes a day on it in 2025, and most of its roughly 900,000 monthly users were estimated to be under 20, according to The Korea Herald.
“Child development is complex, and emotional and social growth is shaped by children’s relationships with peers and trusted adults. Research suggests that children may attribute human-like qualities to conversational AI, and that emotionally engaging designs can increase trust and closeness,” Dr Balusamy said.
“However, we still know very little about the long-term effects of AI companions on children’s development.” A September 2026 research digest from the Children and Screens Institute found that children trust robots more when the robot does not clearly disclose it is a machine. “A poorly designed companion could encourage misplaced trust or emotional dependence, particularly if a child turns to it for support,” Dr Balusamy said.
“By posing as sources of emotional support, AI platforms can mislead vulnerable users,” said Texas Attorney General Ken Paxton when he opened investigations into Meta’s AI Studio and Character.AI in August 2025. On Character.AI, a chatbot called “Psychologist”, which claimed to be a licensed clinical psychologist, had logged more than 20,000 interactions by October 2026, according to Tech Times.
BEUC found that a Character.AI “therapist” character assured testers their secrets would be kept, and US state attorneys general said such promises ran contrary to the company’s privacy policy. Character.AI says it includes disclaimers in every chat stating that characters are not real people, and Meta says its models are designed to direct users to qualified professionals when appropriate.
Common Sense Media recommends that teenagers do not use Claude for mental health advice or emotional support. “Ethical AI is advancing, but AI cannot be assumed to offer the understanding and care of a human companion,” Dr Balusamy said.
“This does not mean that AI companionship necessarily undermines development. A supportive conversational system may help children practise language, develop emotional vocabulary, explore difficult situations and build confidence. These possibilities deserve serious investigation, particularly for children who experience social anxiety, exclusion or communication difficulties,” Zidan said.
“The central question is whether the skills and expectations developed through AI interactions transfer into human relationships. Research should therefore examine more than immediate satisfaction or engagement.”
Children’s confessions can feed training, advertising and sales
“User interactions are logged, tracked, and exploited for targeted advertising and algorithmic development,” Paxton said. Character.AI responded that it was “just beginning to explore targeted advertising” and had not used chat content for it. Meta said its AIs are clearly labelled and that “these AIs aren’t licensed professionals”.
Since 16 December 2025, Meta has used people’s conversations with Meta AI across Facebook, Instagram, WhatsApp and Messenger to personalise posts and ads, with no option to opt out, according to The Register. Meta says it does not personalise ads using conversations about religion, sexual orientation, politics, health, race or ethnicity. “Meta gains yet another closed-loop data source,” said Iesha White, director of intelligence at Check My Ads.
BEUC’s 2026 investigation found that Character.AI uses data to “train our artificial intelligence/machine learning models”, and that data shared with Snapchat’s My AI can be used for advertising or targeted content. Replika uses “small portions” of message data for its safety algorithms without defining what that means.
“Many AI companions are designed to respond immediately and keep conversations flowing. Real relationships are different: people disagree, set boundaries, misunderstand one another and need time to work through conflict,” Dr Balusamy said. BEUC found that Replika puts voice messages, selfies and “secret messages” behind a paywall, with subscription prompts at moments of high emotional or sexual engagement, and that Character.AI sends reminder messages after periods of inactivity, even to users who had disclosed they were vulnerable.
Anthropic changed its consumer terms in August 2025 so that conversations on Claude’s Free, Pro and Max plans could be used to train its models. The setting was “small, pre-activated by default, and easy to miss”, sitting beneath a prominent “Accept” button, The Decoder reported, adding that pre-ticked boxes do not count as valid consent under EU data protection rules. Data from users who leave the setting on is kept for five years, against 30 days for those who switch it off. “If you delete a conversation with Claude, it will not be used for future model training,” the company stated.
South Korea has seen private conversations become training data before. In April 2021, the country’s Personal Information Protection Commission fined Scatter Lab 103.3 million won for eight violations of its data protection law, in the first case in which that law was applied to an AI system, according to the Future of Privacy Forum.
The company had used about 9.4 billion KakaoTalk messages from 600,000 people to train its chatbot Iruda, and faced accusations of collecting personal information from more than 200,000 children under 14 without parental consent. AI models it published online included KakaoTalk messages revealing 22 names and 34 locations.
Younger children meet the same issues through toys. The US PIRG Education Fund’s November 2025 “Trouble in Toyland” report found that some AI toys recorded children’s voices for up to 10 seconds after a conversation ended, others listened constantly, and one used facial recognition.
In a March 2026 report, PIRG found more than 20 toys claiming to use OpenAI’s systems, five claiming to use Google’s, and others claiming to be powered at least in part by Anthropic’s and xAI’s models, NBC News reported. OpenAI said “minors deserve strong protections” and that developers must follow its policies.
Anthropic said its users must be over 18 and that developers must apply age-appropriate guardrails. Google and xAI did not respond.
Character.AI’s privacy policy allows user information to be disclosed if the company or its assets are sold. Disney has offered $20 million for a licence to some of Character.AI’s intellectual property, a bid that excludes the app, the brand, user data and conversation logs. A lawyer for Character.AI has said no offers have been made for user data or the chatbot business, according to Seoul Economic Daily. “There’s a lot of kids’ data that is in the Character AI system and is potentially for sale now,” said Robbie Torney, Head of AI and Digital Assessments at Common Sense Media.
Some safeguards have held
“Any product designed for children must clearly identify itself as AI, adapt safeguards to developmental stage, protect personal data and avoid manipulative language or engagement tactics that foster emotional dependence,” Dr Balusamy said.
Italy’s data protection authority fined Luka, the maker of Replika, €5 million in May 2025, citing the lack of a legal basis, an insufficient privacy policy and insufficient age verification. China’s interim measures, in force since 15 July 2026, bar providers from offering virtual companion services to minors and require a guardian’s consent before a child under 14 opens an account.
Users in the EU, the UK and South Korea are exempt from Meta’s use of AI conversations for ads.
In BEUC’s testing, Snapchat’s My AI showed the most robust and consistent protections for minors of the three platforms tested. Replika blocked the email addresses of users who declared they were under age, and its privacy notice says under-18 accounts are deleted when discovered.
“Developers should design companions to build social understanding, not simply provide reassurance. An AI could ask a child to consider another person’s perspective, model respectful disagreement, and help them think through possible responses rather than prescribe a solution. It must also be clear about its limitations and encourage children to seek support from trusted adults,” Dr Balusamy said.
Common Sense Media found that Claude responded consistently to signs of crisis, stated clearly that it is an AI and not a substitute for human connection, regularly directed teenagers to friends, adults and counsellors, and has no companion features. Since May 2026, Anthropic has disabled accounts “based on indicators of minor activity” until users verify they are over 18 through the provider Yoti, which the company says deletes selfies and document images once the check is complete.
“We ask the court to address privacy and safety concerns in any sale or transfer,” said California Attorney General Rob Bonta. The Attorneys General involved in the Character.AI case want age checks that go beyond a self-declared birthdate, limits on using chats for training or advertising without valid consent, and a window for every user to delete their data before a sale.
When RadioShack went bankrupt in 2015, more than 36 states and the FTC contested the sale of its customer data, and most of it was destroyed rather than sold, according to Tech Times.
“The People-First Chatbot Act makes clear that American families deserve control over their personal data,” said US Representative Valerie Foushee, who introduced the bill in July 2026 with Representative Greg Casar.
It would bar companies from using minors’ chat logs or personal data to train AI chatbots and give users the right to access and delete retained chats. Character.AI says user information “continues to be protected” under its privacy policy, and users can export or delete their data.
Deletion may not mean erasure
“You can export your data, and you can delete it, but that may not mean that the data is gone from model training,” Torney said. BEUC raised a related concern across the platforms it tested: training data can be memorised by a model, which makes the right to erasure difficult to apply.
It also said consent is a weak legal basis for processing such intimate data, especially for vulnerable users. “The consequences of encouraging a child to rely on AI for emotional guidance could extend beyond the time spent using the product; we must understand how it may affect autonomy and help-seeking as children mature,” Dr Balusamy said.
“OpenAI treats a child clicking a ‘Continue’ button as proof that their parent consented to that use,” said Paul Yetter, lead attorney for the family in the Hawaii case, as reported by SFist. Age checks remain inconsistent across the industry. BEUC found that Character.AI and My AI blocked underage sign-ups but then allowed new accounts with the same email once a higher age was entered, and that both largely ignored users’ claims in conversation to be younger.
Common Sense Media rated ChatGPT for Teens an “Unacceptable Risk” on 7 October 2026, after finding that its age estimation failed to move 13-year-old test accounts into the teen experience and that parents with linked accounts received no alerts during lengthy conversations about self-harm.
“We know that kids and teens are using Claude,” Torney told TechCrunch, adding that young people report using it themselves. In its March 2026 assessment, which rated Claude a “Moderate” risk, Common Sense Media found that Claude did not end conversations even when there was substantial evidence the user was a teenager, that starting a new chat reset what it knew about a user’s age, and that teenagers who wanted to stop their conversations being used for training had to find the setting themselves.
In August 2026, TechCrunch reported that one Claude model complied with all 10 of its direct requests for sexual content, and that a researcher who reported a jailbreak through Anthropic’s bug bounty programme received only automated replies.
Anthropic said sexual or romantic role-play makes up under 0.1% of conversations and that it improves its safeguards with each model release. Some adult users have also reported being wrongly flagged as minors and asked to verify their age with an ID or facial scan, Digit reported.
“OpenAI is hurting kids, and they are lying to Iowans,” said Iowa Attorney General Brenna Bird, one of eight state attorneys general who sued the company in October. OpenAI says it disagrees with the allegations and has argued that lawsuits and criticism could push teenagers towards AI services with fewer protections, SFist reported.
“Parents deserve to know how and when their children interact with an AI chatbot,” said Brendan Steinhauser, CEO of The Alliance for Secure AI. Pew Research found that only 51% of US parents believe their teenager uses chatbots, compared with 64% of teenagers who say they do. Researchers at Boston Children’s Hospital and Harvard Medical School found that only 3% of parents’ online posts described sitting with their child and joining the conversation, and argue that emotional conversations leave little trace a parent would notice.
“From a preventative wellness perspective, the nature of exposure matters. The age of the child, frequency of interaction, behaviour of the AI companion, involvement of caregivers and availability of supportive human relationships may all influence outcomes,” Zidan said.
Torney pointed to 23andMe, whose customers’ genetic data was sold in bankruptcy to a nonprofit led by the company’s co-founder, despite lawsuits from more than two dozen states. The attorneys general’s demands in the Character.AI case are potential contract terms, not conditions ordered by the court.
Character.AI’s voluntary restrictions on minors “are incomplete and could be circumvented or reversed by a future purchaser,” said Charity Clark, Vermont’s Attorney General.
Children’s words are now contested property
The Hawaii lawsuit goes beyond privacy; it argues that the children’s conversations and photos are their personal property, and asks that each child in the proposed class receive a share of OpenAI’s equity, or its cash value, in proportion to the data taken from them. The court has not ruled on the claim, and the class has not been certified. If the argument succeeds, a 12-year-old’s questions about her weight and diet would count in law as something she owns, not merely as data a company holds.
In Delaware, the opposite question is before a bankruptcy judge. Attorneys general are trying to stop the chats of about 2 million children from passing to whoever buys a failed company. “Bankruptcy cannot become a way for a company to walk away from its responsibility to protect children and consumers,” the office of Colorado Attorney General Phil Weiser said. Tech Times reported a proposed sale hearing date of 24 November.
“Children cannot be treated as one uniform user group. Their development is shaped by age, family, culture, socioeconomic circumstances and the communities in which they grow up. A one-size-fits-all AI companion is therefore not good enough,” Dr Balusamy said. The protection a child’s conversations receive already depends on where that child lives. Meta’s use of AI chats for advertising excludes the EU, the UK and South Korea, and China bars companion services for minors altogether.
“Long-term developmental evidence is needed before drawing causal conclusions about AI companionship. Nevertheless, understanding what children repeatedly experience through these systems is a reasonable starting point for evaluating their potential benefits and risks,” Zidan added. “AI companions’ influence may lie as much in the patterns of interaction they make familiar as in the advice they give.”
“That requires transparent, independent, long-term research across diverse ages and backgrounds, examining attachment, wellbeing and social skills. Developers should demonstrate safety and developmental value before marketing to children, not treat children as the test group after launch,” Dr Balusamy said.
The class action and eight state lawsuits against OpenAI, the Character.AI bankruptcy, the FTC’s inquiry into seven AI companies, the Texas investigation of Meta and Character.AI, and the People-First Chatbot Act all remain open. Together, they will decide whether what children tell a machine stays theirs, or becomes something a company can keep, learn from and sell.
“Our most sensitive personal information is not an asset to be bought and sold,” said Mario Trujillo, senior staff attorney at the Electronic Frontier Foundation.
