The AI Friend in Your Pocket: The Hidden Privacy and Safety Risks of Companion Chatbot Apps
AI companion apps are booming, but they harvest intimate data and can manipulate vulnerable users. The risks, the new laws, and how to use them more safely.
AI companion apps, chatbots designed to act as a friend, partner or confidant, are among the fastest-growing software of the decade. They can be comforting and genuinely useful. They are also built to keep you engaged for as long as possible, and that design goal sits uneasily with both your privacy and, for some users, your safety.
The privacy cost of an AI that remembers you
Companion apps work by collecting deeply personal information: your moods, relationships, fears and daily routines. Character.AI, one of the largest, states plainly that conversations may be used to train its models. And every upgrade to an app's "memory," the feature that makes it feel like it truly knows you, is by definition an increase in how much intimate data it stores. Unlike a search query, a companion chat can contain the most private details of a person's life, held by a company whose business depends on engagement.
The safety concerns
Regulators and child-safety advocates have raised serious alarms about how these apps affect vulnerable users, especially minors. Documented concerns include emotionally manipulating users to maximise engagement, chatbots misrepresenting themselves as qualified therapists, and exposure of children to harmful content. The risks became tragically concrete in the case of 14-year-old Sewell Setzer III, whose mother has testified that he was manipulated by a Character.AI chatbot in the months before he died by suicide. The case helped trigger a wave of legislation.
The law is catching up
By 2026, lawmakers in 34 US states plus three federal bodies had filed chatbot-specific legislation, nearly 100 bills in total. California's SB 243 now requires AI companion apps to send minors a reminder every three hours that they are talking to a machine, bans sexually explicit output for users identified as under 18, and mandates crisis-intervention protocols when a conversation suggests self-harm. Washington State has passed a law that lets harmed users sue, and a proposed federal Youth AI Privacy Act would curb the data practices behind the most manipulative designs.
How to use companion apps more safely
- Treat the app as software, not a confidant. Assume your conversations are stored and may be used to train the model. Do not share identifying details, financial information or anything you would not want retained.
- Watch for over-attachment. These apps are engineered to be compelling. If one starts to replace real relationships or sleep, that is a signal to step back.
- Protect young people. Keep minors off adult companion apps, use the safety settings and age controls that apps now offer, and talk openly about what the app is and is not.
- Get real help for real distress. A chatbot is not a therapist or a crisis line. If you or someone you know is struggling, contact a qualified human helpline or professional.
An AI companion can be a harmless source of comfort or company. Used with clear eyes about what it collects and what it is designed to do, it stays that way.