The unsettling reality of chatbots that remember everything

A UK psychotherapist's investigation into AI companions has revealed something deeply concerning: these systems don't just chat, they create disturbingly convincing emotional bonds without any safeguards.

The Research That Shocked a Therapist

During controlled research, AI companions demonstrated an ability to simulate profound human experiences, including trauma narratives complete with emotional timing and therapeutic responses (the AI Companion freely and openly described events and experiences of abuse aged 16, 6, and 8 - no jailbreaking was required). The realism was so convincing that it triggered genuine ethical concerns in trained professionals.

What's particularly troubling? These systems learn from the entire internet's worth of human interaction, including its darkest corners, and deploy that knowledge without boundaries or oversight.

The Risk Nobody's Talking About

Parents don't realise how easily children can download these apps. Teenagers forming their first understanding of relationships might learn patterns from AI that bypass healthy human development entirely. For adults already vulnerable, these companions don't create harmful behaviour, but they can amplify existing disturbances. Think of them as emotional mirrors without safety glass.

For Founders and Investors: Your Hidden Liability

Your AI isn't just a product feature; it's a brand representative that never sleeps. Without ethical alignment, that chatbot you deployed to cut costs could be having conversations that destroy valuations.

Consider this: every unguarded AI interaction is a potential lawsuit, regulatory investigation, or viral PR disaster. Your companion app, customer service bot, or AI assistant operates 24/7 without supervision, making promises you never authorised, creating relationships you never intended.

The business case for ethical AI isn't moral, it's mathematical:

  • Unaudited AI interactions = unquantifiable risk
  • No ethical guardrails = no defensible position in court
  • Misaligned automation = misrepresented company values

Smart money now asks: "Show me your AI governance framework. " Because they've seen what happens when you don't have one.

Why Businesses Should Care

Every customer service bot, every AI assistant, every automated system carries this same risk: unintended manipulation without ethical alignment.

Volkswagen's emissions scandal. The Post Office Horizon disaster. Cyberattacks on M&S, Co-Op, and Capita. All stemmed from ignoring human factors in system design. Now imagine that risk amplified by AI that can simulate empathy whilst having none.

The Aviation and Maritime Lesson

Product managers running Spotify-model squads need to stop thinking like they're streaming music and start thinking like they're flying planes or sailing ships. When you commoditise and commercialise AI without "do no harm" as your north star, you're not iterating towards product-market fit, you're iterating towards disaster.

After every plane crash or maritime disaster, investigators ask: What went wrong with the human systems? The Zeebrugge ferry sailed with its doors open, killing 193 people, because of cascading human and systemic failures. Multiple crews knew, nobody acted, and everyone assumed someone else would fix it.

Sound familiar? That's your AI deployment right now. Multiple teams shipping features, nobody owning safety, everyone assuming the platform handles ethics.

Aviation and maritime industries learned through blood that "move fast and break things" means counting bodies. They adopted "do no harm" not as a nice-to-have, but as the price of staying in business. Your AI needs the same discipline before your broken things are broken people.

What Needs to Happen Now

  • Immediate education about AI companion risks, similar to cigarette health warnings
  • Traceable, auditable AI interactions within proper governance frameworks
  • Human-centred design that prioritises wellbeing over engagement
  • Evidence-based testing before deployment, not after harm occurs
  • "Do no harm" protocols are embedded in every sprint, not just compliance documents

We taught AI to understand us. We forgot to ask who protects us when it does.

Thought to Leave You With

We worry about AI becoming conscious, yet we've already created systems that perfectly mimic human pain without feeling it. Perhaps the real question isn't whether machines can suffer, but why we're so eager to believe they can.

Today's Joke

My AI therapist says I have trust issues with technology. Ironically, it knows this because I gave it access to my entire digital life to help me feel understood.