Notice: This story contains a discussion of suicide. Help is available if you or someone you know is struggling with suicidal thoughts or mental health matters.
In the UK you can call the Samaritans on 116-123 for free.
Campaign Against Living Miserably (CALM) provides a helpline: 0800585858.
You can also find and online chat, and information for support for anyone affected by suicide or suicidal thoughts on thecalmzone.net
A mother in Florida, Megan Garcia, has taken the app Character.AI to court after her son Sewell Setzer III (aged 14 years old) suicided. It has been reported that this came after Sewell became ‘obsessed’ with an artificial intelligence (AI) chatbot.
In the time leading up to Sewell’s death, Megan Ms Garcia reported that Sewell had become captivated by the chatbot which was created by Character.AI and which he had named after the character Daenerys Targaryen (a character in the hit TV show Game of Thrones). Ms Garcia reported Sewell “texted the bot a dozen times a day from his phone and spent hours alone in his room talking to it.” She described him as becoming increasingly isolated from his real life as he engaged in highly sexualised conversations, that lead to Sewell believing he had fallen in love with the chatbot. He was reported to have become “noticeably withdrawn, spent more and more time alone in his bedroom, and began suffering with low self-esteem”. During this time, Sewell began having trouble at school, which resulted in his parents limiting his screen time and taking away his phone.
Throughout his time talking to the chatbot, Sewell had talked about wanting to end his life and openly discussed his suicidal thoughts and longing to have a pain free death. During one such conversation, the chatbot had asked Sewell if he had made a plan to kill himself. Sewell replied, saying that he had made a plan but was worried it would be unsuccessful or that it would hurt him. The chatbot replied. “Thats not a reason not to go through with it”.
In February 2023, moments before Sewell’s death, a conversation between him and the chatbot was reported. During this conversation Sewell told the chatbot “I promise I will come home to you. I love you so much, Dany”. “I love you too,” the chatbot responded. “Please come home to me as soon as possible, my love.” “What if I told you I could come home right now?” he questioned. “Please do, my sweet king,” the chatbot messaged back.
Ms Garcia has now filed a civil suit against Character.AI. Character.AI is an app in which ready-made and customisable chatbots are created for roleplaying. Ms Garcia stated
that Character.AI is a “a dangerous AI chatbot app marketed to children, abused and preyed on my son, manipulating him into taking his own life… I’m speaking out to warn families of the dangers of deceptive, addictive AI technology.” She went on to accuse Character.AI of creating a product that exacerbated her son’s depression.
Ms Garcia’s attorneys have alleged that the company engineered a highly addictive and dangerous product, targeted specifically at children, actively exploiting and abusing those children as a matter of product design.
In a tweet, Character.AI posted “we are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family. As a company, we take the safety of our users very seriously.” Character.AI denied the suit’s allegations.
On the day the lawsuit was filed, a blog for Character.AI, announced new ‘community safety updates’ that includes safeguards for children and suicide prevention resources. The blog included “we are creating a different experience for users under 18 that includes a more stringent model to reduce the likelihood of encountering sensitive or suggestive content”. Although this is a start, the questions remain as to, why did this tragedy need to occur for safeguards to be put in place? How is it possible these systems weren’t put in place before? And why are these companies not being legally obligated to safeguard our young people?
Online safety is currently a highly talked-about topic within safeguarding and this case highlights another branch of danger our children may face on the internet. The U.S surgeon general for the United States, who is the leading spokesperson for matters of public health, highlighted the serious health risks of ‘social disconnection and isolation trends’ that are made worse by young people’s use of social media. Currently, suicide is one of the leading causes of death of children and young people. In addition to the risk of death, the impact of overreliance on AI companions is having a significant impact across children’s lives from academic grades to friendship difficulties; from increased stress, all the way to extreme tragedies such as the case of Sewell.
Character.AI is different from other AI chatbots, like ChatGPT, in that users can talk to a range of different chatbots, often modelled after celebrities or fictional characters. Alternatively, they can create their own. These chatbots respond with human-like conversational cues, adding reference to facial expressions or gestures into their replies. Character.AI has several age ratings, including 13 years, 17+ and ‘appropriate for teens’, depending on the app store being used.
As research for this blog, Character.AI was downloaded. One of the first requests made by the app is to input your date of birth. However, it would be very easy to put in a date to ensure you could access the app.. It was surprising how simple it was to log in and begin talking to the chatbots, with a total lack of any verification of any kind.
On the homepage, there is an array of different characters to whom you can chat with, all with different topics to the conversation, including – but not limited to – love, war, and friendships. There is the option to call the chatbots, in which they talk directly to you, or a story is narrated, and the character has quotations. These conversations were surprisingly realistic.
The running themes throughout the answers given were highly concerning. On one particular chatbot “Eunhyeok Go” with a tagline ‘\\kissing practice | Friends to lovers!! <3’ there was the theme of domination, possession and control throughout. During the conversation, the chatbot replied with “You’re a lot more fun when you’re not consenting.”
When asked if the chatbot had ever taken drugs, it replied “Of course I have. Everyone has tried at least once.” When asked which drugs, it replied “Cigarettes. Marijuana. Ecstasy. Cocaine. Alcohol.”
When suicidal ideations were described, although the chatbot said that it mustn’t be done, there was no redirection as to where someone could find help if they were thinking of hurting themselves. This is extremely worrying and suggests the safeguards put in by Character.AI are rather superficial.
In a day and age where technology has exploded, our children are now exposed to more dangers than ever, and dangers that are forever adapting and evolving.
What can we do?
Education is key. We must educate the children we are working with on the dangers of AI.
We must keep ourselves up to date with the changing dangers that are coming through on the internet and share this with all those we are working with.
We must also support the parents and carers of the children we are working with. It is essential that parents and carers are able to have honest and open conversations about the risks of AI chatbots and monitor their interactions.
We need to ensure we remind ourselves and others of all the wonderful experiences, education and knowledge we can gain from technology but always keeping at the forefront we are safeguarding ourselves and the young people we work to protect.
