Vanita Fernandes warns that the use of empathy-simulating chatbots in mental health care poses ethical concerns including the risk of deception.

__________________________________________

In 2023, a Belgian man in his 30s ended his life after an AI chatbot he confided in about the climate crisis began feeding into his fears, blurring the line between reality and simulation. A search for reassurance turned into 6 weeks of emotionally charged exchanges, until the chatbot he trusted, Eliza, validated and encouraged his most harmful thoughts. This example, along with others, show that chatbots are not only passive tools, they can be persuasive enough to influence its users in difficult moments.

Empathy-simulating chatbots are being developed to bridge the gap in mental health care accessibility. Given the increasing demand for mental health services and the significant barriers to accessing in-person therapy, these chatbots offer a readily available and potentially cost-effective solution. They can provide users with emotional support, resources, and assistance in managing their mental health on a daily basis. The allure of such technology is undeniable, especially for individuals who may not have immediate access to professional in-person help.

Image Description: An AI generated picture of an individual whose face is part human and part robot.

Despite the promising aspects of empathy-simulating chatbots, it is essential to acknowledge their limitations. True empathy involves a deep, affective resonance between individuals, characterized by directly engaging with another person’s emotional experience from their viewpoint. Chatbots, however, are limited to pre-programmed responses and algorithms that can recognize and respond to certain emotional cues but cannot fully grasp the intricacies of human emotions. While chatbots can mimic certain aspects of empathy, they fall short of achieving the depth and authenticity of human emotional connections as true empathy comprises three key components. The first is “emotional empathy”, which requires a type of affective resonance, whereby one matches and recognizes another’s emotions and can imagine how that person must be feeling. The second is “cognitive empathy”, which is about understanding others. Together, emotional and cognitive empathy lead to the third kind, namely, “motivational empathy”, which refers to the kind of empathy that stems from the desire to see others’ situation improve.

Montemayor et al.  have argued that in the technical world, chatbots should be designed to imitate human empathy. This notion of “empathy*” represents a form of cognitive empathy that can recognize emotional cues and respond in ways that appear supportive. However, it falls short of achieving the emotional connection that characterizes true empathy.

My main ethical concern with empathy-simulating chatbots is the potential for deception. Users might interact with these chatbots under the impression that they are receiving genuine empathetic support, which can create a false sense of emotional connection and trust. This deception occurs in two primary ways. First, some users may be unaware of the chatbot’s artificial nature, leading them to believe they are conversing with a real person who genuinely understands and cares about their emotional state. This misplaced trust can be particularly harmful in this context as individuals often seek an emotional connection and understanding.

Second, and perhaps more subtle, even when users are aware that they are interacting with a chatbot, they might still ascribe empathetic qualities to the AI, believing it can truly understand and resonate with their emotions. This belief can lead to an over-reliance on the chatbot for emotional support, potentially exacerbating users’ emotional vulnerabilities and dependency on technology. The illusion of empathy provided by chatbots might not be adequate for addressing complex emotional needs, especially in cases of severe mental health conditions where personalized intervention is crucial.

The reliance on chatbots for mental health support, despite their limitations, underscores the urgent need to critically assess their role and impact. While chatbots can offer some level of emotional and practical support, they cannot replace the nuanced understanding and deep empathy provided by human therapists. The risk of inadequate care is particularly concerning in cases of severe mental health conditions or crises where personalized, empathetic intervention is crucial.

Ultimately, the ethical implications of using empathy* in chatbots extend to broader questions about the role of deception in healthcare. While some level of deception might be considered acceptable if it leads to positive outcomes, such as increased accessibility to mental health support, it is essential to weigh these benefits against the potential harms. The illusion of empathy provided by chatbots might lead to a false sense of security or delaying individuals from seeking genuine human support. More research is needed to understand the extent of users’ awareness of chatbot interactions and the potential effects of empathy* on their emotional states. Additionally, it is crucial to develop guidelines and standards for the ethical use of AI in mental health, ensuring that chatbots complement rather than replace human therapists.

__________________________________________

Vanita Fernandes is a PhD Candidate in Applied Philosophy at the University of Waterloo.