Theodoros Zarkos recommends that efforts to protect children from the potential harms of AI should focus on design features that may diminish young people’s agency or foster emotional dependency, rather than treating every emotionally meaningful AI interaction as harmful.

______________________________________

Children and adolescents can experience conversational artificial intelligence as more than a tool. A chatbot may feel familiar, responsive, or easy to talk to. These experiences may offer opportunities for emotional expression or low-pressure social interaction, but can also raise ethical concerns when system design encourages persistent engagement, repeated disclosure, or emotional reliance. The central question is not whether young people can feel connected to AI. It is whether provider-controlled design choices (including prompts and default settings) and accessible user controls preserve their agency to set boundaries, decide how to engage, and disengage without undue pressure.

This distinction matters because protection can become paternalistic when every form of trust, attachment, or emotional closeness is treated as evidence of harm. Young people do not lose their agency simply because an interaction feels relational. These experiences should not, however, be conflated with manipulation, which may involve design practices that exploit trust or vulnerability to unduly influence a young person’s choices. At the same time, their developing capacities can make them more vulnerable to design features that are engineered to maximize attention or make disengagement difficult. A sound ethical approach has to protect both autonomy and well-being.

Photo Credit: Creative Commons. Image Description: An image of binary code.

The European Commission’s proposed EU KIDS Act offers a useful case study. Article 14 introduces safeguards for AI companions and general conversational chatbots accessible to minors. The proposed Act addresses system behaviour likely to create emotional dependency, incorporates anti-addictive and excessive-use measures, and restricts the reuse of information or analysis from a minor’s earlier interactions by default, subject to specified safety and settings exceptions. The proposal would also require pre-deployment testing and post-market monitoring (except where the system is provided by a micro or small enterprise). For chatbots embedded in specified social networking, video-sharing or online gaming services, Article 14(2) additionally addresses automatic activation, encouragement to use the feature, and requires an easy option to opt out at any time.

In a recent study (available as a preprint, not yet peer-reviewed), I systematically examined seven predefined relational mechanisms of AI chatbots and assessed whether they corresponded with safeguards in Article 14 of the proposed EU KIDS Act. This was a descriptive assessment of regulatory correspondence, not a test of safeguard effectiveness. Four mechanisms showed direct correspondence with safeguards in Article 14:

1. dependency-linked relational simulation,
2. emotional dependency or overreliance,
3. conversational memory and the subsequent use of prior-interaction information (subject to specified exceptions), and
4. addictive or compulsive engagement.

Two mechanisms showed partial correspondence:

1. encouragement to engage and disengagement, for which relevant safeguards apply only in specified deployment contexts; and
2. the possible displacement of human relationships.

Trust, emotional closeness, and the authenticity of AI companions showed only indirect correspondence with safeguards in Article 14, as they are not independently regulated outcomes.

These findings suggest that there should be a distinction between feeling connected to a chatbot versus being harmed by it. Trust does not necessarily mean dependency, and sharing feelings does not mean compulsive use. The concern is greater when design choices encourage reliance, exploit vulnerabilities or make it harder for a young person to disengage. The proposed safeguards target some of these more serious design-related risks; they do not establish that every emotionally close interaction is harmful.

Conversational memory is a good example. Remembering earlier exchanges can make an interaction coherent and supportive. It can also deepen the sense that the system “knows” the user and can create a stronger relational history. Yet evidence on the long-term social and emotional effects of conversational AI on minors remains limited; recent reviews describe evidence largely based on short-term observations and designs that cannot establish causation.  In my evidence map, no long-term causal harm study was identified among the nine sources meeting the strict criteria for direct minor-specific empirical evidence.

That uncertainty should not justify inaction, but it should shape the kind of protection we build. Regulation should not assume that a safeguard is effective simply because it targets a plausible risk mechanism. Testing, monitoring, and disengagement controls need to be evaluated in practice. Young people also need to be treated as participants in the design of protection, not merely as passive recipients of adult decisions.

The ethical balance is therefore more demanding than choosing between freedom and safety. Protecting youth autonomy means preserving room for beneficial or meaningful uses of technology while limiting design choices that deliberately deepen dependence or reduce control. The strongest safeguards should distinguish emotional connection from dependency and from manipulative design practices that exploit trust or vulnerability, rather than pathologizing connection itself. They should preserve young people’s ability to understand the interaction, set boundaries, disengage without undue pressure, and exercise meaningful agency, while recognizing the responsibility of providers for the systems they design.

___________________________________________________________

Theodoros Zarkos is a Medical Ethicist, Social Scientist and Independent Researcher based in Kiato, Greece.