National Oxford Uehiro Prize in Practical Ethics: Undisclosed Conversational AIs: A Threat to Users’ Autonomy

This article received an honourable mention in the graduate category of the 2024 National Oxford Uehiro Prize in Practical Ethics. Written by Beatrice Marchegiani.

Introduction

Recent advancements in Large Language Models have enabled AI systems to engage in conversations with users that are virtually indistinguishable from human interactions. The proliferation of advanced Conversational AIs (CAIs)¹ in an increasingly online-interaction-dependent world raises concerns about users being unable to distinguish between AI and human interactions. A 2023 study² revealed that users can accurately distinguish human interlocutors from CAIs only 60% of the time, slightly better than chance. While some scholars advocate for explicit disclosure whenever CAIs are used³, little has been said on why undisclosed CAIs are problematic. I argue that failing to disclose the use of CAIs during user interactions is problematic as it diminishes users’ autonomy. In section 1 I discuss the prevalence of undisclosed CAIs and argue that users often reasonably mistake them for human interlocutors. In section 2 I explore the idea that false beliefs undermine the agent’s autonomy, and outline two characterizations to identify which false beliefs undermine autonomy. In section 3 I argue that, within the context of interacting with a CAI, believing that one is conversing with a human meets the criteria for a false belief that undermines autonomy.

Undisclosed CAIs result in false beliefs

Interacting with CAIs has become a daily occurrence for many internet users. These systems now play crucial roles in various real-life situations, from customer service and social-media⁴, to critical areas like medical hotlines⁵ and finance⁶. In certain jurisdictions companies must disclose their use of CAIs⁷. However, this requirement is not universal. Users respond negatively to disclosed chatbots⁸: This creates an incentive for companies to not disclose their use of CAIs. In many casual interactions with undisclosed CAIs, there are no indicators that the interlocutor is an AI, leading users to assume they are conversing with another human. This assumption is reasonable given that until recently only humans were able to engage in sophisticated conversations.

While distinguishing between AI and humans in a Turing Test scenario may still be feasible (where users explicitly question their interlocutors to identify their nature), I argue that this is impractical for the average user. It is unreasonable to expect users to perform a Turing Test regularly given the pervasiveness of online interactions in our daily lives. Additionally, considering the expected progress in these technologies, it is likely that distinguishing them from humans will become even more challenging.

Summing up, there is a significant risk of users misidentifying undisclosed CAIs as human. This false belief poses a threat to the user’s autonomy, as I argue below.

How false beliefs undermine autonomy

Broadly construed, autonomy is the ability to act according to one’s values. An agent’s autonomy with respect to an action is contingent upon the agent having a sufficient level of understanding and being aware of certain relevant information. For example the agent must hold some general understanding of the world she inhabits. She must also be aware of the nature of the action, recognize it as an available option, and be aware of possible alternative actions. If the agent lacks knowledge or holds false beliefs about relevant information, her autonomy in that action is compromised.

The connection between understanding and autonomy has roots in the Aristotelian idea that actions performed from reasons of ignorance are not voluntary⁹ and remains a common theme in contemporary discussions on autonomy, especially within medical ethics as expressed by the concept of informed consent¹⁰. It intuitively makes sense to believe that information is crucial for autonomy¹¹, after all, one can’t make choices aligned with their values if they’re unaware of their options. The challenge lies in determining what information is relevant. Expecting individuals to know every detail about an action is too demanding and sets an unachievable standard for autonomy. In this paper, I explore the two primary characterizations of relevant information for autonomy.

The first characterization revolves around the connection between an agent’s actions and her underlying intention. Autonomy in an action is contingent on a connection between the action and the motivating intention. Some argue¹² that autonomy is compromised when false beliefs sever this connection. Expanding on this idea provides a criterion for evaluating whether false beliefs or ignorance undermines an agent’s autonomy:

To illustrate how this test operates, consider the following. If I board a train thinking it’s going to Oxford but it’s actually headed to Cambridge, my false belief about the destination (whether it stops at Oxford) hinders my understanding of the action. This lack of understanding prevents me from acting in a way that aligns with my intention, leading me to board a train for Cambridge despite intending to travel to Oxford. The connection between my intentions and my action is compromised by the false belief, thereby undermining my autonomy to some extent¹³.

The second characterization is borrowed from the literature on informed consent. I find it apt to draw a parallel between bioethics and digital ethics when discussing disclosure and autonomy. There is a notable similarity, as both involve disclosure from two parties (patient/doctors and users/tech company) with a power imbalance and informational asymmetry. In both scenarios, we are dealing with information that the patient/user cannot reasonably be expected to know or figure out independently.

In bioethics, informed consent is designed to protect patients’ autonomy by ensuring that individuals have the necessary information to make meaningful decisions about medical care. This process entails providing patients with information that is material to their decision such as details about the nature, purpose, risks, benefits, and alternatives of a medical intervention. The Montgomery ruling¹⁴ introduces a definition for material information that has been adopted in the literature. According to this definition, material information is what a reasonable person in the patient’s position would consider when making decisions, or information that the doctor should be aware the patient would take into account. The second part of this disjunction is not relevant for this paper, so let’s drop it :

False belief about the identity of the interlocutor undermines users’ autonomy in CAI interactions.

Holding a false belief that one’s conversational partner is human, when in reality, it is an AI, meets the criteria of both the [intentions test] and the [materiality test], thereby undermining the user’s autonomy.

To illustrate how misidentifying a CAI as human compromises users’ autonomy, first consider an analogous example known as the “hi mum” scam”¹⁶. In this scam, perpetrators text victims from an unfamiliar number, posing as the victim’s adult child and requesting a money transfer. The message typically reads, “Hi mum, I’ve lost my phone and am texting you from a passerby’s phone. Can you transfer me some money to get a cab home?” In this situation, the victim’s decision to transfer money cannot be deemed fully autonomous because the false belief about the identity of the message’s author severs the connection between the action (transferring money) and the intended goal (protecting their child). Moreover, no reasonable individual would have transferred the money to the scammers had they known the true nature of the interaction. Thus, the identity of the interlocutor satisfies both the [materiality test] and the [intentions test].

Undisclosed CAIs resemble a “hi fellow human” scam, deceiving users into thinking they are interacting with a human. The impact of such false beliefs extends to the initial decision of whether to engage in the conversation and any subsequent decisions within the interaction. Just as in the “hi mum” example, this false belief leads users to behave differently than they would if they knew their conversational partner was an AI. Human-to-human interactions follow established behavioural norms. When users mistakenly think they are engaging with a human instead of an AI, they wrongly apply human-to-human behavioural norms to their interactions with the CAIs. This assertion finds support in empirical evidence. Consider, for example, a study¹⁷ where participants engaged with either a disclosed or undisclosed CAI, both attempting to sell a product. Undisclosed CAIs matched the sales performance of experienced human workers. Yet, when users knew they were interacting with CAIs, sales plummeted by 80%, and interactions became more concise. This implies users mistakenly apply human norms to undisclosed AI interactions. In these situations, users were more willing to engage due to expectations of empathy and politeness typical in human interactions, leading to increased trust and more sales.

Users deciding to apply human-to-human norms when interacting with an undisclosed CAI is comparable to the victim of the ‘hi mum’ scam choosing to transfer money to the scammer. In both instances, the agent’s autonomy is compromised due to the false beliefs they hold.

Consider this example: Smith reports a spam post on their social media through the help chat, unknowingly interacting with a CAI. Remaining polite and patient, Smith answers the AI’s questions about their site usage.

Smith applies several human-to-human norms: her politeness and her willingness to disclose personal information reflects social norms of courteous and informative communication.

When deciding whether to be polite or disclose information, a reasonable person in Smith’s position would want to know if they are interacting with an AI. Politeness in communication is only reasonable if the interlocutor can be offended, a principle not applicable to interactions with AI. Similarly, the decision on what information to disclose depends on the capabilities of the interlocutor. In human-to-human interactions, people share information assuming humans might forget details and trust their judgement to safeguard personal information. However, these considerations do not apply to AI. Thus the false belief that the interlocutor is human satisfies the [materiality test]

Smith’s politeness is motivated by empathy for what she believes is a human customer service representative. Smith acknowledges that humans have feelings and sensitivities, and she intends to avoid causing offence. Smith’s mistaken belief about the identity of her interlocutor disconnects her action (being polite to the CAI) from her intention (wanting to be polite due to empathic concern for a fellow human being). A parallel argument can be made for Smith’s decision to disclose personal information, which she does intending to facilitate conversational continuity and connection-building. However, the AI lacks the characteristic that justifies such a choice. In both instances the adoption of human-to-human norms is driven by the agent’s recognition of certain characteristics in fellow humans (ability to be offended, and ability to create a connection). If the interlocutor lacks the characteristic that underlies a specific norm, the agent’s choice to adopt the norm no longer reflects the intention to acknowledge that particular characteristic. Thus, having a mistaken belief about the CAI’s identity fulfils the [intention test] when deciding whether to apply human-to-human norms in an interaction.

False belief about the identity of the CAI satisfies both the [intention test] and the [materiality test]. Consequently, a user who falsely believes that a CAI is a human has her decisional autonomy undermined in the context of that interaction.

Conclusion

This paper argues that undisclosed CAIs result in a loss in users’ autonomy. I showed that undisclosed AI often leads to users falsely believing that they are interacting with a human. I then argued that in the context of interacting with a CAI, having a false belief about the CAI being human is the kind of false belief that undermines the user’s autonomy. Given the cumulative impact of numerous undisclosed interactions in an increasingly online world, this loss of autonomy raises significant concerns.

Footnotes:

¹ By Conversational AI I mean any AI capable of engaging in natural language interactions with users

² Daniel Jannai, Amos Meron, Barak Lenz, Yoav Levine, and Yoav Shoham. “Human or Not? A Gamified Approach to the Turing Test.” arXiv preprint arXiv:2305.20010 (2023).

³Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. “Generative agents: Interactive simulacra of human behavior.” In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22. 2023.

⁴ Cristina Criddle, “How AI-created fakes are taking business from online influencers” Financial Times, December 9, 2023, https://www.ft.com/content/e1f83331-ac65-4395-a542-651b7df0d454

⁵ “Using an AI chatbot to streamline mental health referrals” NHS, accessed January 30, 2023, https://transform.england.nhs.uk/key-tools-and-info/digital-playbooks/workforce-digital-playbook/using-an-ai-chatbot-to-streamline-mental-health-referrals/#:~:text=Limbic%20is%20the%20most%20widely,f or%20IAPT%20services%20so%20far.

Lauren Aratani, “US eating disorder helpline takes down AI chatbot over harmful advice,” The Guardian, May 31, 2023,.https://www.theguardian.com/technology/2023/may/31/eating-disorder-hotline-union-ai-chatbot-harm

⁶ “Chatbots in consumer finance,” Consumer Finance Protection Bureau, last modified June 06, 2023, https://www.consumerfinance.gov/data-research/research-reports/chatbots-in-consumer-finance/chatb ots-in-consumer-finance/#note27

⁷ “Section 17941 – Unlawful use of bots,” California Business and Professions Code, Division 7, General Business Regulations, Chapter 6 – BOTS

⁸ Stefano Puntoni, Rebecca Walker Reczek, Markus Giesler, and Simona Botti. “Consumers and artificial intelligence: An experiential perspective.” Journal of Marketing 85, no. 1 (2021): 131-151.

⁹ Aristotle, Nicomachean Ethics, 1110a

¹⁰ Alfred R Mele. “Autonomy and Beliefs.” Thick (Concepts of) Autonomy: Personal Autonomy in Ethics and Bioethics (2022): 87-100.

Suzy Killmister. “Autonomy and false beliefs.” Philosophical Studies 164, no. 2 (2013): 513-531.

Jonathan Pugh, Autonomy, rationality, and contemporary bioethics. Oxford University Press, 2020: 131-136

Julian Savulescu, and Richard W. Momeyer. “Should informed consent be based on rational beliefs?.” Journal of medical ethics 23, no. 5 (1997): 282-288.

¹¹ The idea that false beliefs undermine autonomy is not without critics (see for example: Michae McKennal. “The relationship between autonomous and morally responsible agency.” In J. Stacey Taylor (ed.), Personal Autonomy: New Essays on Personal Autonomy and its Role in Contemporary Moral Philosophy.(2005) pp. 205–34. ). In any case for the purpose of this paper I am going to assume that lack of relevant information results in a diminished autonomy.

¹² Killmister (2013), Pugh (2020) and Mele(2022)

¹³ Not all false beliefs affect my autonomy equally. For example, thinking the train stops at Oxford on its way to Manchester, when it actually terminates at Oxford, wouldn’t undermine my autonomy. The crucial information for my decision is whether the train stops at Oxford, not its final destination.

¹⁴ Montgomery v Lanarkshire Health Board (2015) SC 11 [2015] 1 AC 1430.

¹⁵ As I seek to apply this test to a non medical context, I replace “patient” with “agent.”

¹⁶Jess Clark and Alex Hern, “Victim of ‘Hi Mum’ fraud on WhatsApp lost,” The Guardian, June 16, 2023, https://www.theguardian.com/technology/2023/jun/16/victim-of-hi-mum-on-whatsapp-lost-1600

¹⁷ Luo, Xueming, Siliang Tong, Zheng Fang, and Zhe Qu. “Frontiers: Machines vs. humans: The impact of artificial intelligence chatbot disclosure on customer purchases.” Marketing Science 38, no. 6 (2019): 937-947.

Bibliography

1. Aratani, L. “US eating disorder helpline takes down AI chatbot over harmful advice,” The Guardian, May 31, 2023, .https://www.theguardian.com/technology/2023/may/31/eating-disorder-hotline-union-ai-chatb  ot-harm
2. Aristotle, Nicomachean Ethics, 1110a
3. California Business and Professions Code “Section 17941 – Unlawful use of bots,” Division 7, General Business Regulations, Chapter 6 – BOTS
4. Clark, Jess, and Alex Hern, “Victim of ‘Hi Mum’ fraud on WhatsApp lost,” The Guardian, June 16, 2023, https://www.theguardian.com/technology/2023/jun/16/victim-of-hi-mum-on-whatsapp-lost-160  0
5. Consumer Finance Protection Bureau,“Chatbots in consumer finance,” last modified June 06, 2023, https://www.consumerfinance.gov/data-research/research-reports/chatbots-in-consumer-finan  ce/chatbots-in-consumer-finance/#note27
6. Criddle, C. “How AI-created fakes are taking business from online influencers” Financial Times, December 9, 2023, https://www.ft.com/content/e1f83331-ac65-4395-a542-651b7df0d454
7. Jannai, Daniel, Amos Meron, Barak Lenz, Yoav Levine, and Yoav Shoham. “Human or Not? A Gamified Approach to the Turing Test.” arXiv preprint arXiv:2305.20010 (2023).
8. Killmister, S. “Autonomy and false beliefs.” Philosophical Studies 164, no. 2 (2013): 513-531.
9. Luo, Xueming, Siliang Tong, Zheng Fang, and Zhe Qu. “Frontiers: Machines vs. humans: The impact of artificial intelligence chatbot disclosure on customer purchases.” Marketing Science 38, no. 6 (2019): 937-947.
10. McKennal, M. “The relationship between autonomous and morally responsible agency.” In J. Stacey Taylor (ed.), Personal Autonomy: New Essays on Personal Autonomy and its Role in Contemporary Moral Philosophy.(2005) pp. 205–34.
11. Mele, A. R. “Autonomy and Beliefs.” Thick (Concepts of) Autonomy: Personal Autonomy in Ethics and Bioethics (2022): 87-100.
12. Montgomery v Lanarkshire Health Board (2015) SC 11 [2015] 1 AC 1430
13. NHS, “Using an AI chatbot to streamline mental health referrals”, accessed January 30, 2023, https://transform.england.nhs.uk/key-tools-and-info/digital-playbooks/workforce-digital-playbo  ok/using-an-ai-chatbot-to-streamline-mental-health-referrals/#:~:text=Limbic%20is%20the%2  0most%20widely,for%20IAPT%20services%20so%20far.
14. Park, Joon Sung, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. “Generative agents: Interactive simulacra of human behavior.” In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22. 2023.
15. Pugh, J. Autonomy, rationality, and contemporary bioethics. Oxford University Press, 2020: 131-136
16. Puntoni, S. Rebecca Walker Reczek, Markus Giesler, and Simona Botti. “Consumers and artificial intelligence: An experiential perspective.” Journal of Marketing 85, no. 1 (2021): 131-151.
17. Savulescu, J, and Richard W. Momeyer. “Should informed consent be based on rational beliefs?.” Journal of medical ethics 23, no. 5 (1997): 282-288.