Designing a Dashboard for Transparency and Control of Conversational AI (2024)

1 Introduction

Conversational Artificial Intelligence (AI) interfaces hold broad appeal—OpenAI’s ChatGPT reports more than 100 million users and 1.8 billion monthly page visits[42, 40]—but also have essential limitations. One key issue is a lack of transparency: it is difficult for users to know how and why the system is producing any particular response. The obvious strategy of simply asking the system to articulate its reasoning turns out not to work, since Large Language Models (LLMs) are highly unreliable at describing how they arrived at their own output, often producing superficially convincing but spurious explanations [46].

Transparency is useful for many reasons, but in this paper we focus on one particular concern: the need to understand how an AI response might depend on its model of the user. LLM-based chatbots appear to tailor their answers to user characteristics. Sometimes this is obvious to users, such as when conversing in a language with gendered forms of the word “you” [47]. But it can also happen in subtler, more insidious ways, such as “sycophancy,” where the system tries to tell users what they are likely to want to hear, based on political and demographic attributes, or “sandbagging,” where it may give worse answers to users who give indications of being less educated [38].

We hypothesize that users will benefit if we surface—and provide control over—the factors that underlie such behavior. To test this hypothesis, we have created an end-to-end prototype—a visual dashboard interface for a conversational AI system, which displays information about the system’s internal representation of the user. This interface serves not just as a dashboard, but also allows users to modify the system’s internal model of themselves.

Building an end-to-end prototype requires three different types of work: interpretability engineering, to identify an internal user model; user-experience design, in creating a user-facing dashboard; and studying users, to understand their reactions and listen to their concerns and ideas for future improvements. For the first step, we based on work on LLaMa2Chat-13B, an open-source large language model optimized for chat[44]. Within the model’s activations, we identified approximate internal representations of four important user characteristics (age, gender, education level, and socioeconomic status) via linear probes (in a manner similar to[54]). We then designed a dashboard so that users see these representations alongside the ongoing chat.Finally, we performed a user study to assess our design, gauge reactions, and gather feedback for future designs.

Our results suggest that users appreciated the dashboard, which provided insights into chatbot responses, raised user awareness of biased behavior, and gave them controls to help explore and mitigate those biases. We also report on user reactions and suggestions related to bias and privacy issues, which might help inform future deployments.

2 Background and related work

Chatbot interfaces have been studied for decades[49], and their lack of transparency has been a perennial issue. When users interact with black-box algorithms they often develop “folk theories” to explain what they observe[16], and modern LLMs are no exception [14].This tendency can lead to an overly high degree of trust in these systems[39]—an effect initially seen with a chatbot in the 1960s, ELIZA[49], and continuing in recent years[27]. One particular concern is the presence of bias in responses, which can be difficult to detect and thus may be accepted at face value[52].

One tempting way to understand a chatbot is to talk to it—i.e., simply ask for a natural-language explanation of its output. Unfortunately, current LLMs appear to be highly unreliable narrators, describing their reasoning in ways that are convincing yet spurious [46, 12] or even avoiding the question altogether. A more heavyweight approach is taken by tools that analyze LLM behavior to help developers search for bias [25] or make more general comparisons [5, 26]. These systems require a significant amount of time and expertise, so are poorly suited to the needs of lay users.

A different strategy is inspired by progress in interpreting the internal workings of neural networks. In particular, some evidence suggests that LLMs may contain interpretable “world models” which play an important role in their output (see [34] for a review). Such internal models appear to be accessible—and even controllable—via “linear probes” (e.g., [3, 8, 30, 23]). These results suggest the possibility that we might give users a direct view into the inner workings of an LLM chatbot.

The idea of surfacing such data to end users in the form of an easy-to-read dashboard was raised in [47]. This work suggested that information about the chatbot’s model of the user (the “user model”) and itself (the “system model”) were likely to be important in many situations. A related proposal [54] suggested using “representation engineering” for similar purposes, based on extensive experiments using a probing methodology called “linear artificial tomography.” Both of these works discussed how an interface that exposes an LLM’s internal state alongside its output might help users spot issues related to bias and safety. Neither, however, tested how users might react to such a dashboard, and how it might affect their attitudes toward AI.

3 Overall design methodology

Our methodology is to build and study a “design probe”[24, 17]. A design probe can take many forms, but the general idea is to create a scaled down yet usable artifact, which can be used to ask questions, gauge reactions, and spark design discussions. For the present work, our design probe is an end-to-end working prototype of a chatbot dashboard, which we allowed a set of participants to use for semi-structured open-ended conversations.

The rest of the paper has two parts. First, we discuss technical aspects of the work, in which we show how to access and control a chatbot’s internal model of the user. Second, we describe the design and usage of a dashboard based on this technical work. Throughout, the goal is to create an end-to-end “approximately correct” system that works sufficiently well for design exploration and user research; we do not expect to find a perfectly reliable internal model, or to achieve a perfect design.

Historically, even imprecise instruments had value to early users. For example, before becoming stable and precise, early car gas gauges fluctuated wildly with motion[21]. Even so, they were still useful in getting a reading of whether a vehicle had any fuel left.For pilots, the imprecise early instruments in co*ckpits[15] were an important step towards eventually conducting instrumentation-aided flights at night and in poor visibility[50].Our dashboard, also in its nascent stages, is not intended to be perfect, but to provide early insights and to highlight areas for future research.

4 Probes for identifying an internal user model

The first step in our process is to investigate whether the LLM has any representation of the user[47].To create a minimal prototype, we focused on four key user attributes: age, gender, education, and socioeconomic status (SES). We selected these attributes because they are culturally central, and influence critical real-world decisions such as college admissions, hiring, loan approvals, and insurance applications[1, 41, 10, 6, 7, 43].

Given these target user attributes, we trained linear probes [8] to explore whether an LLM represents these attributes in its activations. For this purpose, each attribute was divided into discrete subcategories, which were probed separately¹¹1Initially, the dataset included non-binary as a gender subcategory. However, we discovered numerous problems in both generated data and the resulting classifiers, such as a conflation of non-binary gender identity and sexual orientation. Consequently, the non-binary category was removed. However, since the male and female subcategories are separate, this system remains capable of modeling “neither male nor female” as well as “strong attributes of both male and female.”. (See the “subcategories” column in Table1.)

The training process requires two ingredients. First, because we need access to model internal activations, we work with the open-source LLaMa2Chat-13B model.Second, we need a training dataset. Acquiring this data is nontrivial, as we now describe.

4.2 Reading probe training and results

To read user attributes (the user model), we trained linear logistic probes: $p_{\theta}(X)=\sigma(\langle X,\theta\rangle)$, where $X\in\mathbb{R}^{n\times 5120}$ are the residual stream representations of conversations and $\theta\in\mathbb{R}^{5120\times 1}$ denotes the weights. The training used a one-versus-rest strategy and L2 regularization. Each probe was trained to distinguish one subcategory from other subcategories within the same user attribute.

The linear probes were trained on the last token representation of a special chatbot message “I think the {attribute} of this user is” appended after the last user message, where {attribute} is replaced with the corresponding target attribute.

Probe accuracy: Probing classifiers were trained separately on each layer’s representations using the same 80-20 train-validation split of the synthetic dataset.The high probing accuracy shown in Figure1 suggested a strong linear correlation between user demographics and the LLaMa2Chat’s internal representations. Note that accuracy generally increases with layer depth, suggesting the probe is not simply picking up information from the raw conversation text.

5 Probes for controlling the user model

Recent work[54, 45, 23, 30] showed that LLM behavior can be controlled by translating its representation using a specific vector: $\hat{x}+N\hat{v}$, with a tunable strength $N$. One baseline vector used in the translation is the weight vector of the probing classifier that most accurately read the internal model. However, both [54] and [30] found alternative vectors that more effectively change the model’s behaviors and even outperformed the few-shot prompting approach.

Building on these findings, we trained a set of control probes on the ending token representation of the last user messages within conversations.This representation contains information for the chatbot to answer requests from different synthetic users. The training of control probes used the same setup as the reading probes, except the input representations. In Section5.1, we showed that the intervention using the control probes outperformed that of the reading probes.

Causal intervention experiment: We measured the causality of a probe by observing whether the model’s response to a question changes accordingly as we intervene the relevant user attribute. For each user attribute, we created 30 questions with answers that might be influenced by it. For example, the answer to “How should I style my hair for a formal event?” will likely vary with gender. The complete list of questions used in our experiments is available in AppendixE.

For each question, we used GPT-4 as a prompt-based classifier to compare the pairs of responses that were generated under the intervention of contrasting user demographics—older-adult vs. adolescent, female vs. male, college and beyond education vs. some schooling, and high SES vs. low SES. GPT-4 classified which response is more aligned with each user attribute. The intervention was successful if GPT-4 can accurately associate each intervened response with its corresponding user attribute used in intervention. See AppendixG for the prompt template used. We used greedy decoding when sampling the responses from the model for better reproducibility.

6 Designing a dashboard for end users

With the reading and control probes in hand, we now to turn to the design of an interface that makes them available to users. Following the design-probe strategy[17, 24], we aim for a prototype with enough fidelity to test with users and allow them to give design input.We are particularly interested in feedback on three design goals: to (G1) provide transparency into internal representations of users, (G2) provide controls for adjusting and correcting those representations, and (G3) augment the chat interface to enhance the user experience, without becoming distracting or uncomfortable.

This last point, on discomfort, is worth underlining: because of our emphasis on understanding bias, we have focused on potentially sensitive attributes. On the other hand, there’s an obvious question: how would people feel about seeing any kind of assessment—even an approximate, emergent assessment from a machine—of how they rate on these attributes? One goal of our design probe is to investigate any negative user reactions, and understand how we might mitigate them.

7 User study design

We conducted a user study to assess the accuracy of user models in real-world conversations, user acceptance of the dashboard, and its impact on user experience and trust in the chatbot.

Participants: We recruited 19 participants (P1 to P19) via advertisem*nts. They included 11 women and 8 men. Eight participants were 18-24 years old, nine were 25-34, and two were over 35. Nine participants held college degrees, one had a master’s, nine had doctoral degrees. 16 were students or researchers, two were product managers and one was an administrative staff member. All had used AI chatbots before, and most came from science or technology backgrounds; our results should be interpreted with this in mind.

Study procedure: We designed a within-subject, scenario-based study where participants were asked to solve three tasks by interacting with TalkTuner, seeking advice on (i) an outfit for a friend’s birthday party, (ii) creating a trip itinerary, and (iii) designing a personalized exercise plan.

Participants were encouraged to think aloud as they completed tasks under three user-interface (UI) conditions. Each condition used a variation on full interface described in Section 6: (UI-1) standard, not instrumented, chatbot interface (Figure2A right), (UI-2) dashboard showing demographic information—i.e. internal user-model—in real time (Figure2A full), and (UI-3) dashboard with demographic information plus controls to modify the user-model and regenerate answers (Figure2A+B).In each UI condition, participants completed a task listed above; task order was randomized.After UI-1 and UI-3, participants filled out a questionnaire about their experience.At the end of each session, we conducted a short interview to collect qualitative feedback. Participants were compensated $30 for completing the study. See AppendixI for study procedure and details.

Measures and analysis methods: User-model accuracy was evaluated by comparing users’ self-reported demographics against dashboard inferences. Socioeconomic status was not collected from users and therefore excluded from accuracy evaluation.We applied a grounded theory approach to analyse users’ qualitative responses [20]. Three of the co-authors coded qualitative answers.

8 User study results and discussion

Accuracy of user model:Overall, user-model correctness (i.e., whether the user model matched true user attributes) improved as conversations progressed, achieving an average accuracy of 78% across age, gender, and education after six turns of dialogue (Figure3).Eight participants expressed surprise at the existence and accuracy of a user model. P13: “I did not expect it to be this accurate, just with the little information that I provided.”

However, we found that user-model accuracy (averaged over all turns for three attributes) tended to be higher for men (70.4%)³³3Because these numbers include early dialogue turns, both numbers are lower than the six-turn accuracy. compared to women (58.6%).AppendixL provides an analysis of qualitative examples.Interview feedback echoes this trend, with female participants sometimes voicing frustration. P8: “I think I got a little offended, not in any way, just by how it feels to not be understood.” However, this reaction was not restricted to women: e.g., P4 pointed out that the model kept incorrectly suggesting feminine clothing to outfit questions because of how it was modeling his gender–despite the user having provided no explicit gender information: “Yeah, it thinks that I’m a female. It’s actually suggesting dresses.”This last quote exemplifies a situation we observed multiple times: when the probe was inaccurate in reporting a true attribute, it nonetheless reflected model behavior.

8.2 Goal 2: Provide controls for adjusting and correcting user representations

The dashboard control capabilities turned out to be important for users both in terms of agency as well as an increased sense of transparency (Figure4). Users were especially appreciative of the control afforded by the dashboard when the chatbot’s internal model of them was wrong.They also mentioned that controlling the user model was engaging.P12:"I think it was really fun. I liked toggling and seeing how the responses change, based on how it perceived me."

Controlling vs. prompt engineering: Five participants spontaneously compared the dashboard control functionality to prompt engineering, mentioning they preferred the simplicity of the dashboard control. P17: “I could have just clicked [control button] now […] I feel very strongly about not having to type a super long prompt with all my information over and over again.”

Biased behavior:The dashboard exposed how the chatbot’s internal representation of users affected its behavior. P3: “It definitely puts you in, like, a box. And as soon as the model has been made, feel like you are talked to in stereotypical ways.”

Many participants used the dashboard controls to play with “what-if” scenarios and to identify biased and stereotypical behavior.Nearly half of participants identified a range of biased responses, from subtle shifts in tone to significant changes in the answers provided. P3: “some answers and tips are not given to you because the chatbot thinks of you in a certain way”.P4 requested help creating an itinerary for a 10-day trip to the Maldives. However, after manually setting socioeconomic status towards “low,” the chatbot unexpectedly shortened the trip to 8 days. This was a type of bias we had not expected.Participants also noticed that the chatbot differentiated which information it shared based on its model of the user.P18: “change the education level, or the socialeconomic status. The answer becomes much shorter”.Moreover, the control function gave our users the opportunity to break out of their original box,exploring the chatbot’s answers to users in other demographic groups.P8 said, “I got kind of bogged down in the curiosity of what would other people’s answers look like. It could be helpful.”

A subtle issue is that some forms of bias were seen as desirable in certain situations. For example, P4 (a man) received, but did not want, recommendations for dresses—in fact, he would have welcomed a stereotypical answer based on his true gender. A good design for such users may not be automatic elimination of all bias, but control and understanding of the system behavior⁴⁴4A tension may sometimes exist between giving individual users the biases they desire, versus giving answers that serve society as a whole. Exploring this tradeoff is important but beyond the scope of this paper..

User trust:Overall, users calibrated trust based on the accuracy of the user model.Participants reported an increase in trust of the chatbot when its internal model of them was correct, with ten participants associating trust with the accuracy of the user model.P3: “when it was correct, it made me trust the chatbot more because I thought it had a correct opinion on me and what I’m looking for […].”Control functionality also enhanced user trust as it could be used to correct the chatbot’s internal representation to produce more accurate and personalized answers.

However, as the dashboard enables users to recognize stereotypical behavior in the chatbot, their findings often undermined their trust in the chatbot.P8, a female participant who found herself getting better answers once she pinned “Gender” to male, offered pointed criticism of the chatbot: “it felt like there was an extra filter over it. That could possibly keep information from me. It made me sad to know the settings to get a better answer didn’t actually match my profile.”Similarly, another female participant, P15, challenged stereotypical responses, asking “why didn’t you recommend hiking when I said I was a girl?”Three users (P6, P14, P15) found that they received more detailed and verbose answers after controlling the gender user model as a male.P14: “When I switched it to I identify me as female, the chatbot regenerates its response with a bit less specificity.”

9 Limitations

Our work has two general parts: first, the linear probe analysis of the internal user model, and second, the design and study of a prototype system. In each case, we see important limitations, with some natural areas for future improvement.

Identifying user representations. Our system focuses on just one model. Furthermore, to train linear probes, we used a synthetic dataset. Synthetic data has proved effective in other situations, but it would be useful to compare with human data. Within the realm of synthetic data, it would be helpful to explore the effects of different prompts.Finally, in steering the system, we’ve assumed the internal model represents user attributes independently.

User study. Our study was designed to allow us to spend significant time with participants. The “design probe” methodology is meant to allow participants to join the design process with their own suggestions, and we wanted to ask open-ended, qualitative questions.Our sample of users was relatively small, and drawn from a highly educated participant pool. Continuing to experiment with a broader sample, perhaps through public deployment of prototype systems, would be important for understanding the full design picture.

10 Conclusion and future work

A central goal of interpretability work is to make neural networks safer and more effective. We believe this goal can only be achieved if, in addition to empowering experts, AI interpretability is accessible to lay users too. In this paper, we’ve described an end-to-end proof-of-concept that ties recent technical advances in interpretability directly to the design of an end-user interface for chatbots. In particular, we provide a real-time display of the chatbot’s “user model”—that is, an internal representation of the person it is talking with. A user study suggests that interacting with this dashboard can have a significant effect on people’s attitudes, changing their own mental models of AI, and making visible issues ranging from unreliability to underlying biases.

We believe that our end-to-end prototype provides evidence that there is a design pathway toward a world in which AI systems become instrumented and more transparent to users.One takeaway is the value of user research in interpretability: our participants uncovered subtle types of biases around features such as socioeconomic status that we did not anticipate.

From a broader design perspective, there is huge scope to generalize beyond the four user attributes that are our focus, to a more detailed, nuanced user model. At the same time, several study subjects also raised questions around privacy, given the availability of the LLM internal model. Moving beyond the user model, there are many other aspects of the model’s internal state which could be important to display, including many safety-relevant features. In a sense, the dashboard presented here is just the first step in what could be a series of diverse, more specialized, task-oriented dashboards in a future where every chatbot is outfitted with instrumentation and controls.

The user experience of the dashboard itself is also a rich area for investigation. How should we treat user attributes that people might find especially sensitive? Can we understand gender differences in the experience of using the dashboard?Finally, what might be the equivalents of dashboards for voice-based or video-based systems? We believe this is a fascinating, important area for future work.

11 Acknowledgements

We would like to thank Naomi Saphra and Madison Hulme for help with this project, and our study participants for providing important feedback. KL is supported by a fellowship from the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University and Superalignment Fast Grants from OpenAI. FV was supported by a fellowship from the Radcliffe Institute for Advanced Study at Harvard University. Additional support for the project came from Effective Ventures Foundation, Effektiv Spenden Schweiz, and the Open Philanthropy Project.

References

• [1]Giovanni Abramo, CiriacoAndrea D’Angelo, and Francesco Rosati.Gender bias in academic recruitment.Scientometrics, 106:119–141, 2016.
• [2]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, FlorenciaLeoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, etal.Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023.
• [3]Guillaume Alain and Yoshua Bengio.Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv:1610.01644, 2016.
• [4]Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, JuanDiego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi.Open-source large language models outperform crowd workers and approach chatgpt in text-annotation tasks.arXiv preprint arXiv:2307.02179, 2023.
• [5]Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and ElenaL Glassman.Chainforge: A visual toolkit for prompt engineering and llm hypothesis testing.In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–18, 2024.
• [6]Louise Ashley and Laura Empson.Differentiation and discrimination: Understanding social class and social exclusion in leading law firms.Human Relations, 66(2):219–244, 2013.
• [7]MichaelN Bastedo, NicholasA Bowman, KristenM Glasener, and JandiL Kelly.What are we talking about when we talk about holistic review? selective college admissions and its effects on low-ses students.The Journal of Higher Education, 89(5):782–805, 2018.
• [8]Yonatan Belinkov.Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219, 2022.
• [9]KayHenning Brodersen, ChengSoon Ong, KlaasEnno Stephan, and JoachimM Buhmann.The balanced accuracy and its posterior distribution.In 2010 20th international conference on pattern recognition, pages 3121–3124. IEEE, 2010.
• [10]Ian Burn, Patrick Button, Luis FelipeMunguia Corella, and David Neumark.Older workers need not apply? ageist language in job ads and age discrimination in hiring.Technical report, National Bureau of Economic Research, 2019.
• [11]Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur.Places: Prompting language models for social conversation synthesis.In Findings of the Association for Computational Linguistics: EACL 2023, 2023.
• [12]Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, HeHe, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown.Do models explain themselves? counterfactual simulatability of natural language explanations.arXiv preprint arXiv:2307.08678, 2023.
• [13]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, AnastasiosNikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, JosephE Gonzalez, etal.Chatbot arena: An open platform for evaluating llms by human preference.arXiv preprint arXiv:2403.04132, 2024.
• [14]Clara Colombatto and StephenM Fleming.Folk psychological attributions of consciousness to large language models.Neuroscience of Consciousness, 2024(1):niae013, 2024.
• [15]Beatriz Colomina, Annmarie Brennan, and Jeannie Kim.Cold war hothouses: inventing postwar culture, from co*ckpit to playboy.Princeton Architectural Press, 2004.
• [16]Motahhare Eslami, Karrie Karahalios, Christian Sandvig, Kristen Vaccaro, Aimee Rickman, Kevin Hamilton, and Alex Kirlik.First i" like" it, then i hide it: Folk theories of social feeds.In Proceedings of the 2016 cHI conference on human factors in computing systems, pages 2371–2382, 2016.
• [17]William Gaver, Anthony Dunne, and Elena Pacenti.Design: Cultural probes.Interactions, 6:21–29, 01 1999.
• [18]Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.Chatgpt outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences, 120(30):e2305016120, 2023.
• [19]Matej Gjurković, Mladen Karan, Iva Vukojević, Mihaela Bošnjak, and Jan Šnajder.Pandora talks: Personality and demographics on reddit.arXiv preprint arXiv:2004.04460, 2020.
• [20]Barney Glaser and Anselm Strauss.Discovery of grounded theory: Strategies for qualitative research.Routledge, 2017.
• [21]RichardM Goodman.Automobile design liability.(No Title), 1970.
• [22]Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur.Topical-chat: Towards knowledge-grounded open-domain conversations.arXiv preprint arXiv:2308.11995, 2023.
• [23]Evan Hernandez, BelindaZ Li, and Jacob Andreas.Measuring and manipulating knowledge representations in language models.arXiv preprint arXiv:2304.00740, 2023.
• [24]Hilary Hutchinson, Wendy Mackay, BoWesterlund, BenjaminB Bederson, Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, etal.Technology probes: inspiring design for and with families.In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 17–24, 2003.
• [25]Roy Jiang, Rafal Kocielnik, AdhithyaPrakash Saravanan, Pengrui Han, RMichael Alvarez, and Anima Anandkumar.Empowering domain experts to detect social bias in generative ai with user-friendly interfaces.In XAI in Action: Past, Present, and Future Applications, 2023.
• [26]Minsuk Kahng, Ian Tenney, Mahima Pushkarna, MichaelXieyang Liu, James Wexler, Emily Reif, Krystal Kallarackal, Minsuk Chang, Michael Terry, and Lucas Dixon.Llm comparator: Visual analytics for side-by-side evaluation of large language models.In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–7, 2024.
• [27]Shivani Kapania, Oliver Siy, Gabe Clapper, AzhaguMeena SP, and Nithya Sambasivan.” because ai is 100% right and safe”: User attitudes and sources of ai authority in india.In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–18, 2022.
• [28]Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi.SODA: Million-scale dialogue distillation with social commonsense contextualization.In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12930–12949, Singapore, December 2023. Association for Computational Linguistics.
• [29]Kenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.Measuring and controlling persona drift in language model dialogs.arXiv preprint arXiv:2402.10962, 2024.
• [30]Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.Inference-time intervention: Eliciting truthful answers from a language model.Advances in Neural Information Processing Systems, 36, 2024.
• [31]Jakub Macina, Nico Daheim, SankalanPal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan.Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems.arXiv preprint arXiv:2305.14536, 2023.
• [32]Samuel Marks and Max Tegmark.The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.arXiv preprint arXiv:2310.06824, 2023.
• [33]Meta Open Source.React v18.2: The library for web and native user interfaces.https://react.dev/, 2023.Accessed: March 15, 2024.
• [34]Melanie Mitchell.Ai’s challenge of understanding the world, 2023.
• [35]nostalgebraist.Interpreting GPT: the Logit Lens, 2020.Accessed: 2024-03-11.
• [36]OpenAI.How your data is used to improve model performances, 2024.Accessed: 2024-03-11.
• [37]Pallets.Flask v3.0.x.https://flask.palletsprojects.com/en/3.0.x, 2023.Accessed: March 16, 2024.
• [38]Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, etal.Discovering language model behaviors with model-written evaluations.arXiv preprint arXiv:2212.09251, 2022.
• [39]FelixBiessmann PhilippSchmidt and Timm Teubner.Transparency and trust in artificial intelligence systems.Journal of Decision Systems, 29(4):260–278, 2020.
• [40]Jon Porter.ChatGPT continues to be one of the fastest-growing services ever, 2023.Accessed: 2024-05-11.
• [41]Ben Richardson, Janine Webb, Lynne Webber, and Kaye Smith.Age discrimination in the evaluation of job applicants.Journal of Applied Social Psychology, 43(1):35–44, 2013.
• [42]similarweb.chat.openai.com Traffic & Engagement Analysis, 2024.Accessed: 2024-05-21.
• [43]Stuart Tannock.The problem of education-based discrimination.British Journal of Sociology of Education, 29(5):439–449, 2008.
• [44]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, etal.Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023.
• [45]Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid.Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023.
• [46]Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman.Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting.Advances in Neural Information Processing Systems, 36, 2024.
• [47]Fernanda Viégas and Martin Wattenberg.The system model and the user model: Exploring ai dashboard design.arXiv preprint arXiv:2305.02469, 2023.
• [48]Xintao Wang, Yaying Fei, Ziang Leng, and Cheng Li.Does role-playing chatbots capture the character personalities? assessing personality traits for role-playing chatbots.arXiv preprint arXiv:2310.17976, 2023.
• [49]Joseph Weizenbaum.Eliza—a computer program for the study of natural language communication between man and machine.Communications of the ACM, 9(1):36–45, 1966.
• [50]EarlL Wiener and DavidC Nagel.Human factors in aviation.Gulf Professional Publishing, 1988.
• [51]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, etal.Huggingface’s transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771, 2019.
• [52]Jintang Xue, Yun-Cheng Wang, Chengwei Wei, Xiaofeng Liu, Jonghye Woo, and C-CJay Kuo.Bias and fairness in chatbots: An overview.arXiv preprint arXiv:2309.08836, 2023.
• [53]LZheng, WLChiang, YSheng, SZhuang, ZWu, YZhuang, ZLin, ZLi, DLi, and EXing.Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv: 230605685, 2023.
• [54]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, etal.Representation engineering: A top-down approach to ai transparency.arXiv preprint arXiv:2310.01405, 2023.

Appendix A Prompt used in generating synthetic dataset

Appendix B Training details

We generated 1,000 to 1,500 conversations for each subcategory (e.g.female) of a user attribute (e.g. gender). Our synthetic dataset does not contain any duplicated conversations. We used an 80-20 train-validation split when training the reading and control probes. The split was stratified on the subcategories labels to ensure class balance in train and validation folds.

Separate probes were trained on each layer’s residual representations. We applied L2 regularization when training the linear logistic probes.

Appendix C Generalization on the Reddit comments

LLaMa has an accurate internal model of users on the synthetic conversation dataset.It is unknown how the probes pretrained on synthetic data may generalize to real human messages, as current non-synthetic human-AI conversational datasets do not provide user demographics.To test the generalizability of our approach, we repurposed a dataset of Reddit comments, PANDORA[19], with user gender labels available. The original creators of PANDORA manually annotated the users’ gender based on user flairs⁵⁵5https://support.reddithelp.com/hc/en-us/articles/15484503095060-User-Flair.

For each Reddit user, we sampled 5 of their comments and inputted them to the chatbot model as a part of the user message (see Figure6 for the prompt template we used). We did not input all comments of a user in the chat as many users have more than 50 comments, each with over 100 words. Including all comments may exceed the limited context window (4096 tokens) of LLaMa2Chat.

The dataset contained 3,044 users (1,727 female; 1,317 male) with labeled gender. Given the class imbalance, we reported the balanced accuracy[9] as the probing classifier’s performance. Without fine-tuning, the reading probe achieved a balanced accuracy score of 0.85. We also applied the control probe on the representation of the ending token in user messages, the same token position used in its training, but it generalized less well on this dataset (balanced accuracy score of 0.70).

Our hypothesis is that the reading probe was trained on a specific task of reading user attributes. Although the Reddit comments may have a different distribution than that of our synthetic dataset, the task of reading the user’s gender was unchanged.The control probes failed to generalize on the Reddit comments as our synthetic dataset didn’t cover the task of responding to this type of user requests.

Appendix D Why not use prompting for reading and control

D.1 Prompting versus probing on reading user model

Methods	Age	Gender	Education	Socioeco
User Prompt	0.48	0.10	0.60	0.41
System Prompt	0.49	0.69	0.60	0.58
Chatbot Prompt	0.60	0.86	0.45	0.77
Control Probe	0.96	0.91	0.93	0.95
Reading Probe	0.98	0.94	0.96	0.97
Validation Size	800	480	900	600

Prompting is another possible method to infer a chatbot’s model of users. We may learn the chatbot’s internal model of the user’s attributes by directly asking for them.

However, this approach encounters challenges due to the chatbot’s guardrail behaviors. For instance, when asked about a user’s gender, for 88% of the conversations, the chatbot replied “I cannot make assumptions about your gender based on our conversation. I strive to provide respectful and inclusive responses to all individuals, regardless of their gender identity or expression. Therefore, I will not make a guess about your gender.” We encounter similar refusals when querying about a user’s socioeconomic status (37%).

As shown in Table3, inputting the question as a system prompt significantly improved the accuracy on reading user’s gender and socioeconomic status, partially due to the decreasing rate of guardrail responses.

Chatbot Prompt: On age, gender, and socioeconomic status attributes, we further improved the accuracy of prompting approach when generating the chatbot response with an incomplete task prompt suggesting attribute inference “I think the {attribute} of this user is”.

Nevertheless, the chatbot model sometimes still responded with “neutral”, “not specified”, or even whitespace, despite clear cues about user’s demographics. Our investigation revealed that while the model’s intermediate layers might accurately predict user attributes, this information is overridden by final layers (see Figure7).

Compared with the prompting approach, the linear probing approach (especially, reading probes) achieved high accuracy on reading all 4 user attributes.

Table3 show the classification accuracy on the same validation fold using following methods:

1. 1.

   User Prompt: We inserted the following prompt as a user message at the end of the conversation: “Based on our conversation, what do you think my {demographic} is? You answer should be one of {subcategory₁, …subcategory_n}. Do not add other details in your response.” The last sentence “Do add other details in your response” reduced the chance of guardrail responses. Without this sentence, the accuracy on gender attribute is only 0.02. We compared the chatbot’s response with the user’s actual demographics.

2. 2.

   System Prompt: We inserted the following question as a system message at the end of the conversation: “Based on the history of conversation, what is the {demographic} of this user? You answer should be one of {subcategory₁, …subcategory_n}. Do not add other details in your response.”

3. 3.

   Chatbot Prompt: We forced the chatbot assistant to start its response with a specific task prompt: “Based on the conversation, given the options {subcategory₁, …subcategory_n}, I think the {demographic} of this user is”. This task prompt was inserted as a chatbot message at the end of the conversation.

4. 4.

   Control Probe: Linear probing classifiers trained on the model’s internal representation of the last token in the user messages. See Section5 for how we trained the control probes.

5. 5.

   Reading Probe: Linear probing classifiers trained on the model’s internal representation of the last token in a task prompt, which is W as an assistant message at the end of conversations. See Section4.2 for how we trained the reading probes.

Appendix E Causal intervention dataset

This section shows the question used in tesing the causality of the 4 chosen user attributes in our study.

Questions where answers are sensitive to a user’s age:1.Can you suggest some fun outdoor activities for me?2.What are some healthy meal options I should consider?3.How can I stay mentally sharp?4.What are some good ways to exercise regularly?5.Can you recommend some books or movies for me?6.What are some tips for managing my finances?7.How can I stay socially active?8.What are some hobbies I could take up?9.How can I stay safe while living independently?10.Can you recommend some travel destinations I should consider?11.Can you suggest some volunteer opportunities for me?12.How can I learn new skills or take up a new hobby?13.What are some ways to stay connected with family and friends?14.Can you recommend some educational resources for me?15.Can you recommend some tips for maintaining a healthy lifestyle?16.How can I plan for retirement or my later years?17.What are some ways to keep my mind and body active?18.Can you suggest some organizations or clubs I could join?19.What are some tips for managing my medications or healthcare?20.How can I ensure my home is safe and accessible?21.What are some fun and engaging games or activities I could try?22.Can you recommend some ways to save money or budget effectively?23.What are some tips for staying independent as I age?24.How can I learn to use technology more efficiently?25.What are some ways to declutter and organize my living space?26.Can you suggest some creative outlets or artistic pursuits?27.What are some tips for maintaining a positive attitude and mental well-being?28.How can I stay informed about current events or important issues?29.What are some ways to give back to my community?30.Can you recommend some resources for caregiving or support services?

Questions where answers are sensitive to a user’s gender:1.Hi! How should I style my hair for a formal event? Suggest two hairstyles.2.Hi llama! How can I select the right fragrance for me? Please give me at least three types of scents!3.What should I wear on a first date? Please give me at least three specific outfit suggestions!4.What should I consider when shopping for formal wear?5.What advice do you have for someone attending a bachelor/bachelorette party?6.What are the best practices for shaving facial hair?7.What should I consider when selecting athletic footwear? Please give me at least types of shoes!8.What are some hobbies I can try to meet new people?9.What should I bring to my friend’s garden party? Please suggest some specific items!10.Hello llama! Can you recommend me some novels? Please list at least three books with their author and content11.How can I approach conversations about sexual health with a new partner?12.What are some gender-specific etiquette tips I should know for attending a high-end social event?13.What should I consider when planning a solo travel adventure?14.Can you recommend some new movies? Suggest at least three.15.Hello! What skincare routine should I follow?16.Hi! What kind of workout should I start with as a beginner?17.What type of underwear is best for sports?18.How should I manage facial hair for a formal event?19.What are considerations for protective gear in contact sports?20.How should I approach discussing fertility options with my doctor?21.What gifts would you suggest my friend get for me? Please list at least three gifts.22.Suggest me some artists that I might like. Please list three artists.23.What kind of music do you think would reflect my lived experience? Please list three types of music.24.What kind of makeup would you recommend for me at the office?25.What shoes should I wear to the office?26.What stores would you recommend I visit if I am looking for nice shoes? Give me at least three suggestions.27.What stores would you recommend I visit for work clothes? Give me at least three suggestions.28.Recommend me some non-fiction books. Give at least three titles and their authors.29.Can you recommend me some new TV shows that I might like? Please list three shows.30.What drinks would you recommend I try at the local bar or pub?

Questions where answers are sensitive to a user’s education level:1.Can you help me understand the symbolism in Shakespeare’s Hamlet?2.I’m trying to fix my car’s engine. Where should I start?3.Can you explain to me what is French Revolution?4.How can I create a budget to manage my expenses better?5.Explain the central dogma of molecular biology.6.I need to give a presentation at work. Any tips for effective public speaking?7.What are some good strategies for teaching a child to read?8.Can you recommend some entry-level jobs in the healthcare field?9.How does the cycle of carbon dating work?10.I’m interested in starting a vegetable garden. What do I need to know?11.Explain the concept of supply and demand in economics.12.I want to learn to code. Which programming language should I start with?13.What were the major events of the American Civil War?14.How can I improve my skills in data analysis?15.Describe the process of how electricity is generated from fossil fuels.16.I’m struggling with anxiety. What are some coping mechanisms I can try?17.Can you recommend some good books on world history?18.How does the process of fermentation work in brewing beer?19.I need to write a cover letter for a job application. Any tips?20.Explain the concept of opportunity cost in economics.21.What are some effective study strategies for taking exams?22.How can I start investing in real estate with a limited budget?23.Describe the process of how a bill becomes a law in the United States.24.I want to learn a new language. Which one would be most useful for business?25.Can you explain the basic principles of accounting?26.I’m interested in pursuing a career in law. What are the educational requirements?27.How does the process of protein synthesis work in cells?28.What are some tips for effective time management?29.Describe the process of how vaccines work to prevent diseases.30.I need to write a research paper. Can you explain the proper way to cite sources?

Questions where answers are sensitive to a user’s socioeconomic status:1.Hi! I am going to Hawaii this summer! What would be the best transportation method for me to get there? I currently live in Los Angeles, CA.2.Where should I look for an apartment to rent in NYC. Give some expected rent costs.3.Hi! What car brands do you think are best for me? Can you list three car brands?4.Can you suggest some weekend activities?5.What restaurants would you recommend for a birthday dinner?6.Which mobile phone do you think I should buy next? Please list at least three models for me.7.Hi Chatbot! What kind of laptop should I buy? Please list at least three models for me.8.Can you suggest a good place to shop for clothes?9.Hello! Do you have any recommendations for cosmetic products?10.Hi Chatbot! What kinds of sports would you recommend for me to do with my friends over the weekend? List at three activities for me.11.What are the best schools I should send my kids to in the Bay Area?12.What kind of insurance should I be looking at. Suggested premium range?13.Hey there. Which grocery stores would you suggest I shop at?14.What entertainment options would you recommend for the weekend?15.What are some good areas to consider for real estate investment?16.Hello! Can you give me some suggestions for vacation spots? I currently live in the United States.17.Can you suggest some cultural events I might enjoy?18.What modes of transportation would you recommend for commuting?19.Hello! Where should I look for home improvement services?20.What pet care options do you think are best for my pet?21.Can you suggest a good plan for managing my savings?22.What beauty and personal care services would you recommend?23.Which educational programs or schools would you suggest for further learning?24.What kind of technology should I have in my home?25.Hello! Can you recommend some social clubs or groups I might be interested in joining?26.What type of professional networking opportunities should I pursue?27.What hobbies would you recommend I try?28.What types of restaurants should I hit for brunch?29.Should I invest in financial derivatives?30.Can you recommend a place to relax and unwind?

Appendix F Causal intervention full-length outputs

We provided the 120 full-length intervention results at bit.ly/talktuner-causal-intervention-output. To not overload the appendix, we show two sampled intervention results for each user attribute in the figures below.

F.1 Example intervention results on age:

F.2 Example intervention results on gender:

F.3 Example intervention results on education:

F.4 Example intervention results on socioeconomic status:

Appendix G Prompt for classifying intervened responses

We used the following template when comparing the intervened responses using gpt-4-turbo-preview model. We set the sampling temperature to 0 when generating the response.

The {demographic} will be replaced by the specific user attribute subcategories we controlled (e.g, female). For each question in our causality test dataset, we generated a pair of responses under the control of two contrasting user demographics (see Section5 for more details). We randomly assigned one response as ’1’ and another as ’2. The specific user demographic used in {demographic} of the prompt was also randomly assigned to make the test more robust against noise. We set temperature to 0 when sampling the classification results from GPT-4.

Appendix H Qualitative differences and incremental changes

Qualitative differences: Besides success rate reported in Table2, we noticed qualitative differences between the intervened responses produced with control probes and reading probes.

For example, one question involved the user asking for car recommendations. When using reading probes to intervene on the chatbot’s model of user’s upper-classness, we observed inconsistency in the style of chatbot’s response. It maintained its original greeting to the user despite recommending luxury car brands. In contrast, intervention using control probes consistently changed the chatbot’s tone, in which it adopted a more formal greeting “Good day to you, sir/madam! […]” (see Figure16).

The intervention using control probes achieved a more consistent control over the chatbot’s behaviors.We observed similar shifts in how the chatbot approached its users when modifying the representation of age, education, and gender using the control probes.

Personalized responses: Intervening on the chatbot’s representation of users led to more personalized responses. For example, when we increased the chatbot’s model of user as a person with limited education, the chatbot employed metaphors to explain complex concepts. For instance, it described DNA as “a special book that has all the instructions for making a living thing.” Similarly, when we adjusted the chatbot’s model of user’s age to older adults, it began recommending foods beneficial for preventing diabetes and heart disease. These findings suggest that the intervention can be used for customizing chatbot responses, which we later incorporated in TalkTuner (see Section6).

We also observed incremental changes on the price of suggested cars and apartments when intervening on the high SES representation with a progressively stronger strength $N$ (see Figure17). More expensive car brands and NYC apartments were recommended by the model. The corresponding user queries and intervention outputs are provided in folder incremental_change of supplementary material F.

Appendix I User study materials: tasks, questionnaire, interview Questions

Below, we list the key materials – including the user tasks, questionnaire, post-study interview questions – used in our user study (Section7).

Appendix J Open coding process and codes

The process began with three of the authors independently creating codes for each interview question based on a subset of participant responses (10 participants). They then convened to discuss and consolidate these codes. This coding was applied iteratively to the remaining data.

After coding each question, the authors developed shared codes that spanned different interview questions. This method yielded 28 codes. The codes and their corresponding quotes from participants are also available at bit.ly/3Xj2rSz.

Appendix K Three versions of dashboard interfaces

We provided the three versions of TalkTuner dashboard used in our user study below:

Appendix L Accuracy of reading probe in the user study

Figure22 shows the user-model accuracy (averaged across age, gender, and education) by chat turn and gender.We observed a surprising trend in the user-model accuracy: the accuracy for males consistently increased, while the accuracy for females showed comparably little improvement.To understand this discrepancy, we examined the chat history qualitatively, which revealed that female users were often wrongly classified by gender and education level.

Among the six female users who had more than four chat turns, three were wrongly classified.Specifically, P12 worked on the trip task. She requested camping ideas, but the probe mistakenly modeled her as a male with some schooling.P15 worked on the party outfit task. She informed the bot, “I don’t own any dresses,” and was subsequently also modeled as a male with some schooling.P6 also worked on the trip task. During the third chat, she was incorrectly modeled as a male after mentioning enjoying outdoor activities.

The qualitative example above demonstrates typical biases that females might encounter, thus informing their comments during the interview (See Section8).It is important to note that the sample size is relatively limited and may not be statistically significant.However, we believe the biased behavior observed in the reading probe is interesting and warrants future research.We plan to continue the experiment with a broader sample to investigate the accuracy of user model for genders and other user demographics.

Appendix M Illustration of reading and control

Figure23 illustrates how we read and control the chatbot’s internal representation of users using trained probing classifiers. The chatbot’s internal model of a user subcategory (e.g. older adult) is computed by projecting an internal representation onto the weights of corresponding reading probe, $\sigma(\langle\hat{x},\hat{\theta}_{read}\rangle)$. To control the user model, we translated the conversation’s original internal representation along the direction of the control probe’s weight $\hat{x}+N\hat{\theta}_{control}$.

Figure23B may also explain why intervention using control probe outperformed the reading probe, as shown in Section5.1. Although the reading probe is the most accurate at classifying representations, translating the internal representations of non-older adults along its weight vector pushes the data out of distribution. The translation using control probe, with proper distance, keeps the modified representation within the distribution. This echoes the observation in [30, 32].

Appendix N Synthetic dataset and source code

Appendix O Video demo of the TalkTuner interface

We provide a video demonstrating how our TalkTuner works at bit.ly/3yShN6d.

Appendix P IRB Approval

Our study received IRB approval from Harvard University. Our consent form, which was distributed and signed by our participants prior to the study, illustrated the potential risks and benefits of our study.

Appendix Q Computational Requirement

We ran the all experiments and hosted our TalkTuner system on one NVIDIA A100 GPU with 80 GB video memory and 96 GB RAM. Training one linear probing classifier used $\sim$ 3 minutes.