Robots in Groups Lab · AI IN COMMUNICATION

How might we design AI systems to improve communication?

As AI continues to see dramatic increased use in everyday life, it is critical to understand its impact on our social interactions. Through a mixed methods research process, we investigated the social consequences of using AI in communication and how we can design systems that improve interpersonal outcomes.

BY THE NUMBERS

  • 2,000+Users communicated through our experimental AI messaging tool
  • 700+Individual test sessions organized, recorded, & analyzed
  • 13,000+User messages linguistically analyzed
  • $100,000+Research money awarded for grant proposals I wrote
DATE
Fall 2018 - Summer 2020
ORGANIZATION
Robots in Groups Lab
ROLE
ResearchDesignProject Management
DELIVERABLES
PUBLICATIONS
Nature Scientific ReportsCHIComputers in Human BehaviorDissertationCSCWCSCW
PRESS
WiredSparkDialogLaddersCornell Chronicle

What did we do?

  • Generative Testing & Interviews

    Conducted generative in-lab testing and follow-up interviews to investigate user perceptions and potential effects of AI on communication

  • Messenger Design

    Collaborated with computer scientists to design a web-based experimental AI messaging tool that we deployed in multiple remote, online experiments

  • Moderated Online Testing

    Conducted moderated online tests to determine how AI affects interpersonal dynamics

  • Unmoderated Online Testing

    Ran unmoderated online tests to investigate the linguistic and social consequences of using AI to communicate and make design recommendations

How did we do it?

  1. 01

    Generative In-lab Tests & Interviews

    Conducted 36 in-lab tests where pairs of users had messaging conversations with and without AI mediation, followed by semi-structured interviews with 5 randomly chosen from the AI group. I transcribed and linguistically analyzed the conversations, as well as performed thematic analysis on the interview data.

  2. 02

    Moderated Online Tests for Interpersonal Dynamics

    Ran a 2 (un-/successful conversation) x 2 (no/AI) between-subjects moderated online test where the conversation outcome was controlled by a confederate (me), including a follow-up survey to measure resulting interpersonal perceptions (such as trust). In addition to performing statistical analysis on the main outcome measures, I also transcribed and linguistically analyzed the conversations.

  3. 03

    Unmoderated Online Tests for Social Outcomes

    Deployed our experimental AI communication platform on MTurk in unmoderated tests where 219 pairs of users had messaging conversations with or without AI mediation (we used Google’s Reply). A corresponding survey was used to measure social outcomes (such as dominance and cooperation) and match conversations with outcomes. I also linguistically analyzed the conversations.

  4. 04

    Unmoderated Online Tests for Language Changes

    Deployed our experimental AI communication platform on MTurk in unmoderated tests where 291 pairs of users had messaging conversations with: (1) commercial AI mediation (Reply), (2) positive sentiment AI mediation, (3) negative sentiment AI mediation, or (4) without AI. I then linguistically analyzed the conversations.

How did we design our experimental messenger?

Using pre-existing messaging apps in our initial studies meant that we couldn’t control the AI, control the UI, or collect complete conversation data.

To overcome these limitations, I collaborated with computer scientists to design an open source AI messaging platform, allowing researchers:

  • Control over the UI
  • Control over the AI (developing models, using existing APIs, pulling from a supplied list)
  • Ability to run simultaneous study sessions
  • Automatic data collection
Messaging interface screen from our experimental AI-mediated messaging app

Key Insights & Design Opportunities

  1. 01

    AI is more positive than you

    Commercial AI mediation expresses more positive sentiment than corresponding conversation, and users suspect that its presence is changing the way they communicate.

  2. 02

    AI can be a scapegoat

    AI in communication leads to increased trust between users, and when conversations go awry, users designate some responsibility to the AI. This suggests that AI mediation could be used to improve communication dynamics & outcomes.

  3. 03

    Users don't like thinking that you might be using AI to communicate

    If people think you’re using AI to communicate, they think of you more negatively, but when you actually use AI to communicate, they think better of you. Further, the linguistic characteristics of the AI mediation has a direct effect on users’ language.

  4. 04

    Targeted AI can be used to improve communication outcomes

    We can use targeted AI mediation to improve communication dynamics and outcomes, such as enhancing relational focus by increasing first-person plural pronoun usage.

Potential Design Applications

Algorithms could be used to detect when communication is going awry, and AI mediation could step in to suggest conflict resolution strategies.

In instances where a person has difficulties communicating, AI mediation could facilitate connection by suggesting appropriate things to say.

An email on a phone where the sender is being rude, so AI suggests that the user might want to see conflict resolution strategies above the reply options
A messaging interface on a phone where the communicator isn't sure how to respond, so the AI suggests some topics of conversation