Robots in Groups Lab · HUMAN-AI TEAMS

How might we design AI to improve team collaboration?

Through our work examining how to design human–AI collaboration systems, we realized that this research is inaccessible to most given the considerable time and technical skills necessary to develop and deploy AI. To make this type of research more accessible, we used an iterative design process to develop an experimental method and performed a case study demonstrating its flexibility and ease of deployment. We then used the method to investigate how to design AI to improve team collaboration.

BY THE NUMBERS

  • 168users in the case study of the final research method
  • 92%believed they had interacted with an AI agent
  • 10progressive moderated sessions to design the AI teammate
DATE
Fall 2020 - Winter 2021
ORGANIZATION
Robots in Groups Lab
ROLE
ResearchDesignProject ManagementStrategy
DELIVERABLES
TEAM
7 Information Scientists3 Psychologists1 Network Scientist1 Management Scientists1 Neuroscientist
PUBLICATIONS
Computers in Human Behavior (CHB)Hawaii International Conference on System Sciences (HICSS)

Design Research Process

  1. 01

    Iterative Design of Experimental Method

    Human-AI collaboration research was largely inaccessible given the considerable time and technical skills necessary to develop and deploy AI. We used an iterative design approach to develop a novel experimental method to make this type of research more accessible, eliminating the need to develop AI by employing a human confederate to pose as an AI agent.

  2. 02

    Case Study

    In a case study of the final research method with 168 users, 92% believed that they had interacted with an AI agent. Further, believability was not affected by pre-existing beliefs about robots or technology, suggesting that the paradigm is believable amongst technologically diverse participant groups.

  3. 03

    Iterative Remote Moderated Sessions

    We employed our new research method in an iterative design process through 10 progressive moderated sessions with small groups of users to investigate whether and how we can design an effective AI teammate.

A Zoom screen showing anonymized users working with the AI agent, Vero, during a study
Anonymized users interacting with Vero, the AI agent, in a Zoom session.

How does our research method work?

We used an iterative design approach to develop a method that allows us to simulate an AI teammate (named "Vero") through a setup that is accessible to everyone, with or without technical skills.

  • 1| Standardized AI Introduction Video

    The introduction video should:

    • Establish the agent as state-of-the-art AI
    • Provide an example of similar commercially-existing technology
    • Explain that the AI agent is classified and their name has been changed
    • Show the agent’s different interaction abilities, potential voice patterns, and accents
  • 2| Consistent Language

    The AI agent should be consistently referred to with identical language.

    Similarly, unless gender effects are being studied, the agent should be referred to using gender neutral or non-personifying language.

  • 3| Interaction Animation Indicators

    If video analysis will be used, add an easily-recognizable icon (such as a changing WiFi indicator) to differentiate each animation.

  • 4| Validate Confederates' Tech

    Before each session, ensure that each confederate has:

    • Consistent camera cover
    • Elimination of background noise
    • Consistent internet and clear audio
    • Changed Zoom name and photo to that of the agent
    • Correct default agent state
  • 5| Zoom Breakout Rooms

    By assigning each confederate and paired team to a different Zoom Breakout Room, researchers can run multiple parallel study sessions.

  • 6| Preparation for Negative Interaction

    Prepare confederates for the possibility that participants could treat them differently than their “human” teammates by providing prepared responses to any rude or negative interactions.

Through a combination of animation designs, OBS, Python, and Zoom, confederates can appear as different animated versions of the AI agent by communicating using specific verbal and nonverbal gestures.

Table describing how to activate each Vero animation, eg, if you want to speak, raise your hand  with an open palm facing the camera, and Vero will repeatedly grow and shrink.
A gif of Vero idling (floating up and down slowly)
Vero idling
A gif of Vero speaking (horizontally vibrating slightly)
Vero speaking

What did we find?

  • AI isn't a teammate

    Users do not yet consider the AI agent to be a member of the team because they were too inactive and didn't contribute enough. Some users expressed discomfort about AI becoming a teammate, instead preferring that AI maintains a supporting role only. This suggests that designers need to be aware of situations where AI shouldn't be a member of the team.

  • Discomfort around AI transparency

    Many users felt that they didn't have enough information about the AI, so it would be beneficial to provide obvious access to in-depth information about the AI model, like its capabilities and training.

  • AI should offer context-dependent capabilities that humans don't have

    Users thought that the AI should have task-specific capabilities that humans don't, like quickly providing information or encouraging teamwork. Designers should take user expectations into mind when designing AI agents for specific teamwork tasks.

  • AI shouldn't have unique interaction modalities

    Some of our AI agent design iterations included things like full-screen reactions and sound reactions (like a laugh track), but users do not want AI to have access to different interaction modalities than those afforded to humans. Humans can't react with a laugh track, so the AI shouldn't be able to either.