Teaching agents to understand teamwork: Evaluating and predicting collective intelligence as a latent variable via Hidden Markov Models

Rapid growth in the reliance on teamwork in organizations, coupled with advances in artificial intelligence, has fueled increased use of Human Autonomy Teams (HATs) involving the collaboration of humans and agents to complete work. Although there are many successful examples of HATs, researchers and technology developers can see additional applications if agents were better able to understand the mental states of humans to anticipate what a team is likely to do next. Creating this capability requires the creation of models of team interaction that enable agents to interpret a team’s current state and anticipate its future state. To build this model, we draw on research on collective intelligence (CI), which shows a team’s capability to work together can be characterized by a latent collective intelligence factor, based on observations of work across a range of tasks, and which predicts a team’s ability to accomplish a wide range of goals in the future. While some work uses a specific battery of CI tasks, more recent studies have identified observable collaborative process metrics that can be captured passively. Building on this work, we propose a method of evaluating CI by representing it as a latent variable represented by the hidden state in a Hidden Markov Model. The observations used as input to the model are the team’s observable collaborative process behaviors (i.e., collective effort, use of task-related skills, and task-strategy efficiency). We show by learning the set of hidden states representing a team’s observed collaborative process behaviors over time, we both learn information about the team’s CI, predict how CI will evolve in the future, and suggest when an agent might intervene to improve team performance. Based on the model’s observations, we discuss how it can help agents diagnose teamwork and possibly make interventions to improve CI by identifying areas of collaborative process (collective effort, skill use, or task strategy) that could be improved.