Collaborative Design Decision-Making With Artificial Intelligence: Exploring the Evolution and Impact of Human Confidence in AI and in Themselves

Abstract Decision-making assistance by artificial intelligence (AI) during design is only effective when human designers properly utilize AI input. However, designers often misjudge the AI’s and/or their own ability, leading to erroneous reliance on AI and therefore bad designs. To avoid such outcomes, it is critical to understand the evolution of designers’ confidence in both their AI teammate(s) and themselves during human-AI collaboration. Therefore, this work conducts a cognitive study to explore how experiencing various and changing (without notice) AI performance levels and feedback affects these confidences and consequently the decisions to accept or reject AI suggestions. The results first reveal that designers’ confidence in an AI agent changes with poor, but not with good, AI performance. Interestingly, designers’ self-confidence initially remains unaffected by AI accuracy; however, when the accuracy changes, self-confidence decreases regardless of the direction of the change. Moreover, this work finds that designers tend to infer flawed information from feedback, resulting in inappropriate levels of confidence in both the AI and themselves. Confidence in the AI and in themselves is also shown to affect designers’ probability of accepting AI input in opposite directions. Finally, results that are uniquely applicable to design are identified by comparing the findings from this work to those from a similar study conducted with a non-design task. Overall, this work offers valuable insights that may enable prevention and detection of designers’ inappropriate confidence and their consequent misuse of AI in design.