Natural Language Processing for Content Analysis of Communication in Collaborative Design

Abstract We address the problem of content analysis in text-based engineering design communication. Existing methods to characterize communication content in engineering design are manual or qualitative, which is tedious for large datasets. We formulate the characterization of communication messages as an intent classification task. We identify two intents — Intent 1 captures the presence and flow of information, Intent 2 captures specific topics about design parameters and objectives. We compare the predictive accuracy of convolutional LSTM, character-based convolutional LSTM, XLNet, and BERT models for the intent classification task. The results of our comparison show that the XL-Net model predicts Intents 1 and 2 with 88% and 81% accuracy, respectively, on text data collected from 40 teams in a design experiment with university students. We analyze the differences in communication patterns between high- and low-performing teams. Time-series studies show that high-performing teams have more responsive communication and a higher consistency of information exchange.