The semiotics of the message and the messenger: How nonverbal communication affects fairness perception

Nonverbal communication determines much of how we perceive explicit, verbal messages. Facial expressions and social touch, for example, influence affinity and conformity. To understand the interaction between nonverbal and verbal information, we studied how the psychophysiological time-course of semiotics—the decoding of the meaning of a message—is altered by interpersonal touch and facial expressions. A virtual-reality-based economic decision-making game, ultimatum, was used to investigate how participants perceived, and responded to, financial offers of variable levels of fairness. In line with previous studies, unfair offers evoked medial frontal negativity (MFN) within the N2 time window, which has been interpreted as reflecting an emotional reaction to violated social norms. Contrary to this emotional interpretation of the MFN, however, nonverbal signals did not modulate the MFN component, only affecting fairness perception during the P3 component. This suggests that the nonverbal context affects the late, but not the early, stage of fairness perception. We discuss the implications of the semiotics of the message and the messenger as a process by which parallel information sources of “who says what” are integrated in reverse order: of the message, then the messenger.