In humans, we trust

Algorithms have greatly advanced and become integrated into our everyday lives. Although they support humans in daily functions, they often exhibit unwanted behaviors perpetuating social stereotypes, discrimination, and other forms of biases. Regardless of their accuracy on task, many algorithms do not get scrutinized for unintended behaviors in a systematic way. This phenomenon can propagate and amplify existing societal issues or even create new ones. Many have called for human supervision (human oversight) of algorithmic processes. Oversight is often presented as a way of monitoring algorithmic behavior, as to then address identified issues, by initiating a fix or even correcting the final decision. Unfortunately, a common consensus is missing in the scientific community as to what all human oversight entails. Most importantly, the requirements for a successful application of a human oversight process are only vaguely defined. To address this, we present a critical synthesis of five key articles from different domains, which discuss requirements for human oversight. We use the concept of the Society-in-the-Loop (SITL) [ 1 ] as the baseline for understanding and mapping these requirements. In addition, we comment on the requirements and the overall multidisciplinary trend around the topic. Then, we present the concept of a Modular Oversight Methodology (MOM) following the SITL viewpoint, by also considering the requirements identified from the selected literature. Finally, we present a set of suggestions and future work required for a successful application of a human oversight process in a SITL approach.