On the effect of items measuring different factors across individuals on item inter-correlations

Even when a single factor is to be measured, it may occur in the context of blind random item selection that individuals work on items that are not based on exactly the same factor. Therefore, we explore the consequences of presenting items from different populations measuring different factors across and within individuals. We found that item inter-correlations can be substantial in the total population of individuals even when - in subpopulations of individuals - items are drawn from populations based on different, even uncorrelated factors. In order to address this challenge for convergent validity, we propose a method that helps to detect whether the correlation between items is due to the same common factor measured by the items across all individuals. Based on the analytical results and results from a simulation study, we provide recommendations for the detection of subpopulations of individuals responding to items from different item populations.