Ranking Factors by Importance in Factorial Survey Analysis.

Factorial survey analysis is a statistical technique with a long history of use in decision-oriented organizational and information systems (IS) research. The technique produces a collection of standardized regression coefficients that help one to rank survey factors by importance. However, such rankings may be invalid because a researcher might not account for two related issues: unequal factor (i.e., dimension) manipulation effect sizes and the inherent multilevel structure of factorial survey data. We address these concomitant issues by demonstrating the ranking problem in simulated datasets, explaining the ranking problem's underlying statistical causes, and justifying the use of remediating statistical methods. In particular, we focus on coding proportional to effect, a technique in which one consolidates corresponding dimension-level dummy (0, 1) variables into a single re-calibrated independent variable that is regressed on the dependent variable. One then uses the resulting standardized coefficients to rank the factors. We assess the advantages, disadvantages, and limitations of remediation techniques and offer suggestions for future information systems research.