A better index for analysis of co-occurrence and similarity

Scientists often need to know whether pairs of entities tend to occur together or independently. Standard approaches to this issue use co-occurrence indices such as Jaccard, Sorensen-Dice, and Simpson. We show that these indices are sensitive to the prevalences of the entities they describe and that this invalidates their interpretability. We propose an index, alpha, that is insensitive to prevalences. Published datasets reanalyzed with both alpha and Jaccard's index (J) yield profoundly different biological inferences. For example, a published analysis using J contradicted predictions of the island biogeography theory finding that community stability increased with increasing physical isolation. Reanalysis of the same dataset with the estimator (alpha) over cap reversed that result and supported theoretical predictions. We found similarly marked effects in reanalyses of antibiotic cross-resistance and human disease bio-markers. Our index alpha is not merely an improvement; its use changes data interpretation in fundamental ways.