demuxmix: demultiplexing oligonucleotide-barcoded single-cell RNA sequencing data with regression mixture models

Motivation: Droplet-based single-cell RNA sequencing (scRNA-seq) is widely used in biomedical research for interrogating the transcriptomes of single cells on a large scale. Pooling and processing cells from different samples together can reduce costs and batch effects. To pool cells, they are often first labeled with hashtag oligonucleotides (HTOs). These HTOs are sequenced alongside the cells' RNA in the droplets and subsequently used to computationally assign each droplet to its sample of origin, a process referred to as demultiplexing. Accurate demultiplexing is crucial but can be challenging due to background HTOs, low-quality cells/cell debris, and multiplets.Results: A new demultiplexing method based on negative binomial regression mixture models is introduced. The method, called demuxmix, implements two significant improvements. First, demuxmix's probabilistic classification framework provides error probabilities for droplet assignments that can be used to discard uncertain droplets and inform about the quality of the HTO data and the success of the demultiplexing process. Second, demuxmix utilizes the positive association between detected genes in the RNA library and HTO counts to explain parts of the variance in the HTO data resulting in improved droplet assignments. The improved performance of demuxmix compared with existing demultiplexing methods is assessed using real and simulated data. Finally, the feasibility of accurately demultiplexing experimental designs where non-labeled cells are pooled with labeled cells is demonstrated.