Optimal lower bounds for universal relation, samplers, and finding duplicates

In the communication problem 𝐔𝐑 (universal relation) [KRW95], Alice and Bob respectively receive x and y in {0,1}^n with the promise that x≠ y. The last player to receive a message must output an index i such that x_i≠ y_i. We prove that the randomized one-way communication complexity of this problem in the public coin model is exactly Θ(min{n, log(1/δ)log^2(n/log(1/δ))}) bits for failure probability δ. Our lower bound holds even if promised support(y)⊂support(x). As a corollary, we obtain optimal lower bounds for ℓ_p-sampling in strict turnstile streams for 0≤ p < 2, as well as for the problem of finding duplicates in a stream. Our lower bounds do not need to use large weights, and hold even if it is promised that x∈{0,1}^n at all points in the stream. Our lower bound demonstrates that any algorithm 𝒜 solving sampling problems in turnstile streams in low memory can be used to encode subsets of [n] of certain sizes into a number of bits below the information theoretic minimum. Our encoder makes adaptive queries to 𝒜 throughout its execution, but done carefully so as to not violate correctness. This is accomplished by injecting random noise into the encoder's interactions with 𝒜, which is loosely motivated by techniques in differential privacy. Our correctness analysis involves understanding the ability of 𝒜 to correctly answer adaptive queries which have positive but bounded mutual information with 𝒜's internal randomness, and may be of independent interest in the newly emerging area of adaptive data analysis with a theoretical computer science lens.