The ℓ_p-Subspace Sketch Problem in Small Dimensions with Applications to Support Vector Machines

In the ℓp-subspace sketch problem, we are given an n × d matrix A with n > d, and asked to build a small memory data structure Q(A,ε) so that, for any query vector x ∈ ℝd, we can output a number in given only Q(A,ε). This problem is known to require bits of memory for d = Ω(log (1/ε)). However, for d = o(log(l/ε)), no data structure lower bounds were known. Small constant values of d are particularly important for estimating point queries for support vector machines (SVMs) in a stream (Andoni et al. 2020), where only tight bounds for d = 1 were known.We resolve the memory required to solve the ℓp-subspace sketch problem for any constant d and integer p, showing that it is bits and words, where the Õ(·) notation hides poly(log(1/ε)) factors. This shows that one can beat the Ω(ε-2) lower bound, which holds for d = Ω(log(1/ε)), for any constant d. Further, we show how to implement the upper bound in a single pass stream, with an additional multiplicative poly(log log n) factor and an additive poly(log n) cost in the memory. Our bounds extend to loss functions other than the ℓp-norm, and notably they apply to point queries for SVMs with additive error, where we show an optimal bound of for every constant d. This is a near-quadratic improvement over the lower bound of Andoni et al. Further, previous upper bounds for SVM point query were noticeably lacking: for d =1 the bound was Õ(e-1/2) and for d = 2 the bound was Õ(ε-4//5), but all existing techniques failed to give any upper bound better than Õ(ε-2) for any other value of d. Our techniques, which rely on a novel connection to low dimensional techniques from geometric functional analysis, completely close this gap.