An improved analysis of the ER-SpUD dictionary learning algorithm.

In dictionary learning we observe Y = AX + E for some Y in R^{n*p}, A in R^{m*n}, and X in R^{m*p}, where p u003e= max{n, m}, and typically m u003e=n. The matrix Y is observed, and A, X, E are unknown. Here E is a noise matrix of small norm, and X is column-wise sparse. The matrix A is referred to as a dictionary, and its columns as atoms. Then, given some small number p of samples, i.e. columns of Y , the goal is to learn the dictionary A up to small error, as well as the coefficient matrix X. In applications one could for example think of each column of Y as a distinct image in a database. The motivation is that in many applications data is expected to sparse when represented by atoms in the right dictionary A (e.g. images in the Haar wavelet basis), and the goal is to learn A from the data to then use it for other applications.Recently, the work of [Spielman/Wang/Wright, COLTu002712] proposed the dictionary learning algorithm ER-SpUD with provable guarantees when E = 0 and m = n. That work showed that if X has independent entries with an expected Theta n non-zeroes per column for 1/n ~ n^2 log^2 n with high probability ER-SpUD outputs matrices Au0027, Xu0027 which equal A, X up to permuting and scaling columns (resp. rows) of A (resp. X). They conjectured that p u003e~ n log n suffices, which they showed was information theoretically necessary for any algorithm to succeed when Theta =~ 1/n. Significant progress toward showing that p u003e~ n log^4 n might suffice was later obtained in [Luh/Vu, FOCSu002715].In this work, we show that for a slight variant of ER-SpUD, p u003e~ n log(n/delta) samples suffice for successful recovery with probability 1 - delta. We also show that without our slight variation made to ER-SpUD, p u003e~ n^{1.99} samples are required even to learn A, X with a small success probability of 1/ poly(n). This resolves the main conjecture of [Spielman/Wang/Wright, COLTu002712], and contradicts a result of [Luh/Vu, FOCSu002715], which claimed that p u003e~ n log^4 n guarantees high probability of success for the original ER-SpUD algorithm.