An Improved Online Reduction from PAC Learning to Mistake-Bounded Learning.

A basic result in learning theory is that mistake-bounded learnability implies PAC learnability. It was shown in [Lit89] that, if a problem can be learned with M mistakes, it can be (ε, δ)-PAC-learned from samples. However, this reduction needs to store either samples or O(M) hypotheses. A different reduction, in [KLPV87], only needs to store O(1) samples and hypotheses but was only shown to work with samples.We give a refined analysis of the KLPV reduction, showing that it only uses samples with probability 1 - M-O(1). This gives the optimal sample complexity with only O(1) space overhead, for δ > M-O(1).