Machine Learning Theory Lecture 23 : Boosting

This condition involves three universal quantifiers, so may be quite difficult to satisfy! Interestingly, the ∀D quantifier turns out to be the most important one. We have shown in Assignment 2, Question 4, that the ∀δ quantifier can be satisfied for free: as long as Definition 1.1 holds with δ fixed at 1/2, it is possible to “amplify” the probability of success using validation. The ∀ quantifier can (in some sense) be satisfied “for free”, using the technique of Boosting. That is, if Definition 1.1 holds for bounded away from 1/2, then Boosting shows that a “majority vote” of several classifiers from H can achieve arbitrarily small . However H is not necessarily closed under taking majority votes, so the hypothesis produced by Boosting might not belong to H.