Large relative margin and applications

Over the last decade or so, machine learning algorithms such as support vector machines, boosting etc. have become extremely popular. The core idea in these and other related algorithms is the notion of large margin. Simply put, the idea is to geometrically separate two classes with a large separation between them; such a separator is then used to predict the class of unseen test examples. These methods have been extremely successful in practice and have formed a significant portion of machine learning literature. There are several theoretical results which motivate such algorithms. A closer look at such theoretical results reveals that the generalization ability of these methods are strongly linked to the margin as well as some measure of the spread of the data. Yet the algorithms themselves only seem to be maximizing the margin—completely ignoring the spread information. This thesis focuses on addressing this problem: novel formulations, that not only take into consideration the margin but also the spread aspect of the data, are proposed. In particular, relative margin machine, which is a strict generalization of the well known support vector machine is proposed. Further, generalization bounds are derived for the relative margin machines using a novel method of landmark examples. The idea of relative margin is fairly general; its potential is demonstrated by proposing formulations for structured prediction problems as well as for a transductive setup using graph Laplacian. Finally, a boosting algorithm incorporating both the margin information and the spread information is derived as well. The boosting algorithm is motivated from the recent empirical Bernstein bounds. All the proposed variants of the relative margin algorithms are easy to implement, efficiently solvable and typically show significant improvements over their large margin counterparts—on real-world datasets.