Distilling Informative Content from HTML News Pages

Not only the Web abounds of information overload, but also its component molecules, the Web documents contained therein. In particular HTML news pages have become aggregates of cornucopian information blocks, such as advertisements, link lists, disclaimers and terms of use, or comments from readers. Thus, only a small fraction of all textual content appears dedicated to the actual news article itself. The amalgamation of relevant content with page clutter poses considerable concerns to applications that make use of such news information, such as search engines. We present an approach geared towards dissecting relevant from irrelevant textual content in an automated fashion. Our system extracts linguistic and structural features from merged text segments and applies various classifiers thereafter. We have conducted empirical analyses in order to compare our approach's classification performance with a human gold standard as well as two benchmark systems.