Identifying Web Pages with Major Contents based on Search Engine Suggests and Topic Modeling

This paper addresses the problem of identifying irrelevant items from a small set of similar documents using Web search engine suggests. Specifically, we collected volumes of Web pages through Web search engines and inspected the page contents using topic models. Among each cluster of pages sharing the same topic indicated by the topic model, our technique discovers potential content organization in the current page cluster and identifies pages that are out of focus from that topic. The metrics in our approach mainly consist of search engine suggest frequency and inter-document similarity measures. The intuition is that Web pages collected via the same search queries are more likely to share similar contents. We verify this intuition by implementing a subtopic based document selection framework and making quantitative evaluation against human made labeled data sets. Our evaluation result reveals that suggest frequency analysis along with inter-document similarity measure is effective at filtering off-topic documents in small data sets with satisfactory performance.