Automatic Evaluation of Search Ontologies in the Entertainment Domain using Text Classication

Information Retrieval (IR) research has recently started ad- dressing the information need of exploratory search. where the searcher may be unfamiliar with the domain or not have decided what is the goal of his query. A popular tool to support exploratory search is the use of faceted search. The implementation of faceted search requires that documents be annotated by metadata in the form of attributes and hi- erarchical categories. In many applications, the metadata is maintained manually, in the form of a search ontology. Recent work has also inves- tigated methods to automatically acquire such metadata from sample documents (1, 2). In this work, we propose a new method to automati- cally evaluate the quality of such a search ontology. Our method relies on mapping ontology instances to textual documents. On the basis of this mapping, we evaluate the adequacy of ontology re- lations by measuring their classication potential over the textual docu- ments. This data-driven method provides concrete feedback to ontology maintainers and a quantitative estimation of the functional adequacy of the ontology relations towards search experience improvement. We specically evaluate whether an ontology relation can help the search engine support exploratory search in the form of eective facets. We test this ontology evaluation method on an ontology in the Movies do- main, that has been acquired automatically from the integration of multi- ple semi-structured and textual data sources (e.g., IMDb and Wikipedia). We automatically construct a domain corpus from a set of movie in- stances by crawling the Web for movie reviews (both professional and user reviews). The 1-1 relation between textual documents (reviews) and movie instances in the ontology enables us to translate ontology relations into text classes. We verify that the text classiers induced by key on- tology relations (genre, keywords, actors) achieve high performance and exploit the properties of the learned text classiers to provide concrete feedback on the ontology. The proposed ontology evaluation method is general: it only relies on the possibility to automatically align textual documents to ontology in- stances.