Understanding users' satisfaction for search engine evaluation.

To fulfill users' search needs, the search engine must have good performance, easy-to-use functionalities, and good search result quality. Search quality evaluation becomes challenging when users' satisfaction may not be able to judge by a single search and even within a single search judgments from various sources are not consistent. In this talk, I will discuss how user's satisfaction is decomposed into different components in general, and how we measure them with various means - human judgment, automatic computation with query log, and outsourcing, and their pros and cons with operational implications. For an outlook, I will postulate potential evaluation approaches for a better user's satisfaction.