Relevance Estimation with Multiple Information Sources on Search Engine Result Pages.
CIKM, pp.627-636, (2018)
Relevance estimation is among the most important tasks in the ranking of search results because most search engines follow the Probability Ranking Principle. Current relevance estimation methodologies mainly concentrate on text matching between the query and Web documents, link analysis and user behavior models. However, users judge the r...More
PPT (Upload PPT)
- With the explosive growth of the Web, search engines play an ever more crucial role in information retrieval in the daily lives.
- Queries searching for famous people or places always get image verticals, which consist of several images directly showing the person or the place.
- From these examples, the authors can see that users can directly judge the relevance from the visual pattern, title, snippet and presentation structure of a search result.
- It is essential to incorporate these information sources into the ranking process
- With the explosive growth of the Web, search engines play an ever more crucial role in information retrieval in our daily lives
- Inspired by the recent progresses in computer vision and nature language processing tasks, we propose a novel framework named Joint Relevance Estimation model (JRE), which is composed of four subnets: Visual Pattern Learning Network (VPN), Title Semantics Learning Network (TSN), Snippet Semantics Learning Network (SSN), and HTML Tree Structure Learning Network(HSN)
- We investigate whether visual patterns, textual semantics, and presentation structures help in the estimation of search result relevance
- We propose the JRE framework, which incorporates visual patterns, textual semantics and presentation structures of search results into the relevance estimation process
- To train Learning to rank algorithms (LTR) methods, 19 different statistical features are extracted from visual modality, textual modality and HTML modality (HTML source codes)
- As heterogeneous verticals account for more and more in search results, exploring their contents becomes vital in relevance estimation
- There have been no multimodal neural models utilizing SERP contents to estimate relevance.
- The authors adapt state-of-the-art LTR and neural text matching methods for SERP contents as two types of baselines.
- To train LTR methods, 19 different statistical features are extracted from visual modality, textual modality and HTML modality (HTML source codes).
- The textual features are designed following the methodology used by Microsoft LETOR data .
- The two HTML features are designed to reflect the structure complexity of search results
- 7.1 The Effectiveness of Attention Mechanism
7.1.1 Inter-Modality Attention Mechanism.
- The automatically learned inter-modality attention can better incorporate different information sources, which jointly achieve the best performance.
- The ability of each information source in relevance estimation can be reflected by the respective performance of VPN, TSN, SSN and HSN to a certain extent.
- Inter-modality and intra-modality attention mechanisms are introduced to better utilize information from different sources.
- The proposed JRE model achieves the best performance among all the approaches, and significantly improves the original ranking of the search engine by 8.60%, 6.42%, and 2.54% in terms of NDCG@3,5,10 respectively.
- A more effective fusion method of different information sources can be proposed
- Table1: The descriptions of 19 result types
- Table2: Performance of different query guided attention windows
- Table3: Performance of different search task representations
- Table4: Experimental results of different models. ∗, + and △
- Table5: Performance when discarding any subnet of JRE. HHoowwttooIInnssttaallllPPlluuggiinnssiinnFFiirreeffooxx-EExxppeerriieennccee
- Table6: Weights of each subnet in different frameworks. WhoarestillusingFirefoxbrowser? Why did you abandon Chrome and choose Firefox ?
- 2.1 Search Result Ranking
The ranking process of commercial search engines can be divided into two steps as shown in the upper right part of Figure 1. First, they design a large number of ranking features as indicators for relevance. A lot of works have been proposed to extract different types of features given the query and a large corpus of URLs . Direct text matching methods compute matching scores of queries and Web documents according to their term frequencies, such as BM25  and vector space model . Link analysis (e.g. PageRank ) employs link relations as the proxy of Web page importance based on the Web graph. Click models such as UBM , DCM , DBN  and PSCM , exploit user behavior based on experimental hypotheses. Besides, document statistics (e.g. the number of words in various fields), document classifier (e.g. navigational destination vs informational), query features (e.g. click-through rate of the query), topical matching (e.g. topic level similarity), timeliness features (e.g. freshness of a Web page) and spatial features (e.g. location information) are also widely used .
- This work is supported by Natural Science Foundation of China (Grant No 61622208, 61732008, 61472206) and National Key Basic Research Program (2015CB358700)
- Javad Azimi, Ruofei Zhang, Zhou Yang, Vidhya Navalpakkam, Jianchang Mao, and Xiaoli Fern. 2012. The impact of visual appearance on user response in online display advertising. 457–458.
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. Computer Science (2014).
- Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th international conference on World wide web. ACM, 1–10.
- Danqi Chen, Weizhu Chen, Haixun Wang, Zheng Chen, and Qiang Yang. 2012. Beyond ten blue links: enabling user click modeling in federated web search. In Proceedings of the fifth ACM international conference on Web search and data mining. ACM, 463–472.
- Kan Chen, Trung Bui, Chen Fang, Zhaowen Wang, and Ram Nevatia. 2017. AMC: Attention guided multi-modal correlation learning for image search. arXiv preprint arXiv:1704.00763 (2017).
- Haibin Cheng, Roelof Van Zwol, Javad Azimi, Eren Manavoglu, Ruofei Zhang, Yang Zhou, and Vidhya Navalpakkam. 2012. Multimedia features for click prediction of new ads in display advertising. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 777–785.
- Dan C Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, and Jürgen Schmidhuber. 2011. Flexible, high performance convolutional neural networks for image classification. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence, Vol. 22.
- Jacob Cohen. 196Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213.
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 200Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 248–255.
- Georges E Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations.. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 331–338.
- Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2015. The pascal visual object classes challenge: A retrospective. International journal of computer vision 111, 1 (2015), 98–136.
- Yixing Fan, Jiafeng Guo, Yanyan Lan, Jun Xu, Liang Pang, and Xueqi Cheng. 2017. Learning Visual Features from Snapshots for Web Search. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 247–256.
- Yoav Freund, Raj D. Iyer, Robert E. Schapire, and Yoram Singer. 1998. An Efficient Boosting Algorithm for Combining Preferences. In Fifteenth International Conference on Machine Learning. 170–178.
- Jerome H. Friedman. 2001. Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics 29, 5 (2001), 1189–1232.
- Fan Guo, Chao Liu, and Yi Min Wang. 2009. Efficient multiple-click models in web search. In Proceedings of the Second ACM International Conference on Web Search and Data Mining. ACM, 124–131.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 20Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
- Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In International Conference on Neural Information Processing Systems. 2042–2050.
- Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In ACM Conference on Knowledge Discovery and Data Mining. 133–142.
- D Kinga and J Ba Adam. 2015. A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
- Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123, 1 (2017), 32–73.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.
- Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2016. Knowing when to look: Adaptive attention via A visual sentinel for image captioning. arXiv preprint arXiv:1612.01887 (2016).
- Cheng Luo, Yiqun Liu, Tetsuya Sakai, Fan Zhang, Min Zhang, and Shaoping Ma. 2017. Evaluating Mobile Search with Height-Biased Gain. (2017).
- Minh Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. Computer Science (2015).
- Thomas Mandl. 2006. Implementation and evaluation of a quality-based search engine. In Proceedings of the seventeenth conference on Hypertext and hypermedia. ACM, 73–84.
- SchÃijtze Manning Raghavan. 2008. Introduction to Information Retrieval. Journal of the American Society for Information Science & Technology 43, 3 (2008), 824–825.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems 26 (2013), 3111–3119.
- L Page. 1999. The PageRank Citation Ranking: Bringing Order to the Web. Stanford Digital Libraries Working Paper 9, 1 (1999), 1–14.
- Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text Matching as Image Recognition.. In AAAI. 2793–2799.
- Tao Qin, Tie Yan Liu, Jun Xu, and Hang Li. 2010. LETOR: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval 13, 4 (2010), 346–374.
- Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2017. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence 39, 6 (2017), 1137–1149.
- Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations & TrendsÂő in Information Retrieval 3, 4 (2009), 333–389.
- Kalervo Rvelin, Kek, and Jaana Inen. 2002. Cumulated gain-based evaluation of IR techniques. Acm Transactions on Information Systems 20, 4 (2002), 422–446.
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
- Ruihua Song, Haifeng Liu, Ji-Rong Wen, and Wei-Ying Ma. 2004. Learning block importance models for web pages. In Proceedings of the 13th international conference on World Wide Web. ACM, 203–211.
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1–9.
- Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2015. A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations. (2015), 2835–2841.
- Chao Wang, Yiqun Liu, Meng Wang, Ke Zhou, Jian-yun Nie, and Shaoping Ma. 2015. Incorporating non-sequential behavior into click models. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 283–292.
- Chao Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013. Incorporating vertical results into search click models. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. ACM, 503–512.
- Qiang Wu, Christopher J. C. Burges, Krysta M. Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval 13, 3 (2010), 254–270.
- Jun Xu and Hang Li. 2007. AdaRank:a boosting algorithm for information retrieval. 391–398.
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Computer Science (2015), 2048–2057.
- Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, and Chikashi Nobata. 2016. Ranking Relevance in Yahoo Search. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 323–332.
- Masrour Zoghi, Tomáš Tunys, Lihong Li, Damien Jose, Junyan Chen, Chun Ming Chin, and Maarten de Rijke. 2016. Click-based hot fixes for underperforming torso queries. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 195–204.