Relevance Estimation with Multiple Information Sources on Search Engine Result Pages.

CIKM, pp.627-636, (2018)

Cited by: 10|Views156
EI

Abstract

Relevance estimation is among the most important tasks in the ranking of search results because most search engines follow the Probability Ranking Principle. Current relevance estimation methodologies mainly concentrate on text matching between the query and Web documents, link analysis and user behavior models. However, users judge the r...More

Code:

Data:

0
Introduction
  • With the explosive growth of the Web, search engines play an ever more crucial role in information retrieval in the daily lives.
  • Queries searching for famous people or places always get image verticals, which consist of several images directly showing the person or the place.
  • From these examples, the authors can see that users can directly judge the relevance from the visual pattern, title, snippet and presentation structure of a search result.
  • It is essential to incorporate these information sources into the ranking process
Highlights
  • With the explosive growth of the Web, search engines play an ever more crucial role in information retrieval in our daily lives
  • Inspired by the recent progresses in computer vision and nature language processing tasks, we propose a novel framework named Joint Relevance Estimation model (JRE), which is composed of four subnets: Visual Pattern Learning Network (VPN), Title Semantics Learning Network (TSN), Snippet Semantics Learning Network (SSN), and HTML Tree Structure Learning Network(HSN)
  • We investigate whether visual patterns, textual semantics, and presentation structures help in the estimation of search result relevance
  • We propose the JRE framework, which incorporates visual patterns, textual semantics and presentation structures of search results into the relevance estimation process
  • To train Learning to rank algorithms (LTR) methods, 19 different statistical features are extracted from visual modality, textual modality and HTML modality (HTML source codes)
  • As heterogeneous verticals account for more and more in search results, exploring their contents becomes vital in relevance estimation
Results
  • There have been no multimodal neural models utilizing SERP contents to estimate relevance.
  • The authors adapt state-of-the-art LTR and neural text matching methods for SERP contents as two types of baselines.
  • To train LTR methods, 19 different statistical features are extracted from visual modality, textual modality and HTML modality (HTML source codes).
  • The textual features are designed following the methodology used by Microsoft LETOR data [32].
  • The two HTML features are designed to reflect the structure complexity of search results
Conclusion
  • 7.1 The Effectiveness of Attention Mechanism

    7.1.1 Inter-Modality Attention Mechanism.
  • The automatically learned inter-modality attention can better incorporate different information sources, which jointly achieve the best performance.
  • The ability of each information source in relevance estimation can be reflected by the respective performance of VPN, TSN, SSN and HSN to a certain extent.
  • Inter-modality and intra-modality attention mechanisms are introduced to better utilize information from different sources.
  • The proposed JRE model achieves the best performance among all the approaches, and significantly improves the original ranking of the search engine by 8.60%, 6.42%, and 2.54% in terms of NDCG@3,5,10 respectively.
  • A more effective fusion method of different information sources can be proposed
Tables
  • Table1: The descriptions of 19 result types
  • Table2: Performance of different query guided attention windows
  • Table3: Performance of different search task representations
  • Table4: Experimental results of different models. ∗, + and △
  • Table5: Performance when discarding any subnet of JRE. HHoowwttooIInnssttaallllPPlluuggiinnssiinnFFiirreeffooxx-­EExxppeerriieennccee
  • Table6: Weights of each subnet in different frameworks. WhoarestillusingFirefoxbrowser? Why did you abandon Chrome and choose Firefox ?
Download tables as Excel
Related work
  • 2.1 Search Result Ranking

    The ranking process of commercial search engines can be divided into two steps as shown in the upper right part of Figure 1. First, they design a large number of ranking features as indicators for relevance. A lot of works have been proposed to extract different types of features given the query and a large corpus of URLs [45]. Direct text matching methods compute matching scores of queries and Web documents according to their term frequencies, such as BM25 [34] and vector space model [28]. Link analysis (e.g. PageRank [30]) employs link relations as the proxy of Web page importance based on the Web graph. Click models such as UBM [10], DCM [15], DBN [3] and PSCM [40], exploit user behavior based on experimental hypotheses. Besides, document statistics (e.g. the number of words in various fields), document classifier (e.g. navigational destination vs informational), query features (e.g. click-through rate of the query), topical matching (e.g. topic level similarity), timeliness features (e.g. freshness of a Web page) and spatial features (e.g. location information) are also widely used [45].
Funding
  • This work is supported by Natural Science Foundation of China (Grant No 61622208, 61732008, 61472206) and National Key Basic Research Program (2015CB358700)
Reference
  • Javad Azimi, Ruofei Zhang, Zhou Yang, Vidhya Navalpakkam, Jianchang Mao, and Xiaoli Fern. 2012. The impact of visual appearance on user response in online display advertising. 457–458.
    Google ScholarFindings
  • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. Computer Science (2014).
    Google ScholarLocate open access versionFindings
  • Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th international conference on World wide web. ACM, 1–10.
    Google ScholarLocate open access versionFindings
  • Danqi Chen, Weizhu Chen, Haixun Wang, Zheng Chen, and Qiang Yang. 2012. Beyond ten blue links: enabling user click modeling in federated web search. In Proceedings of the fifth ACM international conference on Web search and data mining. ACM, 463–472.
    Google ScholarLocate open access versionFindings
  • Kan Chen, Trung Bui, Chen Fang, Zhaowen Wang, and Ram Nevatia. 2017. AMC: Attention guided multi-modal correlation learning for image search. arXiv preprint arXiv:1704.00763 (2017).
    Findings
  • Haibin Cheng, Roelof Van Zwol, Javad Azimi, Eren Manavoglu, Ruofei Zhang, Yang Zhou, and Vidhya Navalpakkam. 2012. Multimedia features for click prediction of new ads in display advertising. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 777–785.
    Google ScholarLocate open access versionFindings
  • Dan C Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, and Jürgen Schmidhuber. 2011. Flexible, high performance convolutional neural networks for image classification. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence, Vol. 22.
    Google ScholarLocate open access versionFindings
  • Jacob Cohen. 196Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213.
    Google ScholarLocate open access versionFindings
  • Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 200Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 248–255.
    Google ScholarLocate open access versionFindings
  • Georges E Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations.. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 331–338.
    Google ScholarLocate open access versionFindings
  • Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2015. The pascal visual object classes challenge: A retrospective. International journal of computer vision 111, 1 (2015), 98–136.
    Google ScholarLocate open access versionFindings
  • Yixing Fan, Jiafeng Guo, Yanyan Lan, Jun Xu, Liang Pang, and Xueqi Cheng. 2017. Learning Visual Features from Snapshots for Web Search. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 247–256.
    Google ScholarLocate open access versionFindings
  • Yoav Freund, Raj D. Iyer, Robert E. Schapire, and Yoram Singer. 1998. An Efficient Boosting Algorithm for Combining Preferences. In Fifteenth International Conference on Machine Learning. 170–178.
    Google ScholarLocate open access versionFindings
  • Jerome H. Friedman. 2001. Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics 29, 5 (2001), 1189–1232.
    Google ScholarLocate open access versionFindings
  • Fan Guo, Chao Liu, and Yi Min Wang. 2009. Efficient multiple-click models in web search. In Proceedings of the Second ACM International Conference on Web Search and Data Mining. ACM, 124–131.
    Google ScholarLocate open access versionFindings
  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 20Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
    Google ScholarLocate open access versionFindings
  • Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
    Google ScholarLocate open access versionFindings
  • Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In International Conference on Neural Information Processing Systems. 2042–2050.
    Google ScholarLocate open access versionFindings
  • Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In ACM Conference on Knowledge Discovery and Data Mining. 133–142.
    Google ScholarLocate open access versionFindings
  • D Kinga and J Ba Adam. 2015. A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
    Google ScholarLocate open access versionFindings
  • Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123, 1 (2017), 32–73.
    Google ScholarLocate open access versionFindings
  • Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.
    Google ScholarFindings
  • Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.
    Google ScholarLocate open access versionFindings
  • Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2016. Knowing when to look: Adaptive attention via A visual sentinel for image captioning. arXiv preprint arXiv:1612.01887 (2016).
    Findings
  • Cheng Luo, Yiqun Liu, Tetsuya Sakai, Fan Zhang, Min Zhang, and Shaoping Ma. 2017. Evaluating Mobile Search with Height-Biased Gain. (2017).
    Google ScholarFindings
  • Minh Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. Computer Science (2015).
    Google ScholarLocate open access versionFindings
  • Thomas Mandl. 2006. Implementation and evaluation of a quality-based search engine. In Proceedings of the seventeenth conference on Hypertext and hypermedia. ACM, 73–84.
    Google ScholarLocate open access versionFindings
  • SchÃijtze Manning Raghavan. 2008. Introduction to Information Retrieval. Journal of the American Society for Information Science & Technology 43, 3 (2008), 824–825.
    Google ScholarLocate open access versionFindings
  • Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems 26 (2013), 3111–3119.
    Google ScholarLocate open access versionFindings
  • L Page. 1999. The PageRank Citation Ranking: Bringing Order to the Web. Stanford Digital Libraries Working Paper 9, 1 (1999), 1–14.
    Google ScholarLocate open access versionFindings
  • Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text Matching as Image Recognition.. In AAAI. 2793–2799.
    Google ScholarFindings
  • Tao Qin, Tie Yan Liu, Jun Xu, and Hang Li. 2010. LETOR: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval 13, 4 (2010), 346–374.
    Google ScholarLocate open access versionFindings
  • Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2017. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence 39, 6 (2017), 1137–1149.
    Google ScholarLocate open access versionFindings
  • Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations & TrendsÂő in Information Retrieval 3, 4 (2009), 333–389.
    Google ScholarLocate open access versionFindings
  • Kalervo Rvelin, Kek, and Jaana Inen. 2002. Cumulated gain-based evaluation of IR techniques. Acm Transactions on Information Systems 20, 4 (2002), 422–446.
    Google ScholarLocate open access versionFindings
  • Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
    Findings
  • Ruihua Song, Haifeng Liu, Ji-Rong Wen, and Wei-Ying Ma. 2004. Learning block importance models for web pages. In Proceedings of the 13th international conference on World Wide Web. ACM, 203–211.
    Google ScholarLocate open access versionFindings
  • Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1–9.
    Google ScholarLocate open access versionFindings
  • Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2015. A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations. (2015), 2835–2841.
    Google ScholarFindings
  • Chao Wang, Yiqun Liu, Meng Wang, Ke Zhou, Jian-yun Nie, and Shaoping Ma. 2015. Incorporating non-sequential behavior into click models. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 283–292.
    Google ScholarLocate open access versionFindings
  • Chao Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013. Incorporating vertical results into search click models. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. ACM, 503–512.
    Google ScholarLocate open access versionFindings
  • Qiang Wu, Christopher J. C. Burges, Krysta M. Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval 13, 3 (2010), 254–270.
    Google ScholarLocate open access versionFindings
  • Jun Xu and Hang Li. 2007. AdaRank:a boosting algorithm for information retrieval. 391–398.
    Google ScholarFindings
  • Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Computer Science (2015), 2048–2057.
    Google ScholarLocate open access versionFindings
  • Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, and Chikashi Nobata. 2016. Ranking Relevance in Yahoo Search. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 323–332.
    Google ScholarLocate open access versionFindings
  • Masrour Zoghi, Tomáš Tunys, Lihong Li, Damien Jose, Junyan Chen, Chun Ming Chin, and Maarten de Rijke. 2016. Click-based hot fixes for underperforming torso queries. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 195–204.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科