Designing a Regional Crawler for Distributed and Centralized Search Engines
msra(2004)
摘要
Today, by the growth of WWW, the significance and popularity of search engines are increasing day by day. However, today web crawlers are unable to update their huge search engine indexes concurrent to the growth in the information available on the web. Most of times they download some unimportant pages and ignore the pages that their probability of being searched is noticeable. This sometimes causes users to be unable to search in updated information. Regional Crawler that we introduce as new idea in this paper, improves the problem of updating and finding new pages to some extent by gathering users’ common needs and interests in a certain domain, which can be as small as a LAN in a department of a university or as huge as a country. In this paper, we design the Regional Crawler architecture and introduce its application in centralized and distributed search engines Key-Words: Regional Crawler, Web Crawler Architecture, Multi-agent Systems
更多查看译文
关键词
regional crawler,multi-agent systems,key-words:,web crawler architecture
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络