AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation
CoRR(2024)
摘要
Web automation is a significant technique that accomplishes complicated web
tasks by automating common web actions, enhancing operational efficiency, and
reducing the need for manual intervention. Traditional methods, such as
wrappers, suffer from limited adaptability and scalability when faced with a
new website. On the other hand, generative agents empowered by large language
models (LLMs) exhibit poor performance and reusability in open-world scenarios.
In this work, we introduce a crawler generation task for vertical information
web pages and the paradigm of combining LLMs with crawlers, which helps
crawlers handle diverse and changing web environments more efficiently. We
propose AutoCrawler, a two-stage framework that leverages the hierarchical
structure of HTML for progressive understanding. Through top-down and step-back
operations, AutoCrawler can learn from erroneous actions and continuously prune
HTML for better action generation. We conduct comprehensive experiments with
multiple LLMs and demonstrate the effectiveness of our framework. Resources of
this paper can be found at
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要