Lemur: Log Parsing with Entropy Sampling and Chain-of-Thought Merging
CoRR(2024)
摘要
Logs produced by extensive software systems are integral to monitoring system
behaviors. Advanced log analysis facilitates the detection, alerting, and
diagnosis of system faults. Log parsing, which entails transforming raw log
messages into structured templates, constitutes a critical phase in the
automation of log analytics. Existing log parsers fail to identify the correct
templates due to reliance on human-made rules. Besides, These methods focus on
statistical features while ignoring semantic information in log messages. To
address these challenges, we introduce a cutting-edge Log parsing
framework with Entropy sampling and Chain-of-Thought Merging
(Lemur). Specifically, to discard the tedious manual rules. We propose a novel
sampling method inspired by information entropy, which efficiently clusters
typical logs. Furthermore, to enhance the merging of log templates, we design a
chain-of-thought method for large language models (LLMs). LLMs exhibit
exceptional semantic comprehension, deftly distinguishing between parameters
and invariant tokens. We have conducted experiments on large-scale public
datasets. Extensive evaluation demonstrates that Lemur achieves the
state-of-the-art performance and impressive efficiency.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要