Operational-Log Analysis for Big Data Systems: Challenges and Solutions.

Big data systems (BDSs) are complex, consisting of multiple interacting hardware and software components, such as distributed computing nodes, databases, and middleware. Any of these components can fail. Finding the failures' root causes is extremely laborious. Analysis of BDS-generated logs can speed up this process. The logs can also help improve testing processes, detect security breaches, customize operational profiles, and aid with any other tasks requiring runtime-data analysis. However, practical challenges hamper log analysis tools' adoption. The logs emitted by a BDS can be thought of as big data themselves. When working with large logs, practitioners face seven main issues: scarce storage, unscalable log analysis, inaccurate capture and replay of logs, inadequate log-processing tools, incorrect log classification, a variety of log formats, and inadequate privacy of sensitive data. Some practical solutions exist, but serious challenges remain. This article is part of a special issue on Software Engineering for Big Data Systems.