An event-based data processing system using Kafka container cluster on Kubernetes environment

Smart manufacturing has become a big trend of a new industrial revolution in the manufacturing industry. The advancement of the Internet of Things has made production more efficient and effective through the automated collecting data system and Big Data technology. Dealing with a large amount of real-time production data will be a significant issue for intelligent manufacturing. This paper uses Apache Kafka’s high-performance, low-latency data stream processing platform to process data collection and store it in the Big Data System. Kafka was deployed through Kubernetes, where it has improved on the architecture’s scalability and applies this architecture to the aerospace manufacturing autoclave. These data are then used to analyze the autoclave equipment anomaly. Testing performed on the Kafka Producer Throughput demonstrates that in the event that all other parameters remain unchanged, the real throughput will increase along with the increase in the throughput limit that is being used. For instance, when the throughput limit is 1.2 million, the maximum throughput of this experiment is reached at 1.13 million transactions per second, while the transfer rate is 552.88 megabytes per second (MB/s). The value of the fetch size parameter is set to 10,48,576 by default (1 M). It takes half a time and a quarter of a time down, and it takes up to 2.5 times the value that was preset before you can witness the change in the parameters that affect the performance. The performance achieves its peak of 1.43 million data transferred per second at a speed of 347.93 megabytes per second, and the performance after that has a tendency to remain consistent.