Open Problems in Stream Processing: A Call To Action

In the last four years, stream processing has gone from niche to mainstream, with real-time data processing systems gaining traction in not only fast-moving startups, but also their more skeptical and cautious enterprise brethren. In light of such pervasive adoption, is it safe to say we've we finally reached the point where stream processing is a solved commodity? Are we done and ready to move on to the next big thing? In this talk, I will argue that the answer to those questions is conclusively "no": stream processing as a field of research is alive and well. In fact, as streaming systems evolve to look more and more alike, the need for active exploration of new ideas is all the more pressing. And though streaming systems are more capable and robust than ever, they remain in many ways difficult to use, difficult to maintain, and difficult to understand. But we can change that. I don't claim to have all the answers; no one does. But I do have a few ideas of where we can start. And by sharing my thoughts on some of the more interesting open problems in stream processing, and encouraging others to share theirs, I'm hoping that we as a research community can work together to help move the needle just a little bit further.