Stream processing applications have been widely adopted due to real-time data analytics demands, e.g., fraud detection, video analytics, IoT applications. Unfortunately, prototyping and testing these applications is still a cumbersome process for developers that usually requires an expensive testbed and deep multidisciplinary expertise, including in areas such as networking, distributed systems, and data engineering. As a result, it takes a long time to deploy stream processing applications into production and yet users face several correctness and performance issues. In this paper, we present stream2gym, a tool for the fast prototyping of large-scale distributed stream processing applications. stream2gym builds on Mininet, a widely adopted network emulation platform, and provides a high-level interface to enable developers to easily test their applications under various operating conditions. We demonstrate the benefits of stream2gym by prototyping and testing several applications as well as reproducing key findings from prior research work in video analytics and network traffic monitoring. Moreover, we show stream2gym presents accurate results compared to a hardware testbed while consuming a small amount of resources (enough to be supported in a single commodity laptop even when emulating a dozen of processing nodes).

Commodity-of-the-shelf (COTS) data streaming platforms, e.g., Apache Kafka, RabbitMQ, have been powering companies from several sectors to move and extract further insights from their data. Yet, little is known about the suitability of these platforms for mission critical applications such as disaster recovery, first-aid response and military coordination. This paper provides a provocative discussion on how prepared current data streaming platforms are for mission critical scenarios. We argue that, despite their intrinsic reliability and security features, these platforms are still hard to test/operate which ultimately hinders trust on their safety and security conditions. Moreover, they commonly assume a powerful underlying infrastructure, which makes it tough to keep a high performance in resource-constrained environments (e.g., satellite networks). We conclude by identifying relevant research directions to ameliorate the performance and suitability of data streaming platforms for mission-critical applications.