Dynamic redirection of real-time data streams for elastic stream computing

An elastic stream computing system needs elastic adjustment of computing resource allocation and vertex parallelism to improve latency and throughput, which includes continuously or periodically scaling in/out the workload of computing nodes at runtime. Dynamic redirection can help with this elasticity issue by dynamically redirecting real-time data streams to computing resources. Due to the time-varying and unpredictable nature of real-time data streams, implementing redirection of data streams is challenging. Currently, the requirements of data streams redirection are not fully fulfilled, which directly affects the latency and throughput of stream computing systems. To bridge this gap, we proposed a dynamic redirection framework (called Dr-Stream) for elastic stream computing systems. This paper discussed the following aspects: (1) Investigating the dynamic redirection of real-time data streams, providing a general stream application model, a data stream model and a data stream grouping model, as well as formalizing the problem of load balancing optimization and data stream redirection. (2) Redirecting data streams among multiple instances of an operator at runtime by a lightweight load balancing strategy to improve the load balancing of a data center at the vertex level. Managing system states, especially the states of stateful operators by a logical ring-based strategy to improve accuracy. (3) Determining the number of instances for each operator, and deploying the instance(s) to computing nodes by a modified first-fit strategy at runtime. (4) Evaluating the fulfillment of low latency, high throughput, and load balancing objectives in a real-world distributed stream computing environment. Experimental results showed that Dr-Stream reduced the average system latency and load balancing of the data center by more than 20% and 15%, respectively. It also improved the average system stability by more than 15% and avoided over-utilization of computing nodes, as compared to the existing strategies in Storm.