Deakin University
Browse

File(s) under embargo

Lc-Stream: An elastic scheduling strategy with latency constraints in geo-distributed stream computing environments

journal contribution
posted on 2024-04-12, 06:05 authored by D Sun, Y Wang, J Sui, Shang GaoShang Gao, J Rong, R Buyya
SummaryAn effective scheduling strategy is critical for achieving better performance in real‐time stream processing systems. How to quickly and efficiently process real‐time data stream is always challenging, especially when clusters are collaborating in a Geo‐Distributed computing environment. To address these challenges, we propose an elastic scheduling strategy with Latency Constraints in Geo‐Distributed stream computing environments called Lc‐Stream. This article discusses our work from the following aspects: (1) An optimized data stream redirection method that is proposed based on queuing network algorithm, along with a computing resource model, a latency constrained scheduling model and a communication energy consumption model. (2) An updated node selection method based on the inter‐layer task correlation, to reduce the communication latency between groups at the executor granularity. (3) A network cluster distribution for Geo‐Distributed computing environment to ensure energy saving under low transmission latency. Experimental results show that compared to R‐Storm, Lc‐Stream reduces total latency by over 19% and increases throughput by over 37% in typical cross‐domain multi‐task topologies. Compared to Ts‐Stream, Lc‐Stream also reduces total latency by over 15% and increases throughput by over 21%. At the same time, it helps to balance the load among the systems and avoid overuse of compute nodes.

History

Journal

Concurrency and Computation: Practice and Experience

Location

London, Eng.

ISSN

1532-0626

eISSN

1532-0634

Language

eng

Publication classification

C1 Refereed article in a scholarly journal

Publisher

Wiley

Usage metrics

    Research Publications

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC