The development of eye tracking-based applications has witnessed a number of advancements over the past few years. As a result, a number of low cost commercial remote vision-based eye trackers started to appear in the market. Consequently, a number of research communities started to explore the feasibility of extending the eye-tracking capabilities beyond single computer screen and utilize it in multi-screen setup. One of the main challenges for the wide adoption of such eye trackers in multi-screen setup, is their limitations when it comes to an intuitive and reliable way for tracking human eye movements across these multiple screens without losing much of the eye tracking data itself. In this work, a novel data-driven approach based on deep recurrent neural networks for a reliable and responsive switching mechanism between low cost multi-screen eye trackers is proposed. Our approach has achieved a competent results in terms of higher accuracy and lower positive rate in detecting accurately the screen the subject is attending to with F1 measure score of 85%.