With the emergence of data streaming applications that produce large data in motion, anomaly detection in non-stationary environments has become a major research focus. Unknown and unstable behaviour of data over time, limits the application of traditional anomaly detection methods that have been designed for stationary data. Moreover, basic assumptions of many existing works in the adaptive anomaly detection domain, such as the availability of labelled data over time or dealing with a known type of change in the data, are not valid for real-life applications. In this paper, we propose an unsupervised ensemble based anomaly detection method using One Class Support Vector Machines (OCSVMs). The proposed method is able to detect potential changes in the distribution of the normal data and adapts itself accordingly, without requiring any external feedback, e.g., ground truth labels. Moreover, it is able to automatically select an appropriate set of recent instances during learning phases, and an appropriate set of models during prediction phases by identifying active concepts. We evaluate our proposed method against state-of-the-art adaptive anomaly detection methods that can be applied in an unsupervised manner, on both real and synthetic non-stationary data. The results show that with considerably lower computational cost, our method outperforms the other methods.