Deakin University
Browse

A hybrid approach to clustering in big data

Version 2 2024-06-04, 06:14
Version 1 2016-10-12, 22:34
journal contribution
posted on 2024-06-04, 06:14 authored by D Kumar, JC Bezdek, M Palaniswami, Sutharshan RajasegararSutharshan Rajasegarar, C Leckie, TC Havens
Clustering of big data has received much attention recently. In this paper, we present a new clusiVAT algorithm and compare it with four other popular data clustering algorithms. Three of the four comparison methods are based on the well known, classical batch k-means model. Specifically, we use k-means, single pass k-means, online k-means, and clustering using representatives (CURE) for numerical comparisons. clusiVAT is based on sampling the data, imaging the reordered distance matrix to estimate the number of clusters in the data visually, clustering the samples using a relative of single linkage (SL), and then noniteratively extending the labels to the rest of the data-set using the nearest prototype rule. Previous work has established that clusiVAT produces true SL clusters in compact-separated data. We have performed experiments to show that k-means and its modified algorithms suffer from initialization issues that cause many failures. On the other hand, clusiVAT needs no initialization, and almost always finds partitions that accurately match ground truth labels in labeled data. CURE also finds SL type partitions but is much slower than the other four algorithms. In our experiments, clusiVAT proves to be the fastest and most accurate of the five algorithms; e.g., it recovers 97% of the ground truth labels in the real world KDD-99 cup data (4 292 637 samples in 41 dimensions) in 76 s.

History

Journal

IEEE transactions on cybernetics

Volume

46

Pagination

2372-2385

Location

Piscataway, N.J.

ISSN

2168-2267

eISSN

2168-2275

Language

eng

Publication classification

C Journal article, C1.1 Refereed article in a scholarly journal

Copyright notice

2015, IEEE

Issue

10

Publisher

IEEE