Deakin University
Browse

File(s) under permanent embargo

Engineering complex data integration, harmonization and visualization systems

journal contribution
posted on 2019-12-01, 00:00 authored by Iman Avazpour, John Grundy, Liming Zhu
Complex data transformation, aggregation and visualization problems are becoming increasingly common. These are needed in order to support improved business intelligence and end-user access to data. However, most such applications present very challenging software engineering problems including noisy data, diverse data formats and APIs, challenging data modeling and increasing demand for sophisticated visualization support. This paper describes a data integration, harmonization and visualization process and framework that we have been developing. We discuss our approach used to tackle complex data aggregation and harmonization problems and we demonstrate a set of information visualizations that can be developed from the harmonized data to make it usable for its target audience. We use a case study of Household Travel Survey data mapping, harmonization, aggregation and visualization to illustrate our approach. We summarize a set of lessons that we have learned from this industry-based software engineering experience. We hope these will be useful for others embarking on challenging data harmonization and integration problems. We also identify several key directions and needs for future research and practical support in this area.

History

Journal

Journal of industrial information integration

Volume

16

Article number

100103

Pagination

1-13

Location

Amsterdam, The Netherlands

ISSN

2452-414X

Language

eng

Publication classification

C1 Refereed article in a scholarly journal

Copyright notice

2019, Elsevier

Publisher

Elsevier