Unlocking social media and user generated content as a data source for knowledge management

The pervasiveness of social media and user-generated content has triggered an exponential increase in global data. However, due to collection and extraction challenges, data in embedded comments, reviews and testimonials are largely inaccessible to a knowledge management system. This article describes a KM framework for the end-to-end knowledge management and value extraction from such content. This framework embodies solutions to unlock the potential of UGC as a rich, real-time data source. Three contributions are described in this article. First, a method for automatically navigating webpages to expose UGC for collection is presented. This is evaluated using browser emulation integrated with automated collection. Second, a method for collecting data without any a priori knowledge of the sites is introduced. Finally, a new testbed is developed to reflect the current state of internet sites and shared publicly to encourage future research. The discussion benchmarks the new algorithm alongside existing techniques, providing evidence of the increased amount of UGC data extracted.