XHAMI – extended HDFS and MapReduce interface for Big Data image processing applications in cloud computing environments

Abstract

Hadoop distributed file system (HDFS) and MapReduce model have become popular technologies for large-scale data organization and analysis. Existing model of data organization and processing in Hadoop using HDFS and MapReduce are ideally tailored for search and data parallel applications, for which there is no need of data dependency with its neighboring/adjacent data. However, many scientific applications such as image mining, data mining, knowledge data mining, and satellite image processing are dependent on adjacent data for processing and analysis. In this paper, we identify the requirements of the overlapped data organization and propose a two-phase extension to HDFS and MapReduce programming model, called XHAMI, to address them. The extended interfaces are presented as APIs and implemented in the context of image processing application domain. We demonstrated effectiveness of XHAMI through case studies of image processing functions along with the results. Although XHAMI has little overhead in data storage and input/output operations, it greatly enhances the system performance and simplifies the application development process. Our proposed system, XHAMI, works without any changes for the existing MapReduce models and can be utilized by many applications where there is a requirement of overlapped data. Copyright © 2016 John Wiley & Sons, Ltd.