Data pipeline architecture for serverless platform

Abstract

To provide cost effective cloud resources with high QoS, serverless platform is introduced that allows to pay for the exact amount of resource usage. On the other hand, a number of data management tools are developed to handle the data from a large number of IoT sensing devices. However, the modern data-intensive cloud applications require the power that comes from integrating data management tools with serverless platforms. This paper proposes a novel data pipeline architecture for serverless platform for providing an environment to develop applications that can be broken into independently deployable, schedulable, scalable, and re-usable modules and efficiently manage the flow of data between different environments.