Resource Management for Processing Wide Area Data Streams on Supercomputers

dc.contributor.author Chung, Joaquin
dc.contributor.author Adhikari, Mainak
dc.contributor.author Srirama, Satish Narayana
dc.contributor.author Jung, Eun Sung
dc.contributor.author Kettimuthu, Rajkumar
dc.date.accessioned 2022-03-27T06:03:38Z
dc.date.available 2022-03-27T06:03:38Z
dc.date.issued 2020-08-01
dc.description.abstract Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments' memory directly to a supercomputer's memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis.
dc.identifier.citation Proceedings - International Conference on Computer Communications and Networks, ICCCN. v.2020-August
dc.identifier.issn 10952055
dc.identifier.uri 10.1109/ICCCN49398.2020.9209669
dc.identifier.uri https://ieeexplore.ieee.org/document/9209669/
dc.identifier.uri https://dspace.uohyd.ac.in/handle/1/9230
dc.subject data streaming
dc.subject online resource provisioning
dc.subject wide area networks
dc.title Resource Management for Processing Wide Area Data Streams on Supercomputers
dc.type Conference Proceeding. Conference Paper
dspace.entity.type
Files
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Plain Text
Description: