DVM: Scaling out virtual memory in userspace

Abdullah Al-Mamun, Ke Wang, Jialin Liu, Dongfang Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

One of the most challenging problems in modern distributed big data systems lies in their memory management: these systems pre-allocate a fixed amount of memory before applications start. In the best case where more memory can be acquired, users have to reconfigure the deployment and re-compute many intermediate results. If no more memory is available, users are then forced to manually partition the job into smaller tasks, incurring both development and performance overhead. This paper presents a user-level utility for scaling the memory in a distributed setup - the Distributed Virtual Memory (DVM). DVM enables to efficiently swap data between memory and disk between arbitrary nodes without users' intervention or applications' awareness.

Original languageEnglish (US)
Title of host publication47th International Conference on Parallel Processing, ICPP 2018
Subtitle of host publicationWorkshop Proceedings
PublisherAssociation for Computing Machinery
ISBN (Print)9781450365239
DOIs
StatePublished - Aug 13 2018
Externally publishedYes
Event47th International Conference on Parallel Processing, ICPP 2018 - Eugene, United States
Duration: Aug 13 2018Aug 16 2018

Publication series

NameACM International Conference Proceeding Series

Conference

Conference47th International Conference on Parallel Processing, ICPP 2018
Country/TerritoryUnited States
CityEugene
Period8/13/188/16/18

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'DVM: Scaling out virtual memory in userspace'. Together they form a unique fingerprint.

Cite this