Research on the integration of UnifyFS and Slurm

Juzheng Bai; Wenqiang Zhu; Tao Zhang; Mengzhi Han

doi:10.1117/12.3031392

19 July 2024 Research on the integration of UnifyFS and Slurm

Juzheng Bai, Wenqiang Zhu, Tao Zhang, Mengzhi Han

Proceedings Volume 13181, Third International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT 2024); 131813I (2024) https://doi.org/10.1117/12.3031392
Event: Third International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT 2024), 2024, Beijing, China

Abstract

UnifyFS, a user-level file system specifically designed for burst buffers, aims to integrate the existing node-local storage layers in parallel computing systems, thereby providing a unified namespace for parallel computing applications. This study details the successful integration of UnifyFS with the Slurm resource manager, and emphasizes the synergistic effects of this integration in enhancing the performance of parallel computing systems. Detailed evaluations of this integration on leading parallel computing systems show that the integrated system exhibits significant advantages in scalability of data writing operations, and achieves notable performance improvements in the read-write bandwidth of applications, bringing higher efficiency and improvements to parallel computing applications. The paper thoroughly analyzes the effective integration mechanisms of UnifyFS and Slurm, and explores its potential in optimizing the collaborative operation of distributed storage and task scheduling, offering a viable solution to the broad parallel computing community. Future research will focus on further optimizing this integration framework, adding more advanced features, to achieve broader applications and technological innovations.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Juzheng Bai, Wenqiang Zhu, Tao Zhang, and Mengzhi Han "Research on the integration of UnifyFS and Slurm", Proc. SPIE 13181, Third International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT 2024), 131813I (19 July 2024); https://doi.org/10.1117/12.3031392

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available