Evolution of the HEPS Jupyter-based remote data analysis System
1 Institute of High Energy Physics, CAS, 100049 Beijing, China
2 University of Chinese Academy of Sciences, 100049 Beijing, China
3 Spallation Neutron Source Science Center, Dongguan 523803, China
Published online: 23 August 2021
High Energy Photon Source(HEPS) Experiment is expected to produce large amount of data and have diverse computing requirements for data analysis. Generally, scientists need to spend several days to setup their experimental environment, which greatly reduce the scientists’ work efficiency. In response to the above problems, we introduce a remote data analysis system for HEPS. The system provides users a web-based interactive interface based Jupyter, which makes scientists are able to process data analysis anytime and anywhere. Particularly, we discuss the system architecture as well as the key points of this system. A solution of managing and scheduling heterogeneous computing resources (CPU and GPU) is proposed, which adopts Kubernetes to achieve centralized heterogeneous resources management and resource expansion on demand. An improved Kubernetes resource scheduler is discussed, which dispatches upper applications to nodes combining with the computing cluster status. The system can transparently and quickly deploy the data analysis environment for users in seconds and reach the maximum resource utilization. We also introduce an automated deployment solution to improve the work efficiency of developers and help deploy multidisciplinary applications faster and automatically. A unified certification is illustrated to make sure the security of remote data access and data analysis. Finally, we will show the running status of the system.
© The Authors, published by EDP Sciences, 2021
This is an Open Access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.