High-Performance Big Data and AI
Managing and processing large volumes of data, or Big Data, and gaining meaningful insights is a significant challenge facing the parallel and distributed computing community. This has significant impact in a wide range of domains including health care, bio-medical research, Internet search, finance and business informatics, and scientific computing. Designing and building high-performance Big Data and AI systems are of significant importance to both research communities and industry.
(Note: Logo is designed by PADSYS Lab. All rights reserved.)
As data-gathering technologies and data sources witness an explosion in the amount of input data, it is expected that in the future massive quantities of data in the order of hundreds or thousands of petabytes will need to be processed. Thus, it is critical that distributed computing systems for Big Data and AI (such as Hadoop, Spark, Flink, TensorFlow, PyTorch, etc.) are diligently designed, with high performance and scalability, in order to meet the growing demands of such Big Data applications.
PADSYS Lab members design and develop high-performance Big Data and AI systems and libraries for the emerging HPC and datacenter architectures. Our proposed designs and research studies aim to bring HPC, Big Data processing, AI, and Cloud Computing into a convergent trajectory.
Acknowledgement
Thanks a lot for NSF's support on this research direction!
Grant Information
-
BD Spokes: SPOKE: MIDWEST: Collaborative:Advanced Computational Neuroscience Network (ACNN)
-
BIGDATA: F: DKM: Collaborative Research: Scalable Middleware for Managing and Processing Big Data on Next Generation HPC Systems. Prof. Xiaoyi Lu was one of the senior personel on this grant.
Software Release
-
DataMPI: Extending MPI for Big Data with Key-Value based Communication
-
High-Performance Big Data (HiBD). Prof. Xiaoyi Lu was the co-founder and R&D leader for this project when he worked at NOWLAB@OSU. Some of his students had contributed a lot of effort and time on this project as well.
-
RDMA-TensorFlow. Prof. Xiaoyi Lu was the co-founder and R&D leader for this project when he worked at NOWLAB@OSU.