Dr. Lu's book on 'High-Performance Big Data Computing' got published!

We are very happy to announce that our book on 'High-Performance Big Data Computing' is finally getting launched by the MIT Press.

HPBDC Book Cover

The book provides an in-depth overview of an emerging field that brings together high-performance computing, big data processing, and deep learning. It is targeted at students, researchers, and professionals working in these fields. Suitable chapters can also be used in graduate and/or senior-level classes. It contains the results of many state-of-the-art research directions and publications done worldwide.

A short overview of this book is as follows:

The book covers basic concepts and necessary background knowledge, including data processing frameworks, storage systems, and hardware capabilities; offers a detailed discussion of technical issues in accelerating big data computing in terms of computation, communication, memory and storage, codesign, workload characterization and benchmarking, and system deployment and management; and surveys benchmarks and workloads for evaluating big data middleware systems. It presents a detailed discussion of big data computing systems and applications with high-performance networking, computing, and storage technologies, including state-of-the-art designs for data processing and storage systems. Finally, the book considers some advanced research topics in high-performance big data computing, including designing high-performance deep learning over big data (DLoBD) stacks and HPC cloud technologies.

More details on this book are available from here.

Congratulations to all the authors!

The NVMe-oAF paper got accepted in HPDC22!

Our NVMe-oAF paper got accepted in HPDC 2022! Congratulations to Arjun!

Paper Info

[HPDC'22] NVMe-oAF: Towards Adaptive NVMe-oF for IO-Intensive Workloads on HPC Cloud

Arjun Kashyap and Xiaoyi Lu.

In Proceedings of the 31st ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2022. (Acceptance Rate: 19%)

Abstract Applications running inside containers or virtual machines, traditionally use TCP/IP for communication in HPC clouds and data centers. The TCP/IP path usually becomes a major performance bottleneck for applications performing NVMe-over-Fabrics (NVMe-oF) based I/O operations in disaggregated storage settings. We propose an adaptive communication channel, called NVMe-over-Adaptive-Fabric (NVMe-oAF), that applications could leverage to eliminate the high-latency and low-bandwidth incurred by remote I/O requests over TCP/IP. NVMe-oAF accelerates I/O intensive applications using locality awareness along with optimized shared memory and TCP/IP paths. The adaptiveness of the fabric stems from the ability to adaptively select shared memory or TCP channel and further applying optimizations for the chosen channel. To evaluate NVMe-oAF, we co-design Intel's SPDK library with our designs and show up to 7.1x bandwidth improvement and up to 4.2x latency reduction for various workloads over commodity TCP/IP-based Ethernet networks (e.g., 10Gbps, 25Gbps, and 100Gbps). We achieve similar (or sometimes better) performance when compared to NVMe-over-RDMA by avoiding the cumbersome management of RDMA in HPC cloud environments. Finally, we also co-design NVMe-oAF with H5bench to showcase the benefit it brings to HDF5 applications. Our evaluation indicates up to a 7x bandwidth improvement when compared with the network file system (NFS).