Inside the Message Passing Interface
Description
A hands-on guide to writing a Message Passing Interface, this book takes the reader on a tour across major MPI implementations, best optimization techniques, application relevant usage hints, and a historical retrospective of the MPI world, all based on a quarter of a century spent inside MPI. Readers will learn to write MPI implementations from scratch, and to design and optimize communication mechanisms using pragmatic subsetting as the guiding principle. Inside the Message Passing Interface also covers MPI quirks and tricks to achieve best performance.
Dr. Alexander Supalov created the Intel Cluster Tools product line, including the Intel MP Library that he designed and led between 2003 and 2015. He invented the common MPICH ABI and also guided Intel efforts in the MPI Forum during the development of the MPI-2.1, MPI-2.2, and MPI-3 standards. Before that, Alexander designed new finite-element mesh-generation methods, contributing to the PARMACS and PARASOL interfaces, and developed the first full MPI-2 and IMPI implementations in the world. He graduated from the Moscow Institute of Physics and Technology in 1990, and earned his PhD in applied mathematics at the Institute of Numerical Mathematics of the Russian Academy of Sciences in 1995. Alexander holds 26 patents (more pending worldwide).
More details
Other editions
Additional editions
Person
Dr. Alexander Supalov , Supalov HPC, Germany
Content
What this book is about
Who should read this book
Notation and conventions
How to read this book
Overview
Parallel computer
Intraprocessor parallelism
Interprocessor parallelism
Exercises
MPI standard
MPI history
Related standards
Exercises
MPI subsetting
Motivation
Typical examples
Implementation practice
Exercises
Shared memory - Learn how to create a simple MPI subset capable of basic blocking point-to-point and collective operations over shared memory
Subset definition
General assumptions
Blocking point-to-point communication
Blocking collective operations
Exercises
Communication mechanisms
Basic communication
Intraprocess performance
Interprocess performance
Exercises
Startup and termination
Process creation
Two processes
More processes
Connection establishment
Process termination
Exercises
Blocking point-to-point communication
Limited message length
Blocking protocol
Unlimited message length
Double buffering
Eager protocol
Rendezvous protocol
Exercises
Blocking collective operations
Naive algorithms
Barrier
Broadcast
Reduce and Allreduce
Exercises
Sockets - Learn how to create an MPI subset capable of all point-to-point and blocking collective operations over Ethernet and other IP capable networks
Subset definition
General assumptions
Blocking point-to-point communication
Nonblocking point-to-point operations
Blocking collective operations
Exercises
Communication mechanisms
Basic communication
Intranode performance
Internode performance
Exercises
Synchronous progress engine
Communication establishment
Data transfer
Exercises
Startup and termination
Process creation
Startup command
Process daemon
Out-of-band communication
Host name resolution
Connection establishment
At startup (eager)
On request (lazy)
Process termination
Exercises
Blocking point-to-point communication
Source and tag matching
Unexpected messages
Exercises
Nonblocking point-to-point communication
Request management
Exercises
Blocking collective operations
Communication context
Basic algorithms
Tree based algorithms
Circular algorithms
Hypercube algorithms
Exercises
OFA libfabrics - Learn how to create an MPI subset capable of all point-to-point and collective operations over InfiniBand and upcoming future networks
Subset definition
General assumptions
Point-to-point operations
Collective operations
Exercises
Communication mechanisms
Basic communication
Intranode performance
Internode performance
Exercises
Startup and termination
Process creation
Credential exchange
Connection establishment
Process termination
Exercises
Point-to-point communication
Blocking communication
Nonblocking communication
Exercises
Collective operations
Advanced algorithms
Blocking operations
Nonblocking operations
Exercises
Advanced features - Learn how to add advanced MPI features including but not limited to heterogeneity, one-sided communication, file I/O, and language bindings
Communication modes
Standard
Buffered
Synchronous
Heterogeneity
Basic datatypes
Simple datatypes
Derived datatypes
Exercises
Groups, communicators, topologies
Group management
Communicator management
Process topologies
Exercises
One-sided communication
Mapped implementation
Native implementation
Exercises
File I/O
Standard I/O
MPI file I/O
Exercises
Language bindings
Fortran
C++
Java
Python
Exercises
Optimization - Learn how to optimize MPI internally by using advanced implementation techniques and available special hardware
Direct data transfer
Direct memory access
Remote direct memory access
Exercises
Threads
Thread support level
Threads as MPI processes
Shared memory extensions
Exercises
Multiple fabrics
Synchronous progress engine
Asynchronous progress engine
Hybrid progress engine
Exercises
Dedicated hardware
Synchronization
Special memory
Auxiliary networks
Exercises
Look ahead - Learn to recognize MPI advantages and drawbacks to better assess its future
MPI axioms
Reliable data transfer
Ordered message delivery
Dense process rank sequence
Exercises
MPI-4 en route
Fault tolerance
Exercises
Beyond MPI
Exascale challenge
Exercises
References - Learn about books that may further extend your knowledge
Appendices
MPI Families - Learn about major MPI implementation families, their genesis, architecture and relative performance
MPICH
Genesis
Architecture
Details
MPICH
MVAPICH
Intel MPI
...
Exercises
OpenMPI
Genesis
Architecture
Details
Exercises
Comparison
Market
Features
Performance
Exercises
Alternative interfaces - Learn about other popular interfaces that are used to implement MPI
DAPL
...
Exercises
SHMEM
...
Exercises
GasNET
...
Exercises
Portals
...
Exercises
Solutions to all exercises - Learn how to answer all those questions