Multi-Processor Systems: MPI

• Definition: A portable library for writing parallel programs based on the message-passing model.
• Availability: Universally supported across High-Performance Computing (HPC) platforms.
• Execution: Requires all parallelism to be explicitly defined.

Explicit Parallelism

• Concept: The developer is responsible for designing the parallel decomposition and implementation.
• Key Decisions: Manually determining how to divide the problem and exactly when processes must communicate.
• MPI vs. OpenMP: MPI is fully explicit. OpenMP is less explicit, as the developer only specifies which parts of the program to parallelize, leaving the compiler to generate the parallel executable.

GPGPU (General-Purpose Computing on GPUs)

• Concept: Utilizing a Graphic Processing Unit (GPU) in conjunction with a Central Processing Unit (CPU) to accelerate computations that were traditionally handled solely by the CPU.

CUDA

• Definition: A proprietary Application Program Interface (API) created by NVIDIA specifically for programming their GPU accelerators.
• Execution: Parallelism is implicit in how the kernel is designed and launched, making it conceptually similar to OpenMP.

OpenCL

• Definition: A cross-platform, open standard for programming various accelerators, including GPUs from multiple vendors like NVIDIA and AMD.
• Trade-offs: It offers high portability across different hardware platforms but is generally harder to write than CUDA and makes maintaining consistent performance across those platforms difficult.

Memory Types

Shared Memory: all CPUs share the same memory