1. Definition and Scope of HPC

Using computing resources that provide enhanced compute power to solve large-scale problems in a reasonable amount of time.

• 4 Basic Building Blocks of HPC:
  1. Theory: Numerical laws, physical models, speed-up performance, etc.
  2. Technology: Multi-core systems, supercomputers, networks, storage.
  3. Architecture: Shared-memory, distributed-memory, interconnects.
  4. Software: Libraries, schedulers, monitoring tools, applications.
• Characteristics of HPC Problems: They require a long time to compute, demand significant hardware resources (such as memory), often need multiple runs, and are time-critical (e.g., climate modeling, protein folding, drug discovery, energy research, data analysis).

2. Why We Build Parallel Systems (The Hardware Shift)

• Up until the early 2000s, microprocessor performance grew by ~50% per year. Since then, single-core speedup dropped to ~20% per year due to physical limits:
• The Solution: Move from single-core architectures to multicore processors to introduce parallelism.

3. Why We Need to Write Parallel Programs (The Software Challenge)

• Serial Code Limitation: Serial programs do not automatically run faster on multi-core systems.
• Automatic Parallelization Limitations: Automatically translating serial code into parallel code is extremely difficult, often resulting in highly inefficient execution.
• Algorithm Redesign: Often, obtaining high performance requires stepping back and designing an entirely new parallel algorithm rather than converting a serial loop.

4. Parallel Programming Concepts & Paradigms

• Task Parallelism: Partitioning different tasks or operations across cores.
• Data Parallelism: Partitioning data across cores, where each core performs similar operations on its subset of data.
• Core Coordination Requirements:
  • Communication: Cores sending partial values or messages to one another.
  • Load Balancing: Distributing work evenly across cores to avoid bottlenecks.
  • Synchronization: Ensuring cores operating at different speeds do not get out of step.
• concurrent computing: multiple tasks in progress at any instant — more than a task at the same time
• parallel computing: program = multiple tasks cooperate closely — it was a single task but we made it into many tasks to run in parallel — note that we can have parallelism with concurrency: multiple tasks divided into smaller more multiple tasks
• distributed computing: program cooperate with other programs

5. Parallel Architectures

• Multiprocessor (Shared-Memory): Multiple CPUs share a single global address space.
  • Centralized: Primary memory is in one central location.
  • Distributed: Physical memory is distributed across nodes, but shares a global address space.