High-performance computing

Hello World

• Amdahl's law
• Gustafson's Law
• GFLOPS
• Memory Coalescing
• N-Body Problems
• SoA vs AoS
• Cache line
• NUMA

MPI

• MPI: Setup & Hello World
• MPI: Send & Recv
• MPI: Scatter
• MPI: Bcast
• MPI: gather
• MPI: Reduce
• MPI: Probe

OpenCL

• OpenCL: Setup & Hello World
• OpenCL: OpenCL Specification
• OpenCL: Hello World (lengkap)
• OpenCL: Memory Objects
• OpenCL: Data Transfers
• OpenCL: Memory Region
• OpenCL: 2D Array

Tier 1: Fondasi Wajib

OpenMP

• OpenMP: Setup & Hello World
• OpenMP: Parallel For & Scheduling
• OpenMP: Reduction & Sinkronisasi
• OpenMP: Tasking
• OpenMP: Affinity & First-Touch
• OpenMP: False Sharing
• OpenMP: Hybrid MPI+OpenMP

CUDA

• CUDA: Setup & Hello World
• CUDA: Thread Hierarchy (Grid, Block, Warp)
• CUDA: Memory Model & Shared Memory
• CUDA: Matmul dengan Tiling
• CUDA: Reduction & Scan
• CUDA: Streams & Overlap
• CUDA: Occupancy & Warp Divergence
• CUDA: Perbandingan dengan OpenCL

Teori Algoritma Paralel

• Work-Span Model & Brent's Theorem
• Parallel Prefix Sum (Scan)
• Parallel Sorting
• Isoefficiency & Scalability
• Roofline model

Metode Numerik

• IEEE 754 & Floating Point Error
• BLAS: Level 1, 2, 3
• Dekomposisi LU & Cholesky
• Sparse Matrix: Format CSR & CSC
• SpMV
• Iterative Solvers: Jacobi & Gauss-Seidel
• Conjugate Gradient
• Stencil Computation & Heat Equation
• FFT: Konsep Dasar

MPI Lanjutan

• MPI: Isend & Irecv (Non-Blocking)
• MPI: Overlap Komunikasi-Komputasi
• MPI: Derived Datatypes
• MPI: Communicators & Topology
• MPI: One-Sided (RMA)
• MPI: Algoritma di Balik Collectives

Tier 2: Kehidupan di Cluster

Slurm & Environment

• Slurm: sbatch, srun, squeue
• Slurm: Job Array & Resource Request
• Module Environment (Lmod)

Profiling & Benchmark

• Profiling: perf
• Profiling: likwid
• Profiling: Nsight Compute & Nsight Systems
• Profiling: Score-P & TAU
• Benchmark: STREAM
• Benchmark: HPL & HPCG
• Benchmark: OSU Microbenchmarks

SIMD

• SIMD
• SIMD: AVX2 Intrinsics
• SIMD: Auto-Vectorization & Laporan Compiler

Interconnect

• Topologi Jaringan HPC: Fat Tree, Dragonfly, Torus
• InfiniBand & RDMA

Tier 3: Pembeda

Distributed Deep Learning

• Distributed Training: Data, Model, Pipeline Parallelism
• NCCL & Ring Allreduce
• Mixed Precision Training

Ekosistem & Riset

• Kokkos: Konsep Dasar
• SYCL: Konsep Dasar
• Membaca Paper HPC: SC, PPoPP, IPDPS

Proyek Latihan

• Proyek: Heat Equation 2D dengan MPI Halo Exchange
• Proyek: Hybrid MPI+OpenMP Heat Equation
• Proyek: Port Matmul OpenCL ke CUDA
• Proyek: Benchmark Cluster Heterogen (STREAM, HPL, HPCG, OSU)
• Proyek: Analisis Roofline Matmul