High-performance computing

Halaman induk roadmap belajar HPC. Link merah = artikel yang belum ditulis. Tulis satu per satu; kalau semua sudah biru, kuliah master tinggal sambil tidur.

Fondasi yang sudah ada: Amdahl's law, Gustafson's Law, GFLOPS, Memory Coalescing, N-Body Problems, SoA vs AoS, Cache line, NUMA, seri MPI dan OpenCL.

Hello World

Amdahl's law Gustafson's Law GFLOPS Memory Coalescing N-Body Problems SoA vs AoS Cache line NUMA

MPI

MPI: Setup & Hello World MPI: Send & Recv MPI: Scatter MPI: Bcast MPI: gather MPI: Reduce MPI: Probe

OpenCL

OpenCL: Setup & Hello World OpenCL: OpenCL Specification OpenCL: Hello World (lengkap) OpenCL: Memory Objects OpenCL: Data Transfers OpenCL: Memory Region OpenCL: 2D Array

Tier 1: Fondasi Wajib

OpenMP

• OpenMP: Setup & Hello World
• OpenMP: Parallel For & Scheduling
• OpenMP: Reduction & Sinkronisasi
• OpenMP: Tasking
• OpenMP: Affinity & First-Touch
• OpenMP: False Sharing
• OpenMP: Hybrid MPI+OpenMP

CUDA

• CUDA: Setup & Hello World
• CUDA: Thread Hierarchy (Grid, Block, Warp)
• CUDA: Memory Model & Shared Memory
• CUDA: Matmul dengan Tiling
• CUDA: Reduction & Scan
• CUDA: Streams & Overlap
• CUDA: Occupancy & Warp Divergence
• CUDA: Perbandingan dengan OpenCL

Teori Algoritma Paralel

• Work-Span Model & Brent's Theorem
• Parallel Prefix Sum (Scan)
• Parallel Sorting
• Isoefficiency & Scalability
• Roofline model

Metode Numerik

• IEEE 754 & Floating Point Error
• BLAS: Level 1, 2, 3
• Dekomposisi LU & Cholesky
• Sparse Matrix: Format CSR & CSC
• SpMV
• Iterative Solvers: Jacobi & Gauss-Seidel
• Conjugate Gradient
• Stencil Computation & Heat Equation
• FFT: Konsep Dasar

MPI Lanjutan

• MPI: Isend & Irecv (Non-Blocking)
• MPI: Overlap Komunikasi-Komputasi
• MPI: Derived Datatypes
• MPI: Communicators & Topology
• MPI: One-Sided (RMA)
• MPI: Algoritma di Balik Collectives

Tier 2: Kehidupan di Cluster

Slurm & Environment

• Slurm: sbatch, srun, squeue
• Slurm: Job Array & Resource Request
• Module Environment (Lmod)

Profiling & Benchmark

• Profiling: perf
• Profiling: likwid
• Profiling: Nsight Compute & Nsight Systems
• Profiling: Score-P & TAU
• Benchmark: STREAM
• Benchmark: HPL & HPCG
• Benchmark: OSU Microbenchmarks

SIMD

• SIMD
• SIMD: AVX2 Intrinsics
• SIMD: Auto-Vectorization & Laporan Compiler

Interconnect

• Topologi Jaringan HPC: Fat Tree, Dragonfly, Torus
• InfiniBand & RDMA

Tier 3: Pembeda

Distributed Deep Learning

• Distributed Training: Data, Model, Pipeline Parallelism
• NCCL & Ring Allreduce
• Mixed Precision Training

Ekosistem & Riset

• Kokkos: Konsep Dasar
• SYCL: Konsep Dasar
• Membaca Paper HPC: SC, PPoPP, IPDPS

Proyek Latihan

• Proyek: Heat Equation 2D dengan MPI Halo Exchange
• Proyek: Hybrid MPI+OpenMP Heat Equation
• Proyek: Port Matmul OpenCL ke CUDA
• Proyek: Benchmark Cluster Heterogen (STREAM, HPL, HPCG, OSU)
• Proyek: Analisis Roofline Matmul