Speaker
Description
We introduce PerfAdvisor, a lightweight Python package that uses an LLM-driven agent loop to diagnose GPU performance bottlenecks from supercomputer profiles. Modern HPC profiles contain millions of events across kernels, MPI calls, memory transfers, and idle gaps, making manual diagnosis slow and expert-intensive. PerfAdvisor automates this process by producing ranked, evidence-grounded bottleneck hypotheses and concrete mitigation suggestions in minutes. PerfAdvisor supports profiles from AMD ROCm Systems Profiler and NVIDIA Nsight Systems. It segments each profile into execution phases, computes summary metrics, and gives an LLM agent access to local, read-only tools for top-kernel summaries, idle-gap histograms, MPI and memory-transfer summaries, NVTX/ROCTx summaries, per-rank imbalance scoring, and arbitrary SQL queries. The agent iteratively gathers evidence until it returns ranked diagnoses with suggested fixes. Multiple LLM backends are supported, and prompt caching reduces token cost by up to 80%. We validate PerfAdvisor on NERSC Perlmutter using a CUDA benchmark suite with eight known performance pathologies and on OLCF Frontier using an analogous HIP benchmark suite. PerfAdvisor identifies the primary bottleneck in all eight cases with strong coverage of ground-truth mitigation strategies. In a MILC/QUDA lattice-QCD case study on NERSC Perlmutter, PerfAdvisor correctly diagnosed host-staged halo exchange and recommended GPUDirect RDMA, which reduced wall time by 43%.