intel-gpu-kernel-opt

Intel GPU Kernel Optimization Methodology

Systematic approach to profiling, diagnosing, and optimizing GPU kernels on Intel Xe2 (Lunar Lake/LNL, Battlemage/BMG) and Xe3 (Panther Lake/PTL, Panther Lake-H/PTLH) architectures. Covers roofline analysis, bottleneck identification, profiling tools, hardware comparison, and proven optimization patterns.

Version: 1.0.0 Last Updated: 2026-03-12

---

Table of Contents

1. Optimization Workflow
2. Roofline Analysis
3. Key Performance Metrics
4. Hardware Comparison: Xe2 vs Xe3
5. Common Bottlenecks
6. Profiling Tools
7. Optimization Patterns
8. Optimization Priority Checklist
9. Case Study: SDP HD=256 Optimization Journey

---

Optimization Workflow

The fundamental cycle for GPU kernel optimization:

Profile --> Identify Bottleneck --> Optimize --> Verify --> Repeat

Step-by-Step

1. Baseline: Establish a correct, reproducible benchmark with warmup iterations
2. Profile: Capture hardware metrics (VTune GPU Compute Hotspots or GTPin)
3. Identify bottleneck: Classify as compute-bound, memory-bound, or latency-bound
4. Hypothesize: Form a specific hypothesis about what limits performance
5. Optimize: Apply a single targeted optimization
6. Verify: Measure both correctness AND performance
7. Record: Document the result (positive or negative) for future reference
8. Repeat: Return to step 2

Critical rule: Apply ONE optimization at a time. Multiple simultaneous changes make it impossible to attribute performance differences.

---

Roofline Analysis

Classification

A kernel is either compute-bound or memory-bound:

Arithmetic Intensity (AI) = FLOPs / Bytes_transferred
Ridge Point = Peak_TFLOPS / Peak_BW_TB/s

If AI > Ridge Point: compute-bound (optimize ALU utilization)
If AI < Ridge Point: memory-bound (optimize memory access)

Xe2 (BMG) Roofline

Peak FP16 XMX:  135 TFLOPS
Peak BW:        520 GB/s = 0.52 TB/s
Ridge Point:    135 / 0.52 = ~260 FLOPs/byte

Most attention/GEMM kernels with HEAD_DIM >= 128 are compute-bound on BMG.

Xe3 (PTL) Roofline

Peak FP16 XMX:  TBD (depends on SKU, ~30-50 TFLOPS estimated)
Peak BW:        112 GB/s = 0.112 TB/s
Ridge Point:    ~300-450 FLOPs/byte (estimated)

PTL's lower bandwidth makes more workloads memory-bound compared to BMG.

Practical Roofline Calculation

// GEMM: C[M,N] = A[M,K] x B[K,N]
double flops = 2.0 * M * N * K;
double bytes = (M*K + K*N + M*N) * sizeof(half);  // read A, B; write C
double ai = flops / bytes;

// Flash Attention: 4*q*kv*d*heads + 2*q*kv*heads (softmax)
double attn_flops = 4.0 * q_len * kv_len * head_dim * num_heads
                  + 2.0 * q_len * kv_len * num_heads;

---

Key Performance Metrics

Benchmarking Best Practices

// Standard benchmark pattern
constexpr int WARMUP = 5;
constexpr int ITERATIONS = 100;

// Warmup (let GPU clock stabilize, populate caches)
for (int i = 0; i < WARMUP; i++) {
    kernel.execute();
    q.wait();
}

// Timed iterations
auto start = std::chrono::high_resolution_clock::now();
for (int i = 0; i < ITERATIONS; i++) {
    kernel.execute();
    q.wait();
}
auto end = std::chrono::high_resolution_clock::now();
double avg_ms = std::chrono::duration<double, std::milli>(end - start).count() / ITERATIONS;

---

Hardware Comparison: Xe2 vs Xe3

Architecture Implications

BMG (Xe2):

• Compute-rich: 32 XE cores, high TFLOPS ceiling
• High bandwidth: 520 GB/s dedicated GDDR6
• Optimization focus: maximize XMX utilization, minimize register spill
• doubleGRF essential for large tile sizes (256 GRF entries per thread)

PTL (Xe3):

• Bandwidth-constrained: 112 GB/s shared with CPU
• Fewer cores: 12 XE cores, lower absolute throughput
• Optimization focus: minimize memory traffic, maximize data reuse
• Shared memory: CPU activity impacts GPU bandwidth

---

Common Bottlenecks

1. Register Spill

Symptom: Performance far below compute roofline despite high arithmetic intensity.

Diagnosis: Build with -save-temps, check compiler output for spill/fill counts. In VTune, look for high "Send" cycle counts from register spill loads/stores.

Solutions:

• Reduce tile sizes (e.g., M=16 to M=8)
• Use doubleGRF (-Xs "-options -doubleGRF")
• Reduce live register ranges (reorder computations)
• Split large kernels

2. SLM Bank Conflicts

Symptom: SLM access latency higher than expected, visible in VTune shared memory stalls.

Diagnosis: Check SLM access patterns for stride conflicts. 32 banks, 4 bytes/bank on Intel GPUs.

Solutions:

• Pad SLM rows to avoid stride conflicts
• Ensure consecutive threads access consecutive banks
• Rearrange data layout in SLM

3. Barrier Overhead

Symptom: Many barriers per iteration, visible as synchronization stalls in profiler.

Diagnosis: Count barriers per kernel iteration. Each barrier has 50-100 cycle overhead.

Solutions:

• Minimize barrier count (combine phases)
• Use named barriers for subset synchronization
• Use split arrive/wait to overlap independent work
• Eliminate redundant barriers (e.g., double-buffered SLM may remove a barrier that was never blocking)

4. Memory Coalescing Issues

Symptom: Bandwidth utilization far below peak.

Diagnosis: Check access patterns. Consecutive threads should access consecutive addresses.

Solutions:

• Ensure contiguous block_load patterns
• Use lsc_load_2d for 2D tiled access (hardware handles coalescing)
• Avoid scatter/gather when block operations suffice

5. Low DPAS/XMX Utilization

Symptom: Compute-bound kernel with low XMX busy percentage.

Diagnosis: In ISA, check for consecutive DPAS instructions vs intervening sends/movs. VTune XMX busy metric.

Solutions:

• Increase DPAS density (more consecutive DPAS without interruption)
• Pipeline data loads with compute (prefetch next tile, compute current)
• Reduce register transpose overhead between DPAS phases
• Use {Atomic} annotations where possible (compiler-dependent)

6. Instruction Overhead (mov Instructions)

Symptom: Large blocks of mov(4|M0) instructions in ISA dump, especially for data shuffling.

Diagnosis: ISA disassembly analysis. Common in register transpose operations.

Solutions:

• Use SLM for data layout transformations (write one layout, read another)
• lsc_slm_scatter for transposing via SLM (proved +1.7 TFLOPS in SDP HD=256)
• Separate type conversion from data movement (FP32->FP16 before transpose)

---

Profiling Tools

VTune GPU Compute/Media Hotspots

Full-featured profiler for Intel GPUs. Captures hardware counters, timeline, and per-kernel metrics.

# Capture GPU Compute Hotspots
vtune -collect gpu-hotspots -knob gpu-sampling-interval=1 -- ./kernel.exe

# Key metrics to examine:
# - XMX Busy %
# - EU Active %
# - L3 Bandwidth
# - SLM Bandwidth
# - GPU Occupancy
# - Stall reasons (Scoreboard, Send, Inst Fetch, etc.)

See intel-gpu-vtune-profiling skill for detailed VTune workflow.

GTPin (ISA-Level Profiling)

Instruction-level profiling showing cycle counts per ISA instruction. Essential for micro-optimization.

# GTPin profiling
set GTPIN_KIT=C:\path\to\gtpin
%GTPIN_KIT%\Bin\gtpin.exe --installDir %GTPIN_KIT%\Bin ^
    -t LatencyProfiler -- ./kernel.exe

See xe2-gtpin-profiling skill for GTPin setup and ISA analysis.

ISA Disassembly

# Build with ISA dump
icpx kernel.cpp -o kernel.exe -fsycl -fsycl-targets=spir64_gen `
    -Xs "-device bmg -options -doubleGRF" -O2 -w -save-temps

# Disassemble
ocloc disasm -file <spirv_file> -device bmg -dump isa_dump/

# Key things to look for:
# - DPAS instruction count and density (consecutive DPAS)
# - mov instruction count (especially mov(4|M0) for register shuffles)
# - send instruction count (memory operations)
# - Spill/fill patterns (stack access)

---

Optimization Patterns

1. Load/Compute Separation

Separate all data loads from all compute operations. Allows the GPU to pipeline loads and fill memory latency with compute.

// BAD: interleaved load/compute
for (int i = 0; i < N; i++) {
    auto data = block_load<half, 128>(ptr + i * 128);
    result[i] = compute(data);
}

// GOOD: all loads first, then all compute (+29% measured)
simd<half, 128> data[N];
for (int i = 0; i < N; i++) data[i] = block_load<half, 128>(ptr + i * 128);
for (int i = 0; i < N; i++) result[i] = compute(data[i]);

2. Cross-Phase Prefetch

Prefetch data for the next phase while computing the current phase.

// During VS GEMM phase, prefetch K for next QK iteration
for (int k = 0; k < kv_len; k += KV_CHUNK) {
    // QK phase
    qk_compute(k);

    // Prefetch next K during VS phase
    if (k + KV_CHUNK < kv_len)
        lsc_prefetch_2d<...>(k_payload);  // Next K tile

    // VS phase
    vs_compute(k);
}

3. Pipelined K Loads

Load the next data tile during the current DPAS computation.

// Load first K tile
auto k_tile = lsc_load_2d<...>(k_payload);

for (int d = 0; d < D_BLKS; d++) {
    // Start next load
    k_payload.set_x((d + 1) * K);
    auto k_next = (d < D_BLKS - 1) ? lsc_load_2d<...>(k_payload) : k_tile;

    // Compute with current tile (overlaps with next load)
    acc = xmx::dpas<...>(acc, k_tile, q_tile[d]);

    k_tile = k_next;
}

4. Loop Unrolling

// Explicit unroll for critical inner loops
#pragma unroll
for (int i = 0; i < TILE_SIZE; i++) {
    acc += data[i] * weights[i];
}

#pragma unroll is essential for ESIMD inner loops. Without it, the compiler may generate loop overhead that dominates small tile computations.

5. SIMD Width Selection

Choose SIMD width based on the operation:

// SIMD16 for reductions (reciprocal, sum)
simd<float, 16> inv = 1.0f / sum;  // Single SIMD16 divide

// SIMD32 for mad fusion
// fp32_sum = fp32_sum * delta + local_sum generates SIMD32 mad with 2x throughput

// Match SIMD width to data tile size for zero-cost select operations
simd<half, 128> full_row;
auto slice = full_row.select<16, 1>(offset);  // Zero-cost view, no data movement

6. Deferred Compensation (Online Softmax)

In flash attention, defer the A_tile *= delta correction to before VS GEMM rather than interleaving inside the VS loop:

// BAD: interleaved compensation inside VS inner loop
for (int kv = 0; kv < KV_PER_SG; kv++) {
    a_tile[kv] *= delta;  // Breaks DPAS pipeline
    acc = dpas(acc, s_tile, v_tile);
}

// GOOD: deferred compensation before VS phase
for (int ii = 0; ii < Q_ROWS; ii++) a_tile[ii] *= delta;  // All at once
for (int kv = 0; kv < KV_PER_SG; kv++) {
    acc = dpas(acc, s_tile, v_tile);  // Clean DPAS pipeline
}

7. Type Conversion Before Data Movement

When both type conversion and data shuffling are needed, do conversion first to reduce the data volume for the shuffle.

// BAD: transpose fp32 data (4 bytes per element), then convert
// Moves 2x the data during transpose

// GOOD: convert fp32 -> fp16 first, then transpose fp16 data (+14% measured)
simd<half, N> fp16_data = convert<half>(fp32_data);
// Now transpose fp16_data (half the register traffic)

---

Optimization Priority Checklist

Apply optimizations in this priority order (highest impact first):

1. Eliminate register spill — Use doubleGRF, reduce tile sizes, reorder operations
2. Maximize DPAS density — Consecutive DPAS instructions, pipeline loads
3. Minimize data movement — SLM transpose instead of register transpose, type convert before shuffle
4. Optimize memory access — block_load over gather, 2D loads for tiled access, prefetch
5. Reduce barrier count — Named barriers, split arrive/wait, eliminate redundant barriers
6. Fine-tune arithmetic — SIMD reciprocal, mad fusion, vectorized exp
7. Tune tile sizes — Balance occupancy vs work per thread vs register pressure

---

Case Study: SDP HD=256 Optimization Journey

The HD=256 flash attention kernel on BMG illustrates the methodology:

Key takeaways:

• The two biggest wins were data movement optimizations (FP16 transpose, SLM scatter)
• Arithmetic micro-optimizations gave small but cumulative gains
• Several approaches that seemed promising (double-buffered SLM, early V loads, VNNI interleave) were neutral or regressive
• Always measure; intuition about GPU performance is often wrong

---

Related Skills