ncu-report

Nsight Compute Report Analysis

This skill handles two modes:

1. Analyze an existing report: The user provides a path to an .ncu-rep file (or one exists under examples/). Use the ncu CLI to extract and present metrics.

2. Generate a new report: The user specifies a script or kernel to profile but does NOT provide a .ncu-rep file. In this case, set up profiling using tilus.utils.ncu_utils.ncu_run(), run it, then analyze the resulting report.

Generating a Report

Tilus provides tilus.utils.ncu_utils.ncu_run() to profile kernels with full metrics and source correlation.

API

from tilus.utils.ncu_utils import ncu_run

# ncu_run(func, *args, kernel_regex=".*", **kwargs) -> NsightComputeReport
report = ncu_run(main, bench=False, kernel_regex="tilus|nvjet")

• func: a callable (typically a main() function) that runs the kernel(s) to profile
• *args, **kwargs: passed through to func
• kernel_regex: regex to filter which kernels to profile (default ".*")
• Returns NsightComputeReport with .report_path pointing to the generated .ncu-rep file

What it does

• Runs the function under ncu with --set full, all rules enabled, and --import-source yes
• Saves the report to ncu-reports/reportN.ncu-rep next to the script (auto-increments N)
• Uses the system Python and the ncu binary at /usr/local/cuda/bin/ncu

How to generate a report for the user

IMPORTANT: ncu_run() must be called inside a if __name__ == "__main__": block. It works by re-importing the script as a subprocess under ncu — if ncu_run() is at module level, the subprocess will call ncu_run() again, causing infinite recursion and a runtime error.

Step 1: Read the script — find the example script the user specified and read it to understand how the kernel is invoked.

Step 2: Set up profiling — choose one of these approaches:

• If the script already has a __main__ block with ncu_run(): just run it directly.
• If the script has a __main__ block but no ncu_run(): edit the __main__ block to add ncu_run(). For example, if the block calls main(), change it to call ncu_run(main, bench=False, kernel_regex="tilus").
• If the script has no __main__ block or is hard to modify (e.g., it's a test file, or the kernel launch is deeply nested): create a new script next to it (e.g., profile_<name>.py) that imports and calls the kernel under ncu_run().

Example of editing an existing __main__ block:

if __name__ == "__main__":
    from tilus.utils.ncu_utils import ncu_run
    ncu_run(main, bench=False, kernel_regex="tilus")

Example of creating a new profiling script:

from tilus.utils.ncu_utils import ncu_run
from matmul_v9 import main

if __name__ == "__main__":
    ncu_run(main, bench=False, kernel_regex="tilus")

Step 3: Run the script — python <script_path>. The report will be saved to <script_dir>/ncu-reports/reportN.ncu-rep.

Step 4: Analyze — proceed to the Analysis Workflow below with the generated report.

Note: ncu profiling requires sudo or appropriate permissions (CAP_SYS_ADMIN). If the command fails with permission errors, suggest running with sudo.

Analysis Workflow

Follow this sequence. Skip steps the user doesn't need, but always start with Step 1.

Step 1: Overview — List kernels and session info

Run these in parallel:

# List all kernels with timing
ncu -i <REPORT> --page raw --csv --metrics gpu__time_duration.sum 2>&1

# Session/device info
ncu -i <REPORT> --page session --csv 2>&1

Present a summary table:

• Kernel name (shortened), Block Size, Grid Size, Duration (ms)
• Device name, compute capability, CUDA version

Step 2: Speed of Light — Top-level throughput

ncu -i <REPORT> --page details --csv --section SpeedOfLight 2>&1

Key metrics to highlight per kernel:

• Duration (ms)
• Compute (SM) Throughput (%) — how busy the SMs are
• Memory Throughput (%) — overall memory utilization
• DRAM Throughput (%) — HBM bandwidth utilization
• L1/TEX Cache Throughput (%)
• L2 Cache Throughput (%)
• SOLBottleneck rule — check the Rule Description column for bottleneck guidance

Step 3: Compute & Memory Workload Analysis

# Compute workload
ncu -i <REPORT> --page details --csv --section ComputeWorkloadAnalysis 2>&1

# Memory workload
ncu -i <REPORT> --page details --csv --section MemoryWorkloadAnalysis 2>&1

Key compute metrics: Executed IPC Active, SM Busy %, Issue Slots Busy % Key memory metrics: Mem Busy %, Max Bandwidth %, L1/L2 hit rates

Step 4: Occupancy

ncu -i <REPORT> --page details --csv --section Occupancy 2>&1

Report: Theoretical Occupancy, Achieved Occupancy, and limiters (registers, shared memory, block size).

Step 5: Detailed metrics (on demand)

To extract specific raw metrics:

ncu -i <REPORT> --page raw --csv --metrics <metric1>,<metric2>,... 2>&1

To filter by kernel:

ncu -i <REPORT> --page raw --csv --metrics <metrics> --kernel-name regex:<pattern> 2>&1

Step 6: Source-level analysis (on demand)

SASS-only (default, always available):

ncu -i <REPORT> --page source --csv --kernel-name regex:<pattern> 2>&1

CUDA source correlated with SASS (requires --import-source yes during profiling):

ncu -i <REPORT> --page source --csv --print-source cuda,sass --kernel-name regex:<pattern> 2>&1

Source output columns include per-instruction: Warp Stall Sampling, Instructions Executed, Thread Instructions Executed, stall reasons (stall_barrier, stall_math, stall_wait, etc.), shared memory conflicts, and more.

Step 7: Rules / automated analysis (on demand)

Rules are included in the details page output. Look for non-empty "Rule Name" column entries.

ncu -i <REPORT> --page details --csv --print-rule-details 2>&1 | grep -v '^"[0-9]' | head -5  # header

To see all rule results with descriptions:

ncu -i <REPORT> --page details --csv --print-rule-details 2>&1

Filter for rows where column 17 (Rule Name) is non-empty.

Reference: ncu CLI Options for Report Analysis

Pages (--page)

Key Flags

Section Identifiers

Section Sets (used during profiling with --set)

Commonly Used Raw Metrics

Available Rules (used during profiling with --rule)

Comparing Kernels

When the report contains multiple kernels (e.g., a reference nvjet kernel and a tilus kernel), always present metrics side-by-side for comparison. Highlight:

1. Duration difference (which is faster, by how much)
2. Throughput differences (compute vs memory bound)
3. Occupancy differences
4. Any rule findings that differ

Tips

• The raw page has one row per kernel with all metrics as columns — good for extracting specific values.
• The details page organizes metrics by section — good for browsing all metrics in a section.
• The source page is per-instruction — good for hotspot analysis. Output can be very large; pipe through head or filter with grep.
• Use --print-units base with --csv for consistent numeric parsing.
• Use --print-metric-name name to get programmatic metric names instead of human labels.
• Source analysis with cuda,sass view shows CUDA source lines interleaved with their SASS instructions — extremely useful for correlating high-level code with assembly hotspots.