Table of contents

Visualizing Diffs between Traces

Accessing TraceDiffs

• Open the NCProf Analyzer view.
• Select the 'TraceDiff' option from the top right to access the interface.

Selecting Traces for Comparison

• Choose a base trace (e.g., version 0.12.0 VLLM).
• Select a comparison trace (e.g., version 0.13.0 VLLM).

Running TraceDiff

• Click 'Run TraceDiff' to generate the trace file.
• After generation, click 'Open TraceDiff' to load the results in the Perfetov viewer.

Understanding the TraceDiff Viewer

• The top streams represent the base trace
• Then we display the diff parameters
• At the bottom, there is a stream labelled "Comparison" which is the comparison trace
• This makes it easy to see overall performance difference in one timeline

Analyzing Diff Categories

• Review the five diff categories: Duration, Args, Name, Added, and Removed.

Duration Diffs

• Click on the duration diff to see performance changes in individual kernels.
• Flow events (linking arrows) in the trace show which base and comparison kernels the diff links to

Argument Diffs

• Click on the args diff to view changes in launch arguments between kernels.
• Note any significant changes in resource usage (e.g., registers per thread, shared memory usage).
• These changes show up in the "Current Selection" pane below in Perfetto

Reviewing Renamed, Added and Remove kernels

• Check for any renamed kernels, new kernels added and removed kernels
• We're refining this view, but the combination of these 3 categories is particularly useful in identifying fused kernels

https://www.loom.com/share/90b7707271b04ed09f61202b57202cd7