Table of contents

Adding TorchRecord / NVTX markers

This documentation outlines the process of adding Torch Records and NVTX markers to your code using the nCompass SDK and VS Code extension. It provides a step-by-step guide for runtime injection without modifying the original codebase.

Why Runtime Injection 0:00

• This feature allows you to add Torch Records or NVTX markers without editing your code directly.
• It is part of the open-source nCompass SDK, available on GitHub, and works in conjunction with the VS Code / Cursor extension.
• Future expansions will support various types of debug code that should not remain in the codebase permanently.

Use Cases 0:39

• When debugging performance, you may want to highlight specific regions in your profile.
• Libraries like vLLM support adding profiling markers by default in the code base, but often you need to manually add context statements.
• This can lead to messy code and the risk of leaving debug statements in production branches.

Step 1: Add regions in the extension 2:05

1. Install the nCompass VS Code extension.
2. Highlight the code region you want to profile.
3. Click on the light bulb or (Ctrl/Cmd + '.') and select 'Add Region to Profile'.
4. Choose 'Torch Record' to annotate the selected region.
5. A configuration file (config.json) will be created in the .cache directory, storing the injection locations.

Step 2: Use our SDK 3:07

• Ensure you have the nCompass pip package installed (version > 0.1.12).
• Set the following environment variables:
  • NCOMPASS_CACHE_DIR: Path to the .cache directory (usually the current directory).
  • NCOMPASS_PROFILER_TYPE: Set to 'Torch' or 'NVTX' depending on your profiling needs.
• Run your main application to execute the profiler with the injected markers.

Step 3: Viewing the Results 4:02

• After running your application, a trace will be generated.
• Open the trace in nCompass to see the annotations, which will indicate the lines where the markers were added.
• This process works for both Torch Records and NVTX markers.

Contributions and Future Development 4:19

• Contributions to the open-source SDK are welcome for adding different types of injectors, such as timers or profiling control regions.
• The team plans to develop additional features to enhance debugging capabilities.

https://www.loom.com/share/1e526ff43fa44bfcaebe18a6f56052ff