Detailed Profiling Workflow

This page provides more details on:

• Registering tracepoints to profile through vscode
• Integrating those injected tracepoints when profiling using our SDK
• Viewing the resulting traces.

Overview

The ncprof workflow consists of three main phases:

1. Create Tracepoint: Select code in VS Code and register tracepoints for profiling
2. Instrument & Profile: Use the ncompass SDK to inject profiling markers and capture traces using your favorite profiler
3. Analyze: View traces in the integrated trace viewer and navigate to source code

Let’s explore each phase in detail.

Registering Tracepoints

Why Register Tracepoints?

When you register a tracepoint in ncprof, the backend generates a configuration that tells the ncompass SDK exactly where to inject profiling markers. This allows you to focus profiling on specific functions or code blocks without manually modifying your source code.

Profiling Marker Injections:

The ncompass SDK uses context managers to inject profiling markers around your tracepoints. These are implemented as Python context managers in the SDK:

• Currently shipped: TorchRecordContext - wraps code with torch.profiler.record_function() for PyTorch profiling
• Coming soon:
  • NvtxContext - NVTX range markers for CUDA profiling
  • Custom injector support - ability to define your own profiling markers and contribute them to the SDK

You can see examples of how these injectors work in the ncompass SDK source code :

• Base interface: ncompass/trace/profile/base.py (ProfileContextBase)
• Torch implementation: ncompass/trace/profile/torch.py (TorchRecordContext)
• NVTX implementation: ncompass/trace/profile/nvtx.py (NvtxContext)

How to Register Tracepoints

1. Open a Python file in your workspace
2. Select the code you want to profile

Once you’ve selected code, you can register it using any of these methods:

• Lightbulb Quick Action (Recommended): Click the lightbulb (💡) icon or press Ctrl+. / Cmd+. → Select ”▶ Add Region to Profile”
• Context Menu: Right-click → “Add Region to Profile”
• Keyboard Shortcut: Ctrl+Alt+R (Windows/Linux) or Cmd+Alt+R (Mac)
• Command Palette: Ctrl+Shift+P / Cmd+Shift+P → Type “NCProf: Add Region to Profile”

3. After registering you’ll see:
   • The tracepoint appears in the Profile Regions view in the Explorer sidebar
     • Organized by file: Tracepoints are grouped under their source files
     • Click to navigate: Click any tracepoint to jump to its location in the code
     • Refresh button: Update the view manually if needed (visible in the view title bar)
     • Tracepoint details: Shows function name and line range
   • Visual decorations in the editor gutter

Removing Tracepoints

You can remove specific registered tracepoints using:

• Profile Regions View: Hover over a tracepoint → Click on the “Remove Profile Region” button
• Command Palette: Place cursor in the selected tracepoint → Ctrl+Shift+P / Cmd+Shift+P → Type NCProf: Remove Profile Region

To remove all registered tracepoints:

1. Press Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (Mac)
2. Type NCProf: Clear All Profiler Registrations
3. Confirm the action

Feature Tutorial: TorchRecord/NVTX Contexts

Watch our Feature Tutorial: TorchRecord/NVTX contexts without editing your code  to learn how to use tracepoints with TorchRecord and NVTX contexts.

For example code, check out our examples repository .

Working with Profiles

ncprof supports organizing tracepoints into different profiles.

Profile Settings:

• ncprof.profiler.profileName: Name of the profile (default: .default)
• ncprof.profiler.profileType: Type of the profile (default: .default)

Use cases for multiple profiles:

• Separate profiles for different modules
• Development vs. production profiling configurations
• Team members working on different parts of the codebase

Changing profiles:

1. Open VS Code Settings (Ctrl+,)
2. Search for “ncprof.profiler”
3. Update profileName and/or profileType

Instrumenting and Profiling

Once you’ve registered tracepoints, use the ncompass SDK to profile your system with the recorded tracepoints.

Understanding the Generated Configuration

When you register tracepoints, ncprof creates a config.json file at:

<workspace>/.cache/ncompass/profiles/<profileName>/<profileType>/current/config.json

For the default profile, this is:

<workspace>/.cache/ncompass/profiles/.default/.default/current/config.json

Configuration Structure:

{
  "targets": {
    "your_module": {
      "func_line_range_wrappings": [
        {
          "function": "your_function",
          "start_line": 10,
          "end_line": 25,
          "context_class": "ncompass.trace.profile.torch.TorchRecordContext",
          "context_values": [
            {
              "name": "name",
              "value": "user_annotated: your_function:10-25",
              "type": "literal"
            }
          ],
          "reasoning": "Profile this function for performance analysis",
          "start_line_content": "def your_function():",
          "end_line_content": "    return result"
        }
      ]
    }
  },
  "ai_analysis_targets": [],
  "full_trace_mode": false
}

Key Configuration Fields:

• targets: Map of module names to their configurations
  • func_line_range_wrappings: List of line ranges to wrap with profiling context managers
    • function: Function/method name to target (function in which the selected lines are)
    • start_line / end_line: Line range to wrap (inclusive)
    • context_class: Full path to the context manager class (e.g., ncompass.trace.profile.torch.TorchRecordContext)
    • context_values: List of arguments to pass to the context manager constructor
      • name: Argument name
      • value: Value to pass
      • type: Either "literal" (string constant) or "variable" (variable reference)
    • reasoning: (Optional) Explanation for this wrapping
    • start_line_content / end_line_content: (Optional) Line content for validation
• ai_analysis_targets: List of modules for AI profile analysis (coming soon)
• full_trace_mode: Enable comprehensive trace capture for AI analysis (coming soon)

Basic Profiling with enable_rewrites()

This is an example of how you would use the registered tracepoints in your profiling script to add specific marked tracepoints.

from ncompass.trace.core.rewrite import enable_rewrites
from ncompass.trace.core.pydantic import RewriteConfig
import json
 
# Load the configuration generated by ncprof
config_path = ".cache/ncompass/profiles/.default/.default/current/config.json"
with open(config_path) as f:
    config = json.load(f)
 
# Enable instrumentation
enable_rewrites(config=RewriteConfig.from_dict(config))
 
# Now import and run your code
# The registered tracepoints will automatically have profiling markers
from your_module import your_function
 
# Run with PyTorch profiler
import torch
with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA,
    ],
    with_stack=True,
) as prof:
    your_function()
 
# Export the trace
prof.export_chrome_trace("trace.json")

Key points:

• Call enable_rewrites() before importing your code or libraries that you want to profile
• The ncompass SDK automatically injects profiling markers at your registered tracepoints without editing the codebase
• Use PyTorch profiler (or another profiler) to capture the trace
• Export in Chrome trace format (.json) => this is the interface that the profile viewer uses.
  • Support for .nsys-rep files is coming soon.

Feature Tutorial: Profiling Systems Remotely

Watch our Feature Tutorial: Profiling Systems Remotely using Modal and the nCompass SDK  to learn how to profile remote systems.

For example code, check out our examples repository .

Advanced: Using ProfilingSession

For more control and AI-powered analysis.

from ncompass.trace import ProfilingSession
 
# Initialize a profiling session
session = ProfilingSession(
    trace_output_dir="./traces",
    cache_dir="./.cache/ncompass",
    session_name="my_model"
)
 
# Define your code to profile
def code_to_run():
    from your_module import your_function
    your_function()
 
# Run profiling (automatically uses your registered tracepoints)
trace_file = session.run_profile(
    user_code=code_to_run,
    trace_name_suffix="initial"
)
 
print(f"Trace saved to: {trace_file}")
 
# Get AI-powered trace summary
summary = session.get_trace_summary(trace_path=trace_file)
print(summary['markdown'])

ProfilingSession features:

• Automatic trace naming and organization
• Built-in AI-powered trace analysis
• Iterative profiling workflow support
• Feedback-driven marker generation

Trace Output

Saved location:

• Basic profiling: Where you specify in export_chrome_trace() (typically ./traces/)
• ProfilingSession: Auto-saved to trace_output_dir as {session_name}_{suffix}_{timestamp}.pt.trace.json

Supported formats: Chrome trace format - .json (plain) or .json.gz (auto-extracted)

Viewing and Analyzing Traces

Opening Trace Files

Open trace files in VS Code using:

• Context Menu (Recommended): Right-click the .json or .json.gz file → “Open With…” → “GPU Trace Viewer”
• Command Palette: Ctrl+Shift+P / Cmd+Shift+P → Type NCProf: Open GPU Trace → Select file

This opens it in Perfetto, an industry-standard trace viewer.

Key features:

• Timeline showing CPU/GPU activity, process/thread tracks, and event details
• Navigation: WASD keys (zoom/pan), mouse wheel (zoom), click-drag (pan), arrow keys (navigate events)
• Search: Ctrl+F to find events, M to bookmark

Events created from your registered tracepoints will have names like:

user_annotated: your_function:10-25

These markers make it easy to identify the tracepoints you’re profiling.

Navigating to Source Code

If trace events contain file paths from your workspace:

1. Click on an event in the trace
2. Look for file/line information in the details panel
3. If available, you’ll see an option to navigate to the code
4. Click to jump directly to the relevant source line

Analyzing Traces with AI

If you’re using ProfilingSession, you can get AI-powered insights:

from ncompass.trace import ProfilingSession
 
session = ProfilingSession(
    trace_output_dir="./traces",
    cache_dir="./.cache/ncompass"
)
 
# Get summary for a trace
summary = session.get_trace_summary(
    trace_path="./traces/my_trace.json"
)
 
# View markdown report
print(summary['markdown'])
 
# Access structured data
print(f"Total duration: {summary['structured']['total_duration_ms']}ms")

Next Steps

• Managing the Backend: Learn how to control the Docker infrastructure
• Settings & Commands: Explore all available configuration options