NVIDIA walks through six-step CUDA optimization from 6.8 s to 23 ms

NVIDIA published a step-by-step optimization walkthrough on September 2, 2026, showing how a CUDA image processing pipeline went from 6.8 seconds to 23 milliseconds through six incremental changes. The example pipeline takes red, green, and blue images, converts them to grayscale, computes the median of each 32 by 32 pixel tile, and copies the medians back to the CPU. The original code ran three images in parallel with OpenMP, allocating and freeing GPU memory for each image inside the loop.

The first step uses Compute Sanitizer, NVIDIA's functional correctness checking suite, to find an illegal memory access. The tool reports an invalid shared memory write in the computeMedian kernel at the line that loads tile data. The bug comes from using a global index to address shared memory, which is defined at the thread block level. NVIDIA's solution is the modern CCCL launch API, which distinguishes global and block-level coordinates. The new cuda::launch API takes a configuration built with cuda::make_config, and inside the kernel, cuda::gpu_thread.index with the grid or block argument returns the correct coordinates. The post also mentions cuda::std::span and cuda::std::mdspan as non-owning views that abstract away the container type and help avoid indexing mistakes.

Replacing hand-written kernels with CUB primitives

The second step adds NVTX annotations to improve Nsight Systems benchmarks, making it easier to see where time goes in the pipeline. The third step replaces the hand-written kernels with CUB device and block-level primitives. The original computeMedian kernel used a single-threaded bubble sort in shared memory, which is inefficient. Using CUB's optimized algorithms at the block and device level reduced the median computation from 2.1 seconds to 773 microseconds per image, a 2,717x speedup. The RGB to grayscale conversion also benefits from CUB device-level operations.

Pooled memory and pinned host buffers

The fourth step addresses allocation overhead. The original code called cudaMalloc and cudaFree for every image in the OpenMP loop. CCCL pooled cuda::device_buffer containers backed by cuda::device_memory_pool_ref reuse memory across allocations, cutting total pipeline time by 2.6x. The fifth step moves host data into cuda::host_buffer pinned memory, which accelerates host-to-device transfers 10x. With pinned buffers, the full three-image run dropped to about 25 milliseconds.

The final step parallelizes GPU work across OpenMP threads. Each thread gets its own cuda::stream and uses asynchronous cuda::copy_bytes, which overlaps kernels and transfers across streams. This achieves a final runtime of 23 milliseconds, 300x faster than the original 6.8 seconds. The post includes code examples, a Google Colab notebook, and a YouTube class on the CUDA Developer's Toolbox. For inference engineers, the key takeaway is that small, targeted changes, each verified with the right tool, can produce dramatic end-to-end improvements without rewriting the entire pipeline.