NVIDIA cuDF Skills Rewrite Spark UDFs for GPU, 22x Speedup in Production

NVIDIA has released cuDF skills for Apache Spark, a set of agent-driven workflows that rewrite Spark user-defined functions (UDFs) into GPU-accelerated equivalents. The skills target the common bottleneck where UDFs force otherwise GPU-resident pipelines to fall back to CPU execution, incurring data transfer overhead and losing compute speedup.

The rewrite process is broken into four feedback loops: testing, conversion, benchmarking, and optimization. Each skill defines the work and the evidence required before moving to the next stage. The skills provide a project template with test and benchmark harnesses, output verification routines, and build configuration, so the agent can focus on the UDF rather than reconstructing infrastructure.

Three rewrite approaches are outlined. The simplest is replacing a UDF with an equivalent Spark SQL expression, which keeps the query fully on GPU. For more complex logic, the RapidsUDF interface lets code operate directly on GPU columns via the Java API for cuDF. For the most complex cases, a RapidsUDF can invoke custom CUDA kernels through the Java Native Interface.

NVIDIA reports that a major consumer technology company used the workflow to convert more than 30 production UDFs. One rewrite reduced operator time from 274.55 to 12.78 hours, a 22x speedup across a 600-billion-row workload. That UDF was affecting over 3,000 jobs.

The skills are available on GitHub and can be installed with the npx skill installer. NVIDIA also points to a Data + AI Summit session on coding AI agents for Spark UDF GPU acceleration and a getting-started guide for the cuDF plugin.

For inference engineers and edge AI developers, the practical takeaway is that agent-assisted rewrites can eliminate the CPU fallback penalty in Spark pipelines, but the gains depend on the workload and the rewrite approach chosen. The 22x figure is a vendor-reported result from one production job, not a general benchmark.