NVIDIA Builds Endoscopic Tool Segmentation App With CLI and AI Coding Agent

NVIDIA engineers used a general-purpose coding agent to build a real-time endoscopic tool segmentation application on the Holoscan platform, guided by the Holoscan CLI and HoloHub development skills. The work, published on Aug. 19, 2026, describes an iterative workflow where the agent reused an existing MONAI model and sample video, added visualization and benchmarking, then optimized latency. The final version improved rendered throughput by 50.5% and reduced mean application-path latency by 33.6% compared with the first working implementation.

Iterative development with CLI and skills

The workflow split the goal into reviewable iterations. The first prompt asked the agent to create a minimal working application that reused the MONAI endoscopic tool segmentation model, sample data, preprocessing, and inference, without modifying model weights. The agent read the app lifecycle skill, nearby HoloHub examples, project metadata, and CLI documentation, then used the holohub wrapper commands to scaffold, run, and test the application. The result connected video replay, preprocessing, TensorRT inference, the SDK segmentation postprocessor, telemetry, and HoloViz, with inference and mask postprocessing running for every replayed frame.

The second iteration added a benchmark mode. The agent revised dynamic measurements and screenshot readability, then implemented three named application modes: visual, which runs the full sample at source pace in an interactive window; smoke, a quick 60-frame headless recording with a finite verdict; and benchmark, which processes 300 frames offscreen and exports measurements and plots. Benchmark mode used Holoscan Data Flow Tracking for the path from the video replayer through preprocessing, inference, telemetry, offscreen HoloViz, and the rendered-frame sink.

Latency optimization and ablation study

The third iteration investigated latency. The agent confirmed inference still ran on every frame, considered reusing masks across neighboring frames but kept per-frame inference for this iteration, and removed lower-risk dashboard overhead. It reused HoloViz input specifications and static coordinate tensors while refreshing text and dynamic geometry each frame, and queued the 10-value GPU-to-host telemetry copy asynchronously in two pinned buffers, rendering with the previous completed values. On the same test system, the optimized version was faster in all five measured trials.

NVIDIA also ran an ablation study comparing resource usage with the same coding agent and sandbox environment across identical settings, using the first iteration prompt. The study, last conducted on Aug. 1 with Codex 0.146.0 and GPT-5.6 Sol at max reasoning effort, showed that combining the CLI, HoloHub development skills, and documentation yielded the fastest agentic processing time of 40 minutes and lowest token usage of 11 million, compared with configurations without skills or CLI guidance.

For inference engineers building edge AI applications, the post demonstrates a practical pattern: an engineer defines goals and constraints, the agent inspects examples and runs CLI operations, and the engineer reviews the same artifacts the agent produces. The CLI makes modes and tests discoverable and runnable without remembering detailed container and application script recipes, and the workflow produces repeatable measurements that inform each next iteration.