NVIDIA NemoClaw Turns Video Analysis Into Automated Workflow Actions

NVIDIA introduced NVIDIA NemoClaw, a collection of open blueprints for building autonomous agents, to orchestrate the Video Search and Summarization (VSS) and Retrieval-Augmented Generation (RAG) blueprints into a composable service. The integration moves video analysis beyond answering "What does this video show?" to "What should we do about it," enabling programmatic actions such as ticket creation, pattern comparison, procedure drafting, and anomaly escalation.

How the agent pipeline works

The VSS agent uses three tools working together. The long video summary (LVS) tool performs summarization with mandatory human-in-the-loop (HITL) parameter collection, where users specify the scenario, events of interest, objects of focus, and an optional knowledge-retrieval query. The knowledge retrieval tool calls the RAG Blueprint to pull organization-specific context from documents, policies, and reference data, with the blueprint handling embedding, reranking, and vector search internally. The report generation tool produces a structured, timestamped report combining video analysis with retrieved context, including citations and recommended actions, with optional HITL confirmation before output.

In a demonstration, a healthy eating coach analyzes a meal preparation video. NemoClaw reads a skill definition to learn required parameters, then walks the user through HITL prompts in a terminal asking what to analyze, the scenario, events of interest, objects to track, and an optional RAG query for nutritional guidelines. For automated batch runs, these answers can be supplied programmatically. The LVS tool queries the RAG Blueprint first, then passes parameters, video, and retrieved context to the LVS service, which summarizes hierarchically and weaves reference knowledge into findings. The report generation tool combines results into a structured report with timestamps, narrative analysis, citations, and recommended actions.

NemoClaw then reads the finished report and creates a Jira ticket automatically, summarizing findings and recommended dietary adjustments with priority and assignment so action items are tracked to completion. The downstream step generalizes beyond Jira: NemoClaw can create tickets with appropriate priority, escalate patterns across multiple runs, bundle evidence for compliance, or route gaps to follow-up workflows.

Architecture and deployment

The architecture spans four layers. The orchestration layer includes the NemoClaw agent, the skill definition, and HITL prompts. The VSS agent layer contains tools for video I/O, search, understanding, LVS, knowledge retrieval, and report generation, with knowledge retrieval as part of the agent. The RAG Blueprint layer includes the NVIDIA RAG API, Milvus vector database, NVIDIA Nemotron reranking NIM, and indexed reference documents. The LLM fusion layer enriches the VSS-provided summary with retrieved context. Data flows downward, with agent tools orchestrating calls to LVS and RAG, both feeding the report generation tool.

Deployment requires NVIDIA GPUs with at least 24 GB VRAM, Docker Engine with Docker Compose v2, NGC API key, NVIDIA Build API key, a deployed and reachable RAG Blueprint with its collection name, and NemoClaw installed. Steps include cloning the VSS repository, authenticating with NGC, and configuring the LVS profile environment file, which contains all variables except RAG values that must be added manually. The hardware profile supports H100, L40S, RTXPRO4500BW, RTXPRO6000BW, DGX-SPARK, IGX-THOR, AGX-THOR, or OTHER, with options for local shared or separate GPU placement for LLM and VLM.

Partner deployments show the pipeline reducing footage-to-work-order time from 30 to 45 minutes to roughly 19 seconds for predictive maintenance, and enabling real-time video analytics on the VAST DataEngine. The integrated solution delivers faster response times, consistent application of reference knowledge, full traceability from evidence to decision, and automation of routine workflows.