DeepStream 9.1 Adds Auto-Calibration and Multi-Camera 3D Tracking Skills

NVIDIA DeepStream 9.1 adds two components that let developers track objects across multiple camera views in a shared 3D coordinate system: AutoMagicCalib (AMC) and Multi-View 3D Tracking (MV3DT). The release also includes 13 agentic skills for coding agents such as Claude Code and Codex, plus support for NVIDIA JetPack 7.2 on Jetson edge platforms. The full source code is available through the unified DeepStream GitHub repository.

How MV3DT works

MV3DT extends the DeepStream tracker to handle distributed multi-view 3D tracking across a network of calibrated cameras. Each camera independently projects detections into 3D world space using a 3x4 projection matrix stored in a YAML calibration file, assuming a ground plane. The system then associates observations of the same object across views and assigns one globally consistent object ID.

The tracker supports three detector models out of the box: PeopleNetTransformer, a transformer-based people detector for pedestrian scenes; PeopleNet v2.6.3, a high-efficiency detector based on the DetectNet_v2 architecture; and RT-DETR 2D, a multi-class detector for industrial environments that identifies pedestrians, transporters, and forklifts.

Multi-view association uses MQTT, a lightweight pub/sub messaging protocol, to share tracklets between cameras. When two cameras observe the same person, the algorithm matches tracklets based on proximity in 3D world space and assigns a single ID. Tracking results stream out in three forms: an on-screen display with a live grid of camera feeds and overlaid 2D/3D bounding boxes, a bird's-eye view map showing object trajectories in world coordinates, and Kafka messaging that delivers structured protobuf metadata per frame including sensor ID, object ID, and 3D bounding box.

AutoMagicCalib for camera calibration

MV3DT requires calibrated cameras, and traditional calibration methods involve placing checkerboard patterns in front of cameras, which interrupts operations. AMC automates this by analyzing tracked objects moving across existing video files or streams. It estimates each camera's intrinsic parameters (focal length, principal point, lens distortion) and extrinsic parameters (rotation, translation, world position) to produce calibration files.

The calibration pipeline runs through several stages: per-camera trajectory extraction using DeepStream detection and tracking, single-view calibration and rectification, multi-view tracklet matching using manually provided alignment points as anchors, and bundle adjustment to refine all camera parameters jointly across every view. Users can optionally run VGGT, a model-based calibration workflow useful when object movement is limited. AMC is available as a microservice with REST APIs and a web interface.

Agentic skills for deployment

DeepStream 9.1 introduces modular skills designed for coding agents. Instead of manually running scripts and editing configuration files, developers describe what they want in natural language prompts, and the agent handles setup, configuration, and execution. The MV3DT skill manages the full deployment lifecycle: validating prerequisites, pulling or building the DeepStream container, installing Kafka and Mosquitto broker services, downloading detection model weights, generating pipeline configuration, triggering AMC skills if calibration files are missing, and launching the tracking pipeline.

Four AMC skills handle calibration tasks: amc-setup-calibration-stack pulls the AMC microservice container and starts the service stack, amc-run-sample-calibration runs end-to-end calibration on a bundled sample dataset, amc-run-video-calibration calibrates a new dataset of video files with a layout image and alignment points, and amc-run-rtsp-calibration calibrates directly from RTSP streams.

Prerequisites for deployment include Ubuntu 24.04 on x86_64, NVIDIA driver version 580 or higher, Docker with NVIDIA Container Toolkit, an authenticated coding agent, an NGC API key for pulling containers, and a Hugging Face key for the VGGT model. Setup involves cloning the DeepStream repository, copying skills into the agent's skill directory, and launching the agent. The underlying multi-view association algorithms are detailed in the research paper "Fully Distributed Multi-View 3D Tracking in Real-Time."