NVIDIA Nemotron 3 Ultra LangChain Harness Profile Fixes Pagination, Boosts Benchmark

NVIDIA published a tutorial on July 8, 2026, showing how to create a LangChain Deep Agents harness profile for the Nemotron 3 Ultra model. The profile adds a middleware component that fixes a pagination issue in the built-in read_file tool. With the profile, the evaluation benchmark score improves from 94 to 96 out of 127, and all three failing read_file pagination tests pass.

The read_file tool is used in coding, data analysis, and agentic tasks. In the failing test, the agent reads a large file and stops after the first page, which contains only x values. The correct answer, opal-fox-91, appears later in the file. The model needed to continue reading with offset and limit pagination but answered from the first page.

Harness Profile Fix

The fix is a middleware class called ReadFileContinuationNoticeMiddleware. It wraps tool calls and checks if a read_file result probably continues. When the result contains more lines than the limit, it adds a notice to the tool message. The profile registers this middleware under the name nvidia:nemotron-ultra.

The tutorial emphasizes that harness engineering aims to make agent-to-model calls resemble what the model saw in training data. Changes to the harness profile can include prompt modifications, tool exclusions, or additions like middleware that checks for truncated responses. The goal is to improve accuracy without fine-tuning, which requires expertise and hardware for training and hosting custom models.

Automated Improvement Loop

The manual procedure is a loop: run the benchmark, analyze failures, propose a profile change, and re-run to verify. The tutorial describes automating this loop with a reference implementation in the NemoClaw community repository. The loop uses a model to propose changes, verifies fixes across multiple runs, and re-runs the full suite to prevent regressions.

Key decisions in the loop include verifying before believing, since model calls and LLM-judged benchmarks are stochastic. A fix is accepted only after passing the same test several times in a row. The profile is snapshotted before each attempt and restored if the attempt fails. When an attempt fails, the loop provides the next attempt with what was tried and how it failed. After any fix lands, the entire benchmark is re-run to catch regressions.

The agentic proposer runs as a short, bounded agent session. It gets a writable copy of the profile and read-only access to the failing test, trajectory, and code. Write access is scoped to the profile file only. The agent is told to prefer general fixes over specific hacks, distinguishing between teaching the model a durable behavior and overfitting to a single test.

The tutorial notes that overfitting is a concern. The file pagination middleware is a mechanical fix that teaches the model appropriate tool calls. In contrast, adding definitions of terms like alert or triage to the harness could more likely cause overfitting. The same propose-verify-keep loop can adapt Nemotron 3 Ultra to other agent frameworks by exposing evaluation results, an editable profile file, and a frontier model to propose edits.

LangSmith Engine is LangChain's managed version of this loop for harness improvement. It detects failures in production traces, diagnoses root causes, drafts fixes, and adds evaluators to prevent regressions. Each proposed change is reviewed by a human before shipping. The tutorial recommends reviewing the Nemotron 3 Ultra agent harness profile published by the LangChain team and testing the NVIDIA NemoClaw Blueprint for LangChain Deep Agents.