Self-Host a Guarded AI Coding Assistant with NeMo Guardrails

NVIDIA published a guide for self-hosting a validated AI coding assistant using StarCoder2-7B NIM and NeMo Guardrails. The setup keeps source code on-premises while adding a policy layer that blocks sensitive requests before the model runs. The guardrails proxy sits between the IDE and the model, checking each prompt against a policy that can forbid generation of authentication, payment, or cryptography code, as well as files under directories like src/security/, src/auth/, or src/payments/. The guardrails server runs on port 8100, and the IDE points to http://localhost:8100/v1 instead of the model's direct endpoint.

Deployment and Hardware

Prerequisites include an NGC API key, an NVIDIA GPU with at least 24 GB of memory (A10, L4, L40S, or A100), Docker with the NVIDIA Container Toolkit, Python 3.10 or later, and a Git repository. StarCoder2-7B runs in BF16 precision. H100 and H200 GPUs offer the highest throughput but are not required for pilots. Deploy the NIM via a Docker run command, pinning a specific version from the NGC catalog. The health endpoint is http://localhost:8000/v1/health/ready, and the completions endpoint is http://localhost:8000/v1/completions.

For a pilot, run the NIM on a single shared GPU host and restrict access to one team. For broader rollout, place the NIM behind an internal service mesh or load balancer, store the NGC key in a secrets manager, and publish the pinned image through an internal platform. IDE assistants can point to a custom OpenAI-compatible base URL; an example configuration for Continue uses http://localhost:8000/v1.

Guardrails and CI Verification

NeMo Guardrails installs via pip packages nemoguardrails and openai, with config files under code-rails/config. The guardrails run a self_check_input prompt before the model call, returning YES or NO, and refuse the request if the answer is YES. Suggested conservative human-only paths include authentication, authorization, payment processing, cryptography, deployment manifests, and incident-response automation.

The CI verification gate, labeled ai-assisted, runs unit tests, SAST, secret scan, hallucinated-dependency scan, and license check before human review. The workflow named ai-assisted-pr-checks triggers on pull-request events when the label ai-assisted is present. Steps include checkout, Python 3.11 setup, make test, Semgrep SAST, Gitleaks secret scan, dep-hallucinator dependency scan, and pip-licenses license scan. dep-hallucinator scans requirements.txt for slopsquatting across PyPI, npm, Maven, crates.io, and Go, outputting an SBOM and exit codes. Alternative scanners include slopgate (Python, npm, Go, PR-diff aware, uploads SARIF) and XBOM (CVE scanning, slopsquatting detection, SBOM generation). All scanners should be pinned to specific versions like any other dependency.

Metrics and Rollback

Commit trailers record the assistant name and scope for defect-escape, rollback, and review-latency metrics. Prometheus and Grafana loop escape-rate signals back to tighten Guardrails policy or pause rollout. The StarCoder2 NIM container ships a signed SBOM and VEX record for the model image, providing supply-chain transparency.

This setup gives inference engineers a template for deploying a guarded coding assistant that keeps code on-premises, enforces policy before generation, and adds CI checks plus feedback loops for continuous improvement.