Fintech coding agent scales on GLM-5.2 with dedicated inference

A global fintech now runs its coding assistant on GLM-5.2 through Together AI's Dedicated Model Inference service. The workload is spiky: traffic peaks during engineering hours but overall throughput is low, with token usage at a p50 of 81K, p90 of 163K, and p95 of 178K, and requests per second at a p50 of 3, p90 of 6, and p95 of 7. The previous capacity planning could not keep up with unpredictable team adoption, and the earlier dedicated offering was not built for bursty concurrency. The customer required self-service provisioning, observability, concurrency focus, and model fluidity.

From queues to control

The move to Dedicated Model Inference exposed the endpoint lifecycle through an API, UI, and CLI. The customer reshaped the GLM-5.2 endpoint to fewer, larger replicas with the same total footprint. Initially, near-100% prefill capacity caused request queues of 1 to 3 minutes and decode throughput of about 5 tokens per second. The fix restored cache-session-aware routing and widened the max-inflight-per-worker threshold. The metrics API revealed a 192-second slow request queued behind a 2.3 million-token prefill backlog.

Dedicated Model Inference gives self-serve access to frontier open-source models and performance-aware configurations. The migration from GLM-5.1 to GLM-5.2 happened on the same account without renegotiation. The customer evaluated a 1 million-context configuration but kept 256K and 512K to preserve concurrency headroom. Together built load-testing infrastructure that modeled the customer's actual usage pattern. A weekend concurrency test of GLM-5.1 on 8 to 16 B200 GPUs passed and moved to production.

Production setup and next steps

GLM-5.2 now runs in production at 256K context on 56 B200 GPUs, arranged as 14 replicas with 4 B200 GPUs each. The migration gave the customer direct control over scaling, custom-weight rollouts, and blue/green testing. Config changes happen in-house with zero downtime, eliminating ticket queues to Together. The customer is planning a second workload: a GLM-5.1 node in a new region for chat-style support.

For inference engineers, the case shows that bursty coding traffic needs more than raw throughput. The p95 token usage of 178K and RPS of 7 mean that prefill can dominate, and routing and inflight limits matter as much as GPU count. The shift to fewer, larger replicas with the same footprint is a practical pattern for concurrency-heavy workloads. The metrics API was key to spotting the 192-second queue, and the fix was a routing and threshold change, not a hardware upgrade. Self-service scaling and model fluidity let the team move from GLM-5.1 to GLM-5.2 without renegotiation, which is a real advantage for fast-moving model updates.