Building real-time AI agents isn’t quite like working with the standard web APIs we’re used to. With a typical API, there’s a predictable lifecycle: the client sends a request, the server processes it, returns a response, and then moves on. This ephemeral model is great because it’s easy to track – you can measure performance through familiar metrics like latency, QPS, and CPU usage.
Real-time AI systems change that model. Instead of handling isolated requests, the backend has to manage a continuous, live bidirectional stream. You’re dealing with a constant stream of audio chunks, transcripts, model outputs, and synthesized speech flowing back and forth simultaneously.
Things get even more complex when a user interrupts. The server has to immediately halt its current speech generation, pivot to update the context, maybe trigger a new tool, and start drafting a different response; this must be done without dropping the connection.
This forces us to rethink our infrastructure. In the world of real-time AI, we aren’t just optimizing for the single request anymore; we’re managing the complexities of a living conversation.
Why QPS is not enough
Traditional load balancing strategies often prioritize request throughput and current CPU utilization. These approaches assume that each incoming request consumes a predictable amount of resources and that finishing a task clears capacity. However, for long-lived, stateful AI streams, these assumptions fall apart.
Consider two backend tasks: Task A handles 100 short requests, each finishing in 50 milliseconds. Task B accepts just 5 requests, but each turns into a 20-minute session. If you only judge these by request arrival rates, Task B appears less busy by request rate, yet it may actually be shouldering a significantly heavier, more committed workload.
QPS tracks arrival volume, but fails to capture the number of live conversations a server is already managing.
Similarly, CPU utilization can be deceptive. A voice runtime, for example, might host 20 silent sessions; because there’s no active speech processing or model inference happening, the server looks underutilized. But as soon as those 20 users start speaking simultaneously, CPU usage can spike suddenly.
While CPU metrics reflect the immediate processing load, active session counts reveal the work the backend has already promised to handle. For real-time AI, you need to balance both signals.
Streams create application-level load
Real-time agents typically rely on bidirectional streaming protocols like gRPC or WebSockets. While specific implementations differ, they all face the same infrastructure hurdle: maintaining an open, long-running connection where data flows constantly in both directions.
While a network observer sees a simple connection, the application treats it as a complex, stateful session. Inside a single real-time agent session, you might have audio buffers, partial transcripts, active tool calls, model context, and user-specific metrics all living within the runtime memory.
Standard load balancers struggle with this. They see the stream, but they can’t distinguish between a genuinely active user conversation, an idle listener, or background noise like retries and health checks. Because the infrastructure lacks visibility into these internal states, you can’t rely on generic connection metrics alone. Instead, you need application-level reporting. The backend service itself is the only component with enough context to accurately track when a session is truly active versus when it has failed, finished, or been canceled.
Tracking active sessions inside the runtime
A simple pattern is to track active sessions at the point where the streaming session lifecycle begins and ends.
suspend fun handleAudioSession(audioStream: Flow<AudioFrame>) {
activeSessions.incrementAndGet()
try {
withTimeout(20.minutes) {
audioStream.collect { frame ->
processAndRespond(frame)
}
}
} finally {
activeSessions.decrementAndGet()
}
}Kotlin
The finally block goes beyond basic cleanup. It is what keeps the active-session count accurate enough for routing decisions.
If the counter fails to decrement, your backend might look overloaded long after the session finishes. Conversely, a double-decrement might report false capacity, drawing in excessive traffic. Production implementations also need to carefully manage edge cases where a timeout, cancellation, or disconnect event all trigger simultaneously to clean up the same session.
Essentially, ghost sessions act as misleading routing signals rather than simple memory leaks.
Even with a precise counter, you have to account for synchronization. Because load balancers generally pull metrics at intervals, while sessions start and stop fluidly, your service needs to report a consistent snapshot of active sessions to prevent the load balancer from making decisions based on outdated data.
Using session count as a balancing signal
Once the backend can report active sessions, the load-balancing model becomes more representative of the workload. A naive capacity model might just use static slots:
remaining_capacity = max_sessions - active_sessionsPlain text
If you have room for 100 sessions and are holding 80, you have 20 slots left. But this is brittle. It assumes every session costs the same amount of CPU, which is rarely true in generative AI.
This is why active session counts should not replace utilization-based balancing; they must be combined into a hybrid model. Utilization (CPU/Memory) captures current resource pressure, while the session count captures committed future load.
One way to combine these signals is to estimate each backend’s effective capacity using a feedback loop. But first, they must normalize both signals into a common unit of measurement. Because load balancers inherently think in rates, they translate the static session count into a continuous flow. For example, if a backend holds 90 active sessions over a 10-second reporting window, one implementation could treat this as 9 “pretend QPS.” By converting static sessions into a standard rate, the routing layer can seamlessly add session pressure to traditional signals like QPS.
A simplified way to estimate effective capacity is to ask: how much additional work can this backend accept before it reaches the target utilization? A simplified capacity estimate formula could look like this:
Let’s break that down:
- Target Utilization: The ceiling you want to safely run at (e.g., 80% CPU).
- Average Utilization: The smoothed CPU consumption over the last reporting window (e.g., the last 10 seconds).
- Cost Per Session: The dynamically calculated average CPU cost of an active stream during the last reporting window.
- Safety Scaler: A dampening multiplier (e.g., 2.0). Because CPU utilization often scales non-linearly with concurrent streams, this acts as a penalty factor to prevent the load balancer from dumping too many new sessions onto a seemingly idle backend at once.
Once the load balancer computes this Additional_Session_Rate, it multiplies it by the reporting interval and adds it to the currently active sessions to find the true Effective Capacity.
This kind of hybrid approach helps address the core routing problem for real-time AI workloads. For example, a backend with 10 active sessions and 90% CPU will have a very high Cost_Per_Session, driving its Additional_Session_Rate to zero, resulting in it receiving no new traffic. Meanwhile, a backend with 80 active sessions but only 40% CPU might seem like it has room, but the Safety_Scaler ensures it is only fed new sessions gradually, preventing sudden spikes.
The exact algorithm depends on your proxy and workload, but the principle is consistent: a real-time AI load balancer must understand both the weight of the current state and the volume of committed sessions.
Benchmarking session-aware balancing
Validating these systems with standard fire-and-forget load tests often misses real failure modes. Tests that send bursts of short requests primarily measure request throughput, which doesn’t replicate the behavior of long-lived AI sessions.
Effective benchmarks should vary:
- concurrent session counts;
- session duration;
- arrival patterns;
- ratios of idle to active speaking;
- cancellation and disconnect rates;
- backend counts;
- maximum sessions per backend.
Metrics must also capture streaming behavior. Beyond average latency and QPS, track metrics like active-session distribution across backends, overloaded assignment rates, p95 and p99 startup latency, time-to-first-stream, dropped sessions, and counter behavior after forced disconnects.
The goal is to compare routing behavior over time.
Request-based balancing often creates lumpy traffic, where some backends accumulate long-lived sessions while others sit idle. Session-aware balancing distributes active conversations more evenly, preventing any single backend from becoming overwhelmed by latent work.
Validating tracker overhead
Even simple session trackers sit on a critical path. Every stream start and end hits them, so if you’re managing massive concurrency, it’s important that it’s designed to ensure that this tracking doesn’t bottleneck a service.
For JVM services, that typically means using a proper microbenchmarking framework to account for JIT optimization, JVM warmup, and dead-code elimination, all of which can easily skew naive results.
It’s generally best to focus testing on contention rather than just single-threaded latency. What happens when multiple threads or coroutines hammer the same counter at once? Using Java as an example, an AtomicInteger may be perfectly fine for many workloads, but at high concurrency, it can suffer from severe cache-line contention (bouncing) as multiple threads constantly attempt to update the same memory address. In high-throughput scenarios, designs such as sharded counters or LongAdder-style aggregation may become more appropriate to maintain throughput.
Session tracking succeeds or fails based on reliability, not just implementation complexity. Every signal used for load balancing must be accurate, low-overhead, and vetted against realistic traffic patterns.
Load balancing sessions, not just requests
Real-time AI shifts the load-balancing burden from a network concern to an application-level problem.
QPS tells you about arrivals, CPU reveals current pressure, and active session counts identify your true, committed concurrency.
For interactive AI workloads like voice, video, or world models, the most effective routing strategies synthesize these signals. When a backend communicates its true session state, a load balancer can make smarter decisions, preventing any single instance from becoming a bottleneck.
As AI agents move into production, infrastructure needs to catch up. Scaling these systems requires shifting our focus from balancing discrete requests to balancing continuous, live conversations.
