Google explains that load balancing for AI agents is insufficient based solely on the number of requests.
On August 3, 2026, Google explained the points to be aware of regarding load balancing for the stable operation of AI agents that process voice conversations and other data in real time.
Scaling real-time AI agents with session-aware load balancing - Google Developers Blog
Typical web APIs complete their process once the user sends a request and the server returns a result. In contrast, voice AI agents maintain a connection for extended periods, continuously sending and receiving audio, transcripts, AI responses, and synthesized speech. If the user interrupts the AI's speech, the voice generation must stop, the conversation context updated, and a new response must be constructed.
For example, if server A processes 100 requests that take 50 milliseconds to complete, and server B processes 5 conversations that last 20 minutes, server A might appear busier in terms of QPS (Requests Per Second). However, in reality, server B may be experiencing a greater processing load over a longer period of time.
Furthermore, it is dangerous to judge server workload solely by CPU usage. Even if 20 people are connected to the voice AI, the load may appear low while everyone is silent, but the processing load will increase dramatically if they all start talking at once. Assigning new users based only on low CPU usage could overwhelm the server's processing capacity.
Google explains that in addition to QPS and CPU usage, it is necessary to record the 'number of active sessions.' A session is a series of ongoing connections and conversations between a user and an AI agent. While CPU usage indicates the current load, the number of sessions indicates the amount of work the server has already taken on.
In actual session allocation, the system will not rely solely on the number of sessions, but will also consider CPU and memory usage. Servers with CPU usage reaching 90% will not be assigned new conversations, even if they have few sessions. Conversely, servers with a large number of sessions, even if their CPU usage is low, will be assigned new sessions in stages.
If you only allocate connections based on QPS and CPU usage, long conversations may concentrate on certain servers, leaving other servers idle. By also considering the number of sessions, you can distribute new connections while keeping track of conversations already being handled, thus reducing load imbalances.
The application increases the number of sessions at the start and decreases it at the end. Even if there are multiple timeouts, cancellations, or disconnections, the session counter only needs to be decreased once. 'Ghost sessions,' where completed conversations remain, can make the server appear busier than it actually is, leading to miscalculations in load balancing.
Furthermore, if multiple processes update the same counter simultaneously, the process of counting the number of sessions itself can become a bottleneck. Google explains that while Java's 'AtomicInteger' can handle many processes, in environments requiring high concurrency, updates to the same memory address can conflict, so a mechanism like 'LongAdder,' which distributes updates and sums them up later, may be more suitable.
Simply sending a large number of short requests during load testing is insufficient. You need to vary conversation time, number of concurrent connections, ratio of speaking to silence, and frequency of cancellations and disconnections, and then examine the time until a response begins, the number of sessions lost midway, and any bias in the number of sessions per server.
As Google moves towards deploying AI agents in production, it says there is a need to shift towards a design that treats ongoing conversations, rather than one-off requests, as the unit for load balancing.
Related Posts:
in AI, Web Service, Posted by log1d_ts