Meta has announced 'MetaRoCE,' a new communication protocol designed for a 1 million GPU scale, enabling high-speed connection of AI clusters over Ethernet, assuming packet loss.
On August 24, 2026, Meta announced 'MetaRoCE,' a new protocol designed for connecting AI infrastructure on a scale of 1 million GPUs over Ethernet. It revises the previous RoCEv2's premise of 'delivering packets in order' and 'minimizing packet loss,' and is designed to maintain high communication performance even in large-scale networks where the arrival order of packets may be altered or some packets may be lost.
MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet - Engineering at Meta
When training a large-scale AI model using multiple GPUs, the GPUs need to frequently exchange calculation results. For example, in processes called 'All-Reduce' or 'All-to-All,' many GPUs exchange data, so even one GPU with slow communication can slow down the entire process. In AI inference, when running a model divided across multiple machines, communication latency affects response time, so network performance becomes more important as the number of GPUs increases.
One of the technologies used for GPU-to-GPU communication is 'Remote Direct Memory Access (RDMA).' RDMA is a mechanism that directly exchanges data with the memory of another machine over a network, minimizing the need for data copying by the CPU. The mechanism that uses RDMA over Ethernet is called 'RDMA over Converged Ethernet (RoCE),' and Meta has also been using RoCE for large-scale distributed AI learning.
Conventional RoCEv2 configurations generally assume that packets arrive in sequential order and typically use flow control called 'Priority Flow Control (PFC)' to prevent packet loss. However, as networks grow larger and the number of communication paths increases, distributing packets across multiple paths allows for more efficient use of bandwidth.
MetaRoCE doesn't treat a massive network as one large communication path, but rather divides it into numerous smaller logical paths, with each NIC (Network Interface Card) responsible for communication individually monitoring its status. By communicating the status of each path to the NIC, the NIC can reduce traffic on congested paths or avoid problematic paths, among other advantages. The image below illustrates that while the conventional method on the left treats the entire network as a single 'pipe,' the granular path-based approach adopted by MetaRoCE on the right allows for monitoring round-trip time and congestion for each individual path.
When actively distributing packets across multiple routes, the order in which packets arrive is not necessarily the same as the order in which they were sent. MetaRoCE introduces 'Native Out-of-Order Delivery,' which treats the arrival of packets out of order as normal operation, not an anomaly. By including information in each packet indicating 'where in memory it should be written,' it is now possible to write directly to the final memory location starting with the packets that arrive first.
MetaRoCE's 'Native Multipathing' uses multiple paths within a single connection, distributing communication on a packet-by-packet basis. Because the traffic volume and round-trip time are managed individually for each path, the entire connection is less likely to be affected if one path becomes congested or fails.
MetaRoCE also assumes that packet loss will occur over Ethernet. Instead of using PFC to minimize packet loss as in conventional methods, it manages packet numbers along each path to detect lost packets. It uses a 256-bit Selective Acknowledgment (SACK) bit vector to identify missing packets and retransmits only the lost packets.
In congestion control, the sender not only refers to a congestion notification called 'Explicit Congestion Notification (ECN),' but there is also a mechanism for the receiver to communicate the communication speed to the sender. Meta explains that even when 'incasting' occurs, where communication is concentrated from many machines to one, the receiver can quickly adjust the amount of communication by informing the sender of the available bandwidth.
MetaRoCE's main requirements on the network side are an ECN that notifies of congestion and 'Equal-Cost Multi-Path (ECMP)' that distributes communication to multiple equivalent paths. It does not require PFC, which requests packet suspension from switches, or special in-network telemetry, so it can be used in various Ethernet configurations. Existing RDMA Verbs APIs and software stacks can be used with basically no changes, and functions such as 'multiplane,' which uses multiple independent network planes in parallel, can be handled from the extension API.
Meta collaborated with AMD to implement MetaRoCE on Pensando's programmable NICs and compared it with RoCEv2 on a 64-node AMD GPU cluster. The following are the comparison results presented by Meta. The vertical axis shows the difference in processing completion time relative to RoCEv2, and the horizontal axis shows the size of the data being sent. When using MetaRoCE, processing time is reduced across a wide range of data sizes, especially in All-Reduce. On the other hand, in All-to-All, some data sizes show values exceeding 0%, indicating variability in results depending on the data size.
In tests simulating packet loss, MetaRoCE maintained approximately 86% throughput even with 1% packet loss, and was able to continue communication even with a significant 10% packet loss. Furthermore, in tests handling up to 4000 simultaneous connections in 4-plane and 8-plane multi-plane networks, throughput increased with the increase in network planes, and it was reported that communication was redistributed without application or administrator intervention even when paths were intentionally disrupted.
Meta plans to release the MetaRoCE specification through the Open Compute Project (OCP) to enable different manufacturers to develop compatible hardware. Implementation on NICs other than AMD Pensando is also underway.
Meta plans to release the MetaRoCE specification, a reference implementation optimized for the Data Plane Development Kit (DPDK), a software platform for high-speed packet processing, and a test environment to verify product compliance with the specification at the '2026 OCP Global Summit' in October 2026. In the future, they plan to expand applications beyond large-scale communications within data centers to include short-distance communications within racks, long-distance communications across buildings and regions, distributed storage, and KV caches used in AI inference.
Related Posts:
in AI, Posted by log1d_ts