{"id":864,"date":"2026-05-07T15:50:02","date_gmt":"2026-05-07T07:50:02","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=864"},"modified":"2026-05-07T15:50:02","modified_gmt":"2026-05-07T07:50:02","slug":"openai-introduces-mrc-multipath-reliable-connection-a-new-open-networking-protocol-for-large-scale-ai-supercomputer-training-clusters","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=864","title":{"rendered":"OpenAI Introduces MRC (Multipath Reliable Connection): A New Open Networking Protocol for Large-Scale AI Supercomputer Training Clusters"},"content":{"rendered":"

Training frontier AI models is not just a compute problem \u2014 it is increasingly a networking problem. And OpenAI just introduced its solution.<\/p>\n

OpenAI announced the release of MRC (Multipath Reliable Connection)<\/strong>, a novel networking protocol developed over the past two years in partnership with AMD, Broadcom, Intel, Microsoft, and NVIDIA. The specification was published through the Open Compute Project (OCP), enabling the broader industry to use and build on it. <\/p>\n

Why Networking is the Hidden Bottleneck in AI Training<\/strong><\/h3>\n

To understand why MRC matters, you need to understand what happens inside a supercomputer during model training. When training large AI models, a single step can involve many millions of data transfers. One transfer arriving late can ripple through the entire job, potentially causing GPUs to sit idle.<\/p>\n

Network congestion, link, and device failures are the most common sources of delay and jitter in transfers \u2014 and these problems get more frequent, and harder to solve, as the size of the cluster increases. This is the compounding infrastructure challenge OpenAI set out to fix.<\/p>\n

According to OpenAI, more than 900 million people use ChatGPT every week. Sustaining and improving those models at that scale means every second of GPU idle time represents real cost and capability loss. The OpenAI states its goal as \u201cnot just to build a fast network, but also to build one that delivers very predictable performance, even in the presence of failures, to keep training jobs moving.\u201d<\/p>\n

What MRC Actually Does: Three Core Mechanisms<\/strong><\/h3>\n

MRC is not a ground-up invention. It extends RDMA over Converged Ethernet (RoCE) \u2014 an InfiniBand Trade Association (IBTA) standard that enables hardware-accelerated remote direct memory access among GPUs and CPUs. It draws on techniques developed by the Ultra Ethernet Consortium (UEC) and extends them with SRv6-based source routing to support large-scale AI networking fabrics.<\/p>\n

RoCE is a protocol that allows one machine to read or write memory on another machine directly over an Ethernet network, bypassing the CPU for maximum throughput. SRv6 (Segment Routing over IPv6) takes this further \u2014 the sending machine encodes the exact route the packet should follow directly inside the packet header, so switches no longer need to run complex routing calculations. This reduces the processing load on switches and saves power \u2014 a meaningful factor at data center scale.<\/p>\n

1. Adaptive Packet Spraying to Eliminate Congestion<\/strong><\/h4>\n

Instead of sending each transfer over a single network path, MRC spreads packets across hundreds of paths simultaneously, reducing congestion in the core of the network. With traditional RoCEv2, packets were stuck in a single path from point A to point B, which contributes to congestion. To overcome this, MRC introduced Intelligent Packet-Spray Load Balancing, so that if a packet\u2019s path is unusable, packets can traverse across other paths on the network. This enables higher bandwidth utilization, reduced tail latency, and fine-grained load balancing at the packet level.<\/p>\n

2. Microsecond-Level Failure Recovery via SRv6 Static Source Routing<\/strong><\/h4>\n

When network paths, links, or switches fail, MRC can detect the problem and route around it on a microsecond timescale. Conventional network fabrics can take seconds or even tens of seconds to stabilize after failures. A key architectural decision makes this possible: the switches don\u2019t need to recompute routes or do anything other than blindly follow the static routes they were configured with. All routing intelligence lives at the NIC level, not the switch level. This is a deliberately unconventional design \u2014 disabling dynamic routing in the switches entirely to prevent two adaptive mechanisms from interfering with each other.<\/p>\n

Before MRC, if a link between a GPU\u2019s network interface and a tier-0 switch failed, the training job would fail. With MRC, the job survives with reasonable performance. If an 8-port network interface loses one port, the maximum rate is reduced by one eighth. MRC detects this, recalculates paths to avoid the failed plane, and immediately tells peers not to use that plane for inbound traffic. Most failed links recover within a minute, at which point MRC brings the plane back into use.<\/p>\n

3. Multi-Plane Networks with Fewer Switch Tiers and Lower Cost<\/strong><\/h4>\n

This is where MRC changes cluster architecture fundamentally. Instead of treating each network interface as one 800Gb\/s link, it is split into multiple smaller links. For example, one interface can connect to eight different switches. A switch that can connect 64 ports at 800Gb\/s can instead connect 512 ports at 100Gb\/s. This lets to build a network fully connecting about 131,000 GPUs with only two tiers of switches. A conventional 800Gb\/s network would require three or four tiers.<\/p>\n

The savings compound further: the research team quantifies that for full bisection bandwidth, the two-tier multi-plane design requires 2\/3 of the optics and 3\/5 the number of switches<\/strong> compared to a three-tier network. Fewer switch tiers also means lower latency \u2014 the longest path traverses only three switches rather than five or seven \u2014 and smaller blast radius when any individual component fails.<\/p>\n

Hardware: Which NICs and Switches Run MRC<\/strong><\/h3>\n

As per the research paper,<\/a> MRC is already running in production on specific, named hardware. It is implemented across 400 and 800Gb\/s RDMA NICs \u2014 including NVIDIA ConnectX-8, AMD Pollara, AMD Vulcano, and Broadcom Thor Ultra \u2014 with SRv6 switch support on NVIDIA Spectrum-4 and Spectrum-5 (running Cumulus and SONiC) and Broadcom Tomahawk 5 via Arista EOS. On the protocol side, AMD contributed the NSCC congestion control algorithm, now part of the UEC Congestion Control specification, along with IB\/RDMA transport semantic layer extensions that allow MRC to integrate with existing RDMA programming models while adding the multipath capabilities that set it apart from traditional transports.<\/p>\n

Already in Production: From Stargate to Fairwater<\/strong><\/h3>\n

MRC is not just a prototype. It is already deployed across all of OpenAI\u2019s largest NVIDIA GB200 supercomputers used to train frontier models, including the site with Oracle Cloud Infrastructure (OCI) in Abilene, Texas, and in Microsoft\u2019s Fairwater supercomputers. MRC has been used to train multiple OpenAI models, leveraging hardware from NVIDIA and Broadcom. Microsoft\u2019s Fairwater supercomputers are located in Atlanta and Wisconsin.<\/p>\n

MRC has been used specifically to train frontier large language models for ChatGPT and Codex<\/strong>. During the training of a recent frontier model, OpenAI had to reboot four tier-1 switches. With MRC, the company did not need to coordinate the reboot with the teams running training jobs in the cluster.<\/p>\n

Key Takeaways<\/strong><\/h3>\n