How AI Clusters Talk: Scale-Up vs Scale-Out Networking

A GPU sitting idle waiting for data from its neighbor is a GPU you paid for and aren't using. At the scale modern AI training runs operate — tens of thousands of accelerators, sometimes more — that idle time compounds fast. The compute itself is rarely the constraint anymore. The network connecting the chips is. Understanding how that network is built, and why it's split into two fundamentally different layers, explains more about the economics of frontier AI than almost any other piece of the stack.

The Two-Layer Problem

Training a large model isn't one GPU doing one job. It's thousands of GPUs each holding a slice of the model or a slice of the data, repeatedly exchanging gradients, activations, or parameters so their copies of the model stay in sync. Every training step ends with a synchronization point — an all-reduce, an all-to-all, a pipeline handoff — where every participating chip has to send and receive data before the next step can start. If that exchange is slow, the expensive part of the system (the compute) sits waiting on the cheap part (the wires).

The industry has settled on splitting the interconnect problem into two distinct layers, each optimized for a different scale and a different job:

• Scale-up networking connects GPUs within a server or a tightly packed rack — think 8, 72, or up to 576 accelerators sharing memory-like bandwidth.
• Scale-out networking connects those servers to each other, stitching individual nodes into a cluster of thousands or tens of thousands of GPUs.

These aren't two options to choose between. They're two layers of the same system, built with different hardware, different topologies, and different tolerances for latency and cost, because the traffic patterns at each layer are genuinely different problems.

Scale-Up: Making Many GPUs Look Like One

Inside a server, GPUs need to behave almost like they're on the same chip. Techniques like tensor parallelism split a single matrix multiplication across multiple GPUs, which means those GPUs are exchanging partial results constantly, sometimes multiple times per layer of the model. This traffic pattern tolerates almost no latency and demands enormous bandwidth — closer to memory bandwidth than to network bandwidth.

This is the job of technologies like NVIDIA's NVLink and NVSwitch, AMD's Infinity Fabric, and custom interconnects from cloud providers building their own silicon. NVLink gives each GPU direct, high-bandwidth, low-latency lanes to every other GPU in the domain, and NVSwitch chips act as a crossbar so that any GPU can talk to any other GPU at close to full bandwidth simultaneously — not just to its nearest neighbors.

A few things define a scale-up domain:

The defining constraint of scale-up is physical reach. Copper interconnects are cheap and low-power but can only run useful signal a few meters before losses become unmanageable, which is why the biggest scale-up domains — systems marketed as a single giant "GPU" spanning dozens of physical trays — increasingly rely on optical links even within the rack. The scale-up domain is, in effect, the largest unit of compute you can treat as if it had one shared memory space.

Scale-Out: Stitching Servers Into a Cluster

Once you've built the biggest scale-up domain you can afford or physically fit in a rack, you still need to connect many of those domains together to reach cluster sizes in the tens or hundreds of thousands of GPUs. That's scale-out networking, and it looks much more like traditional data center networking — because at this scale, it has to.

Scale-out fabrics are built from two competing technology families:

1. InfiniBand, purpose-built for HPC and AI, with hardware-level features like RDMA (remote direct memory access, letting one server write directly into another's memory without CPU involvement), adaptive routing, and in-network computing (switches that can perform partial reduction operations themselves).
2. Ethernet, specifically RDMA-over-Converged-Ethernet (RoCEv2), which brings InfiniBand-like low-latency, CPU-bypass transfers to commodity Ethernet switches — at the cost of needing careful congestion control to avoid packet loss, since Ethernet wasn't originally built as a lossless fabric.

The industry has broadly converged on a physical topology for scale-out called a rail-optimized fat-tree (or Clos network). Instead of connecting servers randomly, each GPU's network port is grouped with the "same-numbered" port on every other server into its own dedicated set of switches — a "rail." GPU 0 across every server in the cluster talks over rail-0 switches, GPU 1 over rail-1 switches, and so on. This keeps the common collective-communication patterns (where GPU i on one node talks mostly to GPU i on other nodes) from crossing unnecessary switch hops, which reduces congestion and keeps latency predictable.

Why Not Just Use One Network For Everything

It's a fair question: if InfiniBand and high-end Ethernet can both do RDMA, why not run one flat fabric across the whole cluster instead of maintaining two distinct layers?

The honest answer is cost and physics. Building scale-up-grade bandwidth (aggregate terabytes per second per GPU) across an entire cluster of tens of thousands of nodes would require switch radix, cabling, and optics budgets that don't scale economically — you'd be paying scale-up prices for scale-out distances. Instead, engineers exploit the fact that most communication in a training job is local: heavy, latency-sensitive exchange within the small groups doing tensor parallelism (handled by scale-up), and lighter, more tolerant exchange between groups doing data or pipeline parallelism (handled by scale-out). Matching network cost to actual traffic locality is the entire reason the two-layer design exists.

Why This Matters Right Now

The scale-out layer is going through a step change in capability. Ethernet switch silicon has recently crossed into the 100+ terabit-per-second range per chip — 115 Tb/s class switches are now shipping from major silicon vendors, roughly double the previous generation, which lets network architects build flatter, fewer-hop fat-trees for the same cluster size, cutting both latency and the switch count needed to connect a given number of GPUs. At the same time, 1.6 terabit optical modules are moving through qualification, doubling per-link bandwidth over the 800G optics that are standard in current-generation clusters.

Neither of these upgrades is incremental for its own sake. They're direct responses to a specific design point the industry is now openly discussing: million-GPU clusters. At that scale, the scale-out fabric alone involves hundreds of thousands of optical links, and even small inefficiencies in switch radix or optics power draw multiply into real percentage points of total cluster cost and megawatts of power. A switch generation that doubles bandwidth per chip roughly halves the number of switch tiers (or the number of chips per tier) needed to connect the same number of endpoints, which cascades into savings on cabling, optics, floor space, and power — all of which matter enormously when the cluster itself might cost billions of dollars to build and run.

This is also why optics, not just switch silicon, has become a headline technology in AI infrastructure. Every doubling of cluster size and every doubling of per-GPU bandwidth roughly doubles the number of optical transceivers needed, and transceivers are already a meaningful fraction of network capital cost and a nontrivial fraction of total data center power draw. Co-packaged optics — integrating the optical engine directly into the switch package instead of using pluggable modules — is being pursued specifically to cut that power and cost as 1.6T and eventual 3.2T generations arrive.

Practical Implications for Businesses and Builders

Not every organization is building a million-GPU cluster, and the good news is that most of these lessons scale down. A few practical takeaways apply whether you're running 8 GPUs or 8,000:

• Topology-aware job placement matters more than raw GPU count. Two GPUs in the same scale-up domain communicate orders of magnitude faster than two GPUs in different racks. Scheduling training jobs (or even large fine-tuning runs) with awareness of which GPUs sit in the same NVLink domain can meaningfully change throughput without touching a line of model code.
• Parallelism strategy should match network topology, not the other way around. Tensor parallelism, which is the most communication-hungry technique, should stay inside the scale-up domain. Data and pipeline parallelism, which are more tolerant of latency, should be the techniques that cross scale-out links. Getting this mapping wrong is one of the most common causes of GPUs sitting idle during training.
• Cloud instance types encode network topology whether or not it's advertised. When renting GPU capacity, the interconnect generation (which NVLink version, what fabric between nodes, whether InfiniBand or Ethernet) often matters more for large-scale training throughput than the GPU generation itself. It's worth asking specifically, not inferring from the GPU model name.
• Networking cost doesn't scale linearly with cluster size. Because scale-out fabrics have to grow in width (more switch tiers, more cabling) as clusters grow, the network's share of total infrastructure cost tends to rise with cluster size, not stay flat. Budgeting for AI infrastructure at scale means budgeting for a network that becomes a larger fraction of spend, not a smaller one.
• Inference has different needs than training. Training is dominated by synchronous, bandwidth-heavy collective operations. Serving models — especially with techniques like disaggregated prefill/decode — has its own network demands, often more latency-sensitive and bursty, which is pushing some of the same scale-up/scale-out thinking into inference cluster design as well.

Real Limitations and Open Questions

The two-layer model is well established, but several parts of it are still unsettled:

InfiniBand vs Ethernet Is Not Fully Resolved

InfiniBand still generally leads on out-of-the-box performance for large training clusters — lower latency, better congestion behavior, in-network reduction support. Ethernet's advantage is the broader supplier ecosystem, familiarity to network operations teams, and generally lower per-port cost at volume, plus features like link-level and end-to-end congestion control maturing quickly (Ultra Ethernet Consortium specifications, improved RoCEv2 tuning). Which one wins a given cluster build usually comes down to operational familiarity and vendor relationships as much as raw benchmark numbers, and large operators frequently run both, choosing per cluster.

Optics Reliability at Scale Is a Real Operational Burden

Optical transceivers are one of the more failure-prone components in a large cluster simply because there are so many of them — a single large training job can be stalled by one bad link. As per-lane speeds increase (moving toward 1.6T and beyond), signal integrity margins shrink, and the industry is still working through how much of that risk co-packaged optics genuinely retires versus how much new complexity it introduces around field-replaceability, since integrated optics can't simply be unplugged and swapped like a pluggable module.

The Million-GPU Cluster Is Still a Design Exercise, Not a Solved Problem

Public discussion of million-GPU-class clusters is real, but the network engineering to support that scale — enough switch radix and optical bandwidth to keep a fat-tree from ballooning into an impractical number of tiers, power delivery for the optics alone, and fault tolerance when link failure rates are multiplied by the sheer link count — is still being worked out in public roadmaps rather than proven in deployed systems. Practical questions about mean time between link failures, graceful degradation when parts of the fabric fail mid-training-run, and whether new topologies (like dragonfly variants, which trade some bandwidth for fewer hops) beat fat-trees at this scale remain open.

Software Hasn't Fully Caught Up to Hardware

Collective communication libraries (NCCL and equivalents) and job schedulers are still evolving to fully exploit topology-aware placement automatically. Today, getting the best performance out of a given cluster topology often still requires manual tuning or vendor-specific optimization rather than being handled transparently by the software stack — a gap that narrows with each framework release but hasn't closed.

What to Watch Next

The throughline across all of these is simple: as model and cluster sizes grow, the network stops being plumbing and becomes an architectural decision with direct consequences for cost, power, and how fast a training run actually finishes.

FAQ

What's the difference between scale-up and scale-out in AI networking?

Scale-up connects GPUs within a single server or tightly packed rack using technologies like NVLink, offering extremely high bandwidth and very low latency, closer to shared memory than to a traditional network. Scale-out connects those servers to each other across a data center using fabrics like InfiniBand or Ethernet, prioritizing scale and cost-efficiency over the raw bandwidth achievable at short range.

Why can't AI clusters just use one network for everything?

Building scale-up-level bandwidth (near-memory speeds) across an entire cluster of tens of thousands of GPUs would require far more switch capacity, cabling, and optics than is economically practical. Since most heavy communication in a training job happens within small local groups, splitting the network into a fast local layer and a broader distributed layer matches cost to actual traffic patterns.

Is InfiniBand or Ethernet better for AI training?

InfiniBand generally has an edge in out-of-the-box latency and congestion handling for large training clusters, while Ethernet offers a broader vendor ecosystem and is often more familiar to existing network operations teams. Many large AI infrastructure operators use both, choosing per cluster based on operational fit rather than one technology being universally superior.

What is a rail-optimized network topology?

It's a scale-out design where each GPU's network port across every server is grouped into its own dedicated set of switches, called a rail, so that communication between same-numbered GPUs on different servers stays within a predictable, low-hop path. This reduces congestion for the collective communication patterns common in distributed training.

Why does optical networking matter so much for AI clusters?

As clusters grow and per-link bandwidth increases, the number of optical transceivers needed grows sharply, and those transceivers are a significant source of both capital cost and power draw. Advances like 1.6T optical modules and co-packaged optics aim to keep bandwidth scaling without proportionally scaling cost and energy consumption.

How does network design affect AI training cost?

Idle GPU time caused by network bottlenecks is wasted compute spend, so faster, better-matched networking directly improves the return on expensive accelerator hardware. Additionally, as clusters grow, the network's share of total infrastructure cost tends to increase rather than stay flat, making network architecture a first-order budgeting concern, not an afterthought.

Do I need to understand cluster networking to rent cloud GPUs?

For small-scale work, not really — but for large or multi-node training jobs, the interconnect generation between GPU instances often affects throughput more than the GPU model itself. It's worth explicitly checking what scale-up and scale-out networking a cloud provider offers before committing to a large training run.

For teams designing or evaluating GPU infrastructure and unsure where their network topology is leaving performance on the table, Woyce Technologies can help assess the fit between workload, hardware, and network architecture.