How AI Clusters Talk: Scale-Up vs Scale-Out Networking

A GPU sitting idle waiting for data from its neighbor is a GPU you paid for and aren't using. At the scale modern AI training runs operate — tens of thousands of accelerators, sometimes more — that idle time compounds fast. The compute itself is rarely the constraint anymore. The network connecting the chips is. Understanding how that network is built, and why it's split into two fundamentally different layers, explains more about the economics of frontier AI — and what actually happens inside an AI data center — than almost any other piece of the stack.

If you are planning a training run, renting multi-node GPU capacity, or budgeting for AI infrastructure, this matters in a concrete way. Two clusters with identical GPU counts can finish the same job at very different speeds depending on how their chips are wired together, and the network's share of infrastructure spend grows as clusters get larger. AI cluster networking is no longer background plumbing; it is a design decision that shows up directly in cost per training run.

This guide explains the two layers that every large AI cluster is built from: scale-up networking, which makes a handful of GPUs behave like one big accelerator, and scale-out networking, which stitches thousands of servers into a single cluster. It covers the hardware behind each (NVLink, InfiniBand, RoCE Ethernet), the rail-optimized topology most builders use, why the layers are kept separate, what is changing with 1.6T optics and faster switch silicon, practical lessons that apply even at small scale, and the questions the industry has not settled yet.

The Two-Layer Problem

Training a large model isn't one GPU doing one job. It's thousands of GPUs each holding a slice of the model or a slice of the data, repeatedly exchanging gradients, activations, or parameters so their copies of the model stay in sync. Every training step ends with a synchronization point — an all-reduce, an all-to-all, a pipeline handoff — where every participating chip has to send and receive data before the next step can start. If that exchange is slow, the expensive part of the system (the compute) sits waiting on the cheap part (the wires).

The industry has settled on splitting the interconnect problem into two distinct layers, each optimized for a different scale and a different job:

• Scale-up networking connects GPUs within a server or a tightly packed rack — think 8, 72, or up to 576 accelerators sharing memory-like bandwidth.
• Scale-out networking connects those servers to each other, stitching individual nodes into a cluster of thousands or tens of thousands of GPUs.

These aren't two options to choose between. They're two layers of the same system, built with different hardware, different topologies, and different tolerances for latency and cost, because the traffic patterns at each layer are genuinely different problems.

Scale-Up: Making Many GPUs Look Like One

Inside a server, GPUs need to behave almost like they're on the same chip. Techniques like tensor parallelism split a single matrix multiplication across multiple GPUs, which means those GPUs are exchanging partial results constantly, sometimes multiple times per layer of the model. This traffic pattern tolerates almost no latency and demands enormous bandwidth — closer to memory bandwidth than to network bandwidth.

This is the job of technologies like NVIDIA's NVLink and NVSwitch, AMD's Infinity Fabric, and custom interconnects from cloud providers building their own silicon. NVLink gives each GPU direct, high-bandwidth, low-latency lanes to every other GPU in the domain, and NVSwitch chips act as a crossbar so that any GPU can talk to any other GPU at close to full bandwidth simultaneously — not just to its nearest neighbors.

A few things define a scale-up domain:

The defining constraint of scale-up is physical reach. Copper interconnects are cheap and low-power but can only run useful signal a few meters before losses become unmanageable, which is why the biggest scale-up domains — systems marketed as a single giant "GPU" spanning dozens of physical trays — increasingly rely on optical links even within the rack. The scale-up domain is, in effect, the largest unit of compute you can treat as if it had one shared memory space.

Scale-Out: Stitching Servers Into a Cluster

Once you've built the biggest scale-up domain you can afford or physically fit in a rack, you still need to connect many of those domains together to reach cluster sizes in the tens or hundreds of thousands of GPUs. That's scale-out networking, and it looks much more like traditional data center networking — because at this scale, it has to.

Scale-out fabrics are built from two competing technology families:

1. InfiniBand, purpose-built for HPC and AI, with hardware-level features like RDMA (remote direct memory access, letting one server write directly into another's memory without CPU involvement), adaptive routing, and in-network computing (switches that can perform partial reduction operations themselves).
2. Ethernet, specifically RDMA-over-Converged-Ethernet (RoCEv2), which brings InfiniBand-like low-latency, CPU-bypass transfers to commodity Ethernet switches — at the cost of needing careful congestion control to avoid packet loss, since Ethernet wasn't originally built as a lossless fabric.

The industry has broadly converged on a physical topology for scale-out called a rail-optimized fat-tree (or Clos network). Instead of connecting servers randomly, each GPU's network port is grouped with the "same-numbered" port on every other server into its own dedicated set of switches — a "rail." GPU 0 across every server in the cluster talks over rail-0 switches, GPU 1 over rail-1 switches, and so on. This keeps the common collective-communication patterns (where GPU i on one node talks mostly to GPU i on other nodes) from crossing unnecessary switch hops, which reduces congestion and keeps latency predictable.

Why Not Just Use One Network For Everything

It's a fair question: if InfiniBand and high-end Ethernet can both do RDMA, why not run one flat fabric across the whole cluster instead of maintaining two distinct layers?

The honest answer is cost and physics. Building scale-up-grade bandwidth (aggregate terabytes per second per GPU) across an entire cluster of tens of thousands of nodes would require switch radix, cabling, and optics budgets that don't scale economically — you'd be paying scale-up prices for scale-out distances. Instead, engineers exploit the fact that most communication in a training job is local: heavy, latency-sensitive exchange within the small groups doing tensor parallelism (handled by scale-up), and lighter, more tolerant exchange between groups doing data or pipeline parallelism (handled by scale-out). Matching network cost to actual traffic locality is the entire reason the two-layer design exists.

Benefits of Two-Layer AI Cluster Networking

The split between scale-up and scale-out is a cost decision first, but it pays off in several other ways for anyone training or serving large models.

Network spend follows actual traffic

Most of the heavy, latency-sensitive exchange in a training job happens inside small groups of GPUs. The two-layer design puts the expensive, near-memory bandwidth exactly there and uses a cheaper, wider fabric for the lighter traffic between groups. Instead of paying scale-up prices across thousands of nodes, builders pay for top-tier bandwidth only where the workload needs it, which is what makes very large clusters affordable at all.

GPUs spend more time computing

Every synchronisation point stalls the step until the slowest exchange finishes. Keeping tensor-parallel traffic on NVLink-class links and sending data-parallel traffic over a rail-optimized fabric shortens those stalls. The result is higher utilisation of the most expensive hardware in the building, which shows up directly as faster training runs for the same accelerator budget.

Latency stays predictable as clusters grow

Rail-optimized fat-trees keep the common collective patterns on short, consistent paths. That predictability matters because a collective operation is only as fast as its slowest participant. A fabric where most traffic takes the same number of hops is easier to tune, easier to model, and less prone to the long-tail delays that idle thousands of GPUs at once.

Each layer can evolve on its own schedule

Because the layers are separate, operators can adopt a larger scale-up domain without redesigning the whole fabric, or move to faster switch silicon and 1.6T optics in the scale-out layer without changing the servers. That independence makes upgrades incremental rather than all-or-nothing, which matters when a single cluster represents years of planning.

Buyers keep supplier choice at scale

The scale-out layer can be built on InfiniBand or on RoCE Ethernet, and some operators run both. Having a credible second option keeps pricing and roadmaps honest and lets teams with deep Ethernet operations experience use skills they already have, rather than adopting an unfamiliar stack only because the cluster got bigger. It also gives operators a fallback if one supply chain tightens during a build.

AI Cluster Networking Use Cases

The same two-layer pattern shows up across very different workloads, each stressing the network in its own way.

Frontier model pretraining

The largest training runs spread a model across tens of thousands of accelerators using a mix of tensor, pipeline, and data parallelism. Here the full design is in play: tensor parallelism inside large scale-up domains, data and pipeline parallelism across a rail-optimized scale-out fabric. Network efficiency decides how much of the cluster's theoretical compute actually turns into training progress, and small improvements in collective performance save meaningful time over a long run.

Multi-node fine-tuning on rented capacity

A company fine-tuning a large open model rents several GPU nodes from a cloud provider. If the instances share a fast scale-up domain and a well-provisioned node-to-node fabric, the job scales cleanly. If the interconnect between nodes is weak, adding nodes adds little speed. Teams that check the interconnect before renting avoid paying for GPUs that spend much of the job waiting.

Mixture-of-experts models

Mixture-of-experts architectures route tokens to different expert sub-networks, which produces heavy all-to-all traffic rather than the neat all-reduce of dense models. That pattern is unforgiving of congestion and benefits from large scale-up domains that keep expert exchanges local. It is one reason scale-up domains have been growing beyond a single server.

Large-scale inference serving

Serving systems that split prefill and decode across different machines move cached attention data between them for every request. The traffic is smaller than training but bursty and latency-sensitive, because users are waiting on each token. Operators are applying the same scale-up and scale-out reasoning to inference clusters, keeping the most latency-critical exchanges inside fast domains.

Shared research clusters

Universities and labs often run one cluster for many teams and job sizes. Topology-aware scheduling, which places each job inside as few scale-up domains and rails as possible, lets small jobs run efficiently alongside large ones instead of scattering across the fabric and slowing everyone down. The same principle helps administrators decide whether a new purchase should add more nodes or larger scale-up domains, based on the job mix the cluster actually runs rather than on peak GPU counts.

Why This Matters Right Now

The scale-out layer is going through a step change in capability. Ethernet switch silicon has recently crossed into the 100+ terabit-per-second range per chip — 115 Tb/s class switches are now shipping from major silicon vendors, roughly double the previous generation, which lets network architects build flatter, fewer-hop fat-trees for the same cluster size, cutting both latency and the switch count needed to connect a given number of GPUs. At the same time, 1.6 terabit optical modules are moving through qualification, doubling per-link bandwidth over the 800G optics that are standard in current-generation clusters.

Neither of these upgrades is incremental for its own sake. They're direct responses to a specific design point the industry is now openly discussing: million-GPU clusters. At that scale, the scale-out fabric alone involves hundreds of thousands of optical links, and even small inefficiencies in switch radix or optics power draw multiply into real percentage points of total cluster cost and megawatts of power. A switch generation that doubles bandwidth per chip roughly halves the number of switch tiers (or the number of chips per tier) needed to connect the same number of endpoints, which cascades into savings on cabling, optics, floor space, and power — all of which matter enormously when the cluster itself might cost billions of dollars to build and run.

This is also why optics, not just switch silicon, has become a headline technology in AI infrastructure. Every doubling of cluster size and every doubling of per-GPU bandwidth roughly doubles the number of optical transceivers needed, and transceivers are already a meaningful fraction of network capital cost and a nontrivial fraction of total data center power draw. Co-packaged optics — integrating the optical engine directly into the switch package instead of using pluggable modules — is being pursued specifically to cut that power and cost as 1.6T and eventual 3.2T generations arrive.

Common AI Cluster Networking Mistakes

Most network-related slowdowns in AI training are not exotic hardware faults. They come from a few recurring decisions about placement, purchasing, and budgeting.

Running tensor parallelism across nodes

Tensor parallelism exchanges partial results many times per layer. Spread it across servers and those exchanges hit the scale-out fabric, which has far less bandwidth and far more latency than NVLink-class links. Throughput collapses, and the GPUs report busy while mostly waiting. Keep tensor-parallel groups inside one scale-up domain and let data or pipeline parallelism cross the slower links.

Choosing instances by GPU model alone

Two cloud offerings with the same GPU can differ sharply in scale-up domain size and node-to-node fabric. Teams that compare only GPU names and hourly prices often pick capacity that looks cheaper but trains slower, so the total cost of the run ends up higher. Ask what connects the GPUs inside a node and what connects the nodes before committing to a large job.

Ignoring job placement on shared clusters

A scheduler that places a job's processes wherever there is free capacity can scatter them across racks and rails. The job still runs, but its collectives cross extra switch hops and compete with other traffic. Topology-aware placement costs nothing in hardware and often recovers performance that teams assumed required new equipment.

Treating the network as a fixed share of budget

Planning documents sometimes assume networking stays a constant fraction of infrastructure spend. In practice, larger clusters need more switch tiers, more cabling, and many more optical transceivers, so the network's share tends to grow. Budgets built on the small-cluster ratio come up short when the design is scaled.

Running Ethernet fabrics without congestion tuning

RoCE Ethernet can perform well for AI, but it depends on careful congestion control and lossless configuration. Teams that deploy it with default settings, expecting InfiniBand-like behaviour out of the box, see packet loss and retransmissions that stall collectives. Plan for the tuning and monitoring work as part of the build.

AI Cluster Networking Best Practices

Not every organization is building a million-GPU cluster, and the good news is that most of these lessons scale down. A few practical takeaways apply whether you're running 8 GPUs or 8,000:

• Topology-aware job placement matters more than raw GPU count. Two GPUs in the same scale-up domain communicate orders of magnitude faster than two GPUs in different racks. Scheduling training jobs (or even large fine-tuning runs) with awareness of which GPUs sit in the same NVLink domain can meaningfully change throughput without touching a line of model code.
• Parallelism strategy should match network topology, not the other way around. Tensor parallelism, which is the most communication-hungry technique, should stay inside the scale-up domain. Data and pipeline parallelism, which are more tolerant of latency, should be the techniques that cross scale-out links. Getting this mapping wrong is one of the most common causes of GPUs sitting idle during training.
• Cloud instance types encode network topology whether or not it's advertised. When renting GPU capacity, the interconnect generation (which NVLink version, what fabric between nodes, whether InfiniBand or Ethernet) often matters more for large-scale training throughput than the GPU generation itself. It's worth asking specifically, not inferring from the GPU model name.
• Networking cost doesn't scale linearly with cluster size. Because scale-out fabrics have to grow in width (more switch tiers, more cabling) as clusters grow, the network's share of total infrastructure cost tends to rise with cluster size, not stay flat. Budgeting for AI infrastructure at scale means budgeting for a network that becomes a larger fraction of spend, not a smaller one.
• Inference has different needs than training. Training is dominated by synchronous, bandwidth-heavy collective operations. Serving models — especially with techniques like disaggregated prefill/decode — has its own network demands, often more latency-sensitive and bursty, which is pushing some of the same scale-up/scale-out thinking into inference cluster design as well.

Real Limitations and Open Questions

The two-layer model is well established, but several parts of it are still unsettled:

InfiniBand vs Ethernet Is Not Fully Resolved

InfiniBand still generally leads on out-of-the-box performance for large training clusters — lower latency, better congestion behavior, in-network reduction support. Ethernet's advantage is the broader supplier ecosystem, familiarity to network operations teams, and generally lower per-port cost at volume, plus features like link-level and end-to-end congestion control maturing quickly (Ultra Ethernet Consortium specifications, improved RoCEv2 tuning). Which one wins a given cluster build usually comes down to operational familiarity and vendor relationships as much as raw benchmark numbers, and large operators frequently run both, choosing per cluster.

Optics Reliability at Scale Is a Real Operational Burden

Optical transceivers are one of the more failure-prone components in a large cluster simply because there are so many of them — a single large training job can be stalled by one bad link. As per-lane speeds increase (moving toward 1.6T and beyond), signal integrity margins shrink, and the industry is still working through how much of that risk co-packaged optics genuinely retires versus how much new complexity it introduces around field-replaceability, since integrated optics can't simply be unplugged and swapped like a pluggable module.

The Million-GPU Cluster Is Still a Design Exercise, Not a Solved Problem

Public discussion of million-GPU-class clusters is real, but the network engineering to support that scale — enough switch radix and optical bandwidth to keep a fat-tree from ballooning into an impractical number of tiers, power delivery for the optics alone, and fault tolerance when link failure rates are multiplied by the sheer link count — is still being worked out in public roadmaps rather than proven in deployed systems. Practical questions about mean time between link failures, graceful degradation when parts of the fabric fail mid-training-run, and whether new topologies (like dragonfly variants, which trade some bandwidth for fewer hops) beat fat-trees at this scale remain open.

Software Hasn't Fully Caught Up to Hardware

Collective communication libraries (NCCL and equivalents) and job schedulers are still evolving to fully exploit topology-aware placement automatically. Today, getting the best performance out of a given cluster topology often still requires manual tuning or vendor-specific optimization rather than being handled transparently by the software stack — a gap that narrows with each framework release but hasn't closed.

What to Watch Next

The throughline across all of these is simple: as model and cluster sizes grow, the network stops being plumbing and becomes an architectural decision with direct consequences for cost, power, and how fast a training run actually finishes.

For teams designing or evaluating GPU infrastructure and unsure where their network topology is leaving performance on the table, Woyce Technologies can help assess the fit between workload, hardware, and network architecture.

FAQ

What's the difference between scale-up and scale-out in AI networking?

Scale-up connects GPUs within a single server or tightly packed rack using technologies like NVLink, offering extremely high bandwidth and very low latency, closer to shared memory than to a traditional network. Scale-out connects those servers to each other across a data center using fabrics like InfiniBand or Ethernet, prioritizing scale and cost-efficiency over the raw bandwidth achievable at short range.

Why can't AI clusters just use one network for everything?

Building scale-up-level bandwidth (near-memory speeds) across an entire cluster of tens of thousands of GPUs would require far more switch capacity, cabling, and optics than is economically practical. Since most heavy communication in a training job happens within small local groups, splitting the network into a fast local layer and a broader distributed layer matches cost to actual traffic patterns.

Is InfiniBand or Ethernet better for AI training?

InfiniBand generally has an edge in out-of-the-box latency and congestion handling for large training clusters, while Ethernet offers a broader vendor ecosystem and is often more familiar to existing network operations teams. Many large AI infrastructure operators use both, choosing per cluster based on operational fit rather than one technology being universally superior.

What is a rail-optimized network topology?

It's a scale-out design where each GPU's network port across every server is grouped into its own dedicated set of switches, called a rail, so that communication between same-numbered GPUs on different servers stays within a predictable, low-hop path. This reduces congestion for the collective communication patterns common in distributed training.

Why does optical networking matter so much for AI clusters?

As clusters grow and per-link bandwidth increases, the number of optical transceivers needed grows sharply, and those transceivers are a significant source of both capital cost and power draw. Advances like 1.6T optical modules and co-packaged optics aim to keep bandwidth scaling without proportionally scaling cost and energy consumption. Reliability is also an open question at very high link counts, so optics increasingly shape how large a cluster can practically grow.

How does network design affect AI training cost?

Idle GPU time caused by network bottlenecks is wasted compute spend, so faster, better-matched networking directly improves the return on expensive accelerator hardware. Additionally, as clusters grow, the network's share of total infrastructure cost tends to increase rather than stay flat, making network architecture a first-order budgeting concern, not an afterthought.

Do I need to understand cluster networking to rent cloud GPUs?

For small-scale work, not really — but for large or multi-node training jobs, the interconnect generation between GPU instances often affects throughput more than the GPU model itself. It's worth explicitly checking what scale-up and scale-out networking a cloud provider offers before committing to a large training run. Ask for the interconnect explicitly instead of inferring it from the GPU name, because a slow link leaves expensive accelerators idle.

Conclusion

The central point of this article is that AI training speed is often limited by how quickly GPUs can exchange data, not by how fast each GPU computes. Clusters handle that with two layers. Scale-up fabrics such as NVLink give small groups of GPUs near-memory bandwidth for the chattiest work, like tensor parallelism. Scale-out fabrics built on InfiniBand or RoCE Ethernet, usually in a rail-optimized fat-tree, connect those groups at a cost that stays workable across thousands of nodes.

For practitioners, the lessons carry down to modest deployments. Map your parallelism strategy to the topology you have, place jobs with awareness of which GPUs share a fast domain, and ask cloud providers exactly what interconnect sits between instances instead of inferring it from the GPU name. Expect networking to take a larger share of the budget as you scale.

Several parts of this picture are still unsettled: the InfiniBand-versus-Ethernet choice, optics reliability at very high link counts, and whether software can exploit topology automatically. Treat vendor roadmaps for million-GPU clusters as plans, not proof. If you are sizing GPU infrastructure for training or serving and want a second opinion on the architecture, our cloud architecture team can review the fit between workload, hardware, and network.