Liquid Cooling: How Datacenters Handle 100kW+ Racks

A single Nvidia GB200 NVL72 rack pulls somewhere around 120kW. A decade ago, a "hot" rack pulled 10-15kW and datacenter operators considered that a design challenge. The gap between those two numbers is not incremental — it's an order of magnitude — and it explains why the industry that spent thirty years perfecting air conditioning is now retrofitting itself around pipes, coolant, and pumps.

Air cooling didn't fail because someone got lazy. It failed because physics stopped cooperating. Air is a poor conductor of heat compared to liquid, and moving enough of it fast enough to pull 100+kW out of a 42U cabinet without turning the datacenter floor into a wind tunnel is no longer practical. Liquid cooling isn't a nice-to-have upgrade anymore — for the highest-density AI racks, it's the only thing standing between a cluster and thermal throttling.

Why Air Cooling Hits a Wall

Air cooling works by blowing chilled air across heat sinks attached to chips, carrying the heat away, and returning warmer air to a CRAC (computer room air conditioner) or CRAH (computer room air handler) unit to be cooled again. It's simple, cheap, and well understood, and it's fine for racks pulling single-digit or low-double-digit kilowatts.

The problem is heat capacity. Water carries roughly 3,500 times more heat per unit volume than air for the same temperature rise. That means moving the same amount of thermal energy with air requires vastly more volume and velocity than moving it with liquid. Datacenter engineers describe this in practical terms:

• Above roughly 15-20kW per rack, air-cooling fan power starts rising faster than useful cooling gained, because you need increasingly forceful airflow to compensate for air's poor thermal conductivity.
• Above roughly 30-40kW per rack, hot spots become unmanageable — GPUs at the front of a chassis get cooler air than GPUs at the back, creating thermal gradients that force conservative clock speeds across the board.
• Beyond about 50kW, most air-cooling configurations simply cannot keep junction temperatures in a safe operating range without unacceptable acoustic noise, energy draw, or physical airflow constraints (raised floors, hot/cold aisle containment, CRAC capacity).

GPU-dense AI training and inference racks now regularly exceed all three thresholds at once. A rack of eight to seventy-two high-end accelerators, packed tightly for interconnect bandwidth reasons, generates a heat load that air simply cannot evacuate fast enough — not without cooling infrastructure so large it defeats the purpose of dense compute in the first place.

How Liquid Cooling Actually Works

"Liquid cooling" isn't one technology — it's a family of approaches that all substitute liquid for air at some stage of heat removal. The three that matter for datacenters today differ in how close the liquid gets to the chip.

Direct-to-Chip (Cold Plate) Cooling

This is the dominant approach in new AI-cluster builds. A metal cold plate, machined with internal microchannels, sits directly on top of the CPU, GPU, or other high-power silicon in place of a traditional heat sink. Coolant — typically a water-glycol mixture — is pumped through the cold plate, absorbs heat at the source, and carries it out of the server to a coolant distribution unit (CDU), which transfers that heat to a facility-level liquid loop or an outdoor heat rejection system.

Direct-to-chip cooling handles the hottest components (GPUs, CPUs) directly while leaving lower-power components like memory, NICs, and power delivery on conventional air cooling, often with rear-door heat exchangers or in-row cooling handling the residual air-cooled load. This hybrid approach is why most 100kW+ AI racks aren't purely liquid-cooled — they're liquid-cooled where it counts and air-cooled everywhere else.

Immersion Cooling

Here, entire servers — motherboard, GPUs, drives, everything except perhaps power supplies — are submerged in a bath of dielectric fluid, a liquid engineered not to conduct electricity. Heat moves directly from every component into the surrounding fluid, which is then circulated to a heat exchanger.

Immersion comes in two flavors:

1. Single-phase immersion: the fluid stays liquid throughout the cycle, absorbing heat and being pumped to an external cooler, similar in principle to a cold plate loop but bathing the whole server rather than just the chip.
2. Two-phase immersion: the fluid is engineered to boil at a low temperature directly on the hot components, and the vapor condenses on a cooled lid or coil above the tank, releasing its heat and dripping back down as liquid. This uses the latent heat of vaporization — a much larger energy transfer per unit of fluid than simple temperature-based heat absorption — making it extremely efficient, though the fluids involved are typically more expensive and the fluorochemical formulations have drawn environmental scrutiny (some are PFAS-related compounds).

Rear-Door and In-Row Heat Exchangers

A less invasive step up from pure air cooling: liquid-cooled panels are mounted on the back of a rack (rear-door heat exchangers) or between rows (in-row units), intercepting hot exhaust air and cooling it with a liquid coil before it re-enters the room. This doesn't touch the chip directly, so it's less effective at very high densities, but it's a lower-disruption retrofit for facilities not ready for full direct-to-chip infrastructure.

Comparing the Approaches

Why It Matters Right Now

Average rack density hit 27kW in 2026, up 69% year over year. That single number captures the whole story: this isn't a niche problem affecting a handful of frontier AI labs running exotic superclusters. It's the average rack across surveyed datacenters climbing fast enough that infrastructure designed even three or four years ago is already being outpaced.

A few forces are converging to push density this hard:

• GPU power draw is climbing per chip, not just per rack. Successive generations of AI accelerators have increased thermal design power significantly, so even a rack with the same physical GPU count now dissipates far more heat than its predecessor.
• Interconnect physics favors tight packing. Training large models efficiently depends on high-bandwidth, low-latency links between accelerators (NVLink, InfiniBand, and similar fabrics). Spreading GPUs across more, less-dense racks to ease cooling would degrade training performance — so operators are choosing to solve the heat problem rather than sacrifice interconnect density.
• Colocation and cloud customers are demanding it. Enterprises renting AI capacity want it delivered in the densest, most performant configuration available, and providers that can't offer high-density liquid-cooled space are losing that business to those that can.

This creates a genuine bind for datacenter operators: the facilities that were profitable and fully depreciated under an air-cooling design are now the wrong shape for the workloads customers want to run in them. That's driving a wave of retrofits, new-build liquid-ready facilities, and a scramble among colocation providers to add CDU and liquid-loop capacity before their air-cooled floor space becomes a stranded asset.

Practical Implications for Businesses and Builders

If you're renting or building datacenter capacity for AI workloads, rack density and cooling method aren't back-office details — they directly affect what you can deploy, how fast, and at what cost.

For colocation and cloud customers

• Ask about facility liquid-loop capacity, not just rack power. A datacenter can advertise "100kW racks" but if it lacks a facility-level liquid distribution system and adequate heat rejection (dry coolers, cooling towers, or chillers sized for the load), you may be sold theoretical capacity that isn't actually deliverable at scale.
• Understand the hybrid reality. Most GPU servers today still rely on some air cooling for non-GPU components even when direct-to-chip handles the accelerators. Ask what percentage of total rack heat load is liquid-cooled versus air-cooled, since that ratio determines how much of your density gain is real.
• Factor in water usage and sustainability commitments. Liquid cooling loops need water somewhere in the chain (even closed loops usually reject heat to water-cooled chillers or evaporative systems at some point), which matters for organizations with water-usage or sustainability reporting obligations.

For datacenter operators and builders

• CDU sizing and redundancy matter as much as chip selection. A coolant distribution unit failure can take down an entire liquid-cooled row, so N+1 or 2N CDU redundancy is becoming standard practice for mission-critical AI infrastructure.
• Facility retrofits are expensive but often cheaper than new builds. Adding liquid loops, CDUs, and piping to an existing shell is disruptive but typically faster and less capital-intensive than a greenfield liquid-ready facility, especially in supply-constrained real estate markets.
• Plumbing standardization is still shaking out. Coolant quality standards, quick-disconnect fitting types, and facility-to-rack interface specs vary by vendor, and interoperability issues can lock an operator into a single hardware ecosystem if not planned for carefully.

A simplified decision framework

For teams evaluating whether a workload needs liquid cooling, the rough rule of thumb operators use is:

1. Below ~20kW/rack — air cooling with good hot/cold aisle containment is usually sufficient and cheapest.
2. 20-50kW/rack — rear-door or in-row heat exchangers can often bridge the gap without a full liquid-loop buildout.
3. 50-100kW/rack — direct-to-chip cooling becomes close to mandatory for sustained, quiet, efficient operation.
4. 100kW+/rack — direct-to-chip is standard, and immersion cooling starts becoming competitive, particularly for extreme-density or specialized HPC deployments.

Cost considerations

Liquid cooling isn't just a technical decision — it changes the capital and operating cost profile of a facility in ways worth planning for explicitly.

• Upfront capital costs are higher. CDUs, facility-level piping, coolant, and specialized racks all add cost compared to a conventional air-cooled build. For a retrofit, the disruption of running new plumbing through a live datacenter floor adds further expense and scheduling complexity.
• Operating costs typically fall. Once installed, liquid cooling generally reduces the energy spent on cooling relative to compute delivered, since pumps use far less power than the fans and chillers needed to move an equivalent amount of heat with air. Facilities that shift to warm-water loops can cut mechanical chiller use substantially, lowering the electricity bill tied to cooling specifically.
• Density gains offset per-rack cost. Because a single liquid-cooled rack can replace several air-cooled racks at lower density, the cost comparison isn't liquid-vs-air on a like-for-like rack basis — it's liquid-vs-air for the same total compute delivered in the same floor space, which usually favors liquid once density passes the crossover point discussed above.
• Total cost of ownership favors early adoption for AI-heavy workloads. Operators who wait to retrofit until density forces the issue tend to pay a premium for rushed construction and equipment lead times, compared to those who build liquid-ready capacity ahead of demand.

Limitations and Open Questions

Liquid cooling solves the heat-removal problem, but it introduces new ones that the industry hasn't fully settled.

• Leak risk is real, and the stakes are higher. A coolant leak near live electronics is a far more consequential failure mode than a fan dying in an air-cooled server. Cold-plate systems mitigate this with negative-pressure loops (so any leak draws air in rather than pushing coolant out) and leak-detection sensors, but the failure mode is qualitatively different from anything air-cooling operators had to plan for.
• Serviceability changes. Swapping a GPU in a direct-to-chip system means disconnecting and reconnecting coolant lines — a more involved procedure than pulling a card out of an air-cooled chassis, and one that requires trained technicians and quick-disconnect fittings rated for repeated use without degradation.
• Immersion cooling raises material and environmental questions. Two-phase dielectric fluids have historically included fluorochemicals facing regulatory scrutiny over persistence in the environment, and the industry is actively working on alternative formulations. Single-phase immersion fluids avoid some of these concerns but bring their own handling, disposal, and material-compatibility considerations (not every cable jacket or component coating tolerates long-term immersion equally well).
• Standardization is incomplete. Unlike air cooling, where a rack from one vendor works in essentially any datacenter with adequate CRAC capacity, liquid-cooled racks depend on facility-specific loop designs, coolant chemistry, and connector standards that aren't yet fully interoperable across vendors. This raises switching costs and can create vendor lock-in at the facility level.
• Retrofitting older buildings has real limits. Not every existing datacenter has the floor loading, ceiling height, or piping runways to add a facility liquid loop economically. Some older facilities may simply not be viable candidates for high-density AI tenancy regardless of investment.

What to Watch Next

A few trends will shape how this plays out over the next several years:

• Facility-level liquid readiness becomes a standard spec sheet item, the way power density and network uplink speeds already are, making it easier for customers to compare providers on true delivered cooling capacity rather than marketing claims.
• Coolant and connector standardization efforts from industry bodies aiming to reduce vendor lock-in and make liquid-cooled hardware more interchangeable across facilities.
• Warm-water cooling adoption, where facilities cool with higher-temperature water (reducing or eliminating mechanical chiller use) to cut energy consumption — an approach some large-scale operators have already used to lower cooling-related power draw significantly.
• Immersion fluid reformulation as environmental regulation tightens around certain fluorochemical classes, pushing vendors toward alternative dielectric fluids with lower environmental persistence.
• Continued climb in per-rack density as accelerator power draw keeps increasing generation over generation, which will keep pushing the threshold at which liquid cooling becomes mandatory rather than optional further downward over time.

FAQ

What rack density requires liquid cooling?

There's no hard cutoff, but most operators consider air cooling impractical above roughly 30-50kW per rack. Below that, containment and airflow management can usually keep up; above it, direct-to-chip or immersion cooling becomes necessary to maintain safe operating temperatures without excessive fan power or noise.

What's the difference between direct-to-chip and immersion cooling?

Direct-to-chip cooling routes liquid through a cold plate mounted on individual hot components like GPUs and CPUs, while other server parts stay air-cooled. Immersion cooling submerges the entire server in dielectric fluid, cooling every component simultaneously. Direct-to-chip is generally easier to retrofit into existing server designs; immersion typically requires purpose-built enclosures.

Is liquid cooling more energy-efficient than air cooling?

Generally yes. Liquid's higher heat capacity means less energy is spent moving the cooling medium around, and liquid-cooled facilities often report lower Power Usage Effectiveness (PUE) than comparable air-cooled ones because they need less mechanical chiller and fan power to remove the same amount of heat.

Does liquid cooling increase the risk of hardware damage from leaks?

It introduces a new failure mode that didn't exist with air cooling, but modern direct-to-chip systems mitigate it with negative-pressure loops, leak sensors, and quick-disconnect fittings designed to seal automatically when detached. Leak risk is a real engineering consideration, not a solved problem, but it's actively managed rather than a reason to avoid the technology.

Can existing datacenters be retrofitted for liquid cooling?

Many can, particularly with rear-door heat exchangers or in-row units that don't require a full facility liquid loop. Full direct-to-chip or immersion retrofits are more involved, requiring piping, CDUs, and sometimes floor-loading or ceiling-height changes, so feasibility depends heavily on the building's original design.

Why are AI workloads specifically driving this shift?

AI training and inference clusters pack many high-power accelerators tightly together to maximize interconnect bandwidth between chips, which concentrates far more heat per rack than traditional enterprise computing. Combined with each new GPU generation drawing more power than the last, this pushes density well past what air cooling can handle.

Is immersion cooling safe for standard server hardware?

Most modern servers aren't submersion-rated out of the box — immersion typically requires hardware validated for compatibility with the specific dielectric fluid used, since some materials, adhesives, and coatings can degrade over prolonged immersion. Vendors increasingly offer immersion-validated server configurations, but it's not a universal drop-in replacement for air-cooled chassis.

Teams evaluating high-density AI infrastructure decisions can get hands-on help scoping cooling and deployment tradeoffs from Woyce Technologies.