Inside an AI Data Centre: Power, Cooling, and Silicon

Walk into a traditional enterprise data center and you'll hear the steady hum of air handlers cooling racks that draw maybe 5-10 kilowatts each. Walk into a modern AI data center and the sound changes entirely — a wall of forced air or the quiet gurgle of liquid coolant moving through racks that can draw 100 kilowatts or more, all to feed racks of chips doing nothing but matrix multiplication, over and over, at a scale that would have seemed absurd a decade ago.

The building looks similar from the outside. Inside, almost everything about how it's designed, powered, and cooled has been rethought. Understanding why — and where the physical limits are — matters for anyone trying to reason about how far AI can scale, how much it will cost, and how quickly.

This guide walks through the AI data center layer by layer: the accelerator silicon that sets the requirements, the power path from grid to chip, the cooling methods that replaced chilled air at high density, and the water and siting trade-offs behind them. It closes with what these constraints mean for teams that rent AI compute rather than build it.

What actually makes a data center an "AI data center"

A conventional data center is built to run a huge diversity of workloads: web servers, databases, email, virtual desktops. The chips are general-purpose CPUs, power draw per rack is modest, and the main design goal is flexibility — any rack can run any application.

An AI data center is built around one dominant workload: training and running large neural networks. That workload has a specific shape:

• Highly parallel math. Training and inference both boil down to enormous numbers of matrix multiplications, which map well onto specialized accelerator chips rather than general-purpose CPUs.
• Heavy chip-to-chip communication. Large models are split across hundreds or thousands of chips, which must constantly exchange intermediate results. Network bandwidth between chips matters almost as much as the chips themselves.
• Sustained, near-100% utilization. Unlike a typical enterprise server that idles most of the day, a training cluster is often run flat-out for weeks, which changes how you think about power delivery, cooling headroom, and hardware wear.
• Extreme power density. A single AI accelerator can draw 700-1,000+ watts, and a server holding eight of them can pull well over 10 kilowatts by itself — before counting the CPUs, memory, storage, and networking gear around it.

That combination — parallel silicon, dense interconnects, constant full-load operation, and concentrated power draw — is what forces every other design decision in the building.

The basic anatomy

Strip away the marketing language and an AI data center is a stack of four layers, each dependent on the one below it:

1. Silicon — the accelerator chips (GPUs or custom AI chips) that do the actual computation.
2. Servers and racks — the physical enclosures that hold chips, memory, and networking, wired together into clusters.
3. Power — the substation, switchgear, and distribution that gets electricity from the grid to each chip reliably and cleanly.
4. Cooling — the systems that remove the heat all that power turns into, since virtually every watt consumed by a chip is eventually released as heat.

Each layer is discussed below roughly in the order a facility is actually designed: you start by deciding what silicon you're deploying, because that determines the power and cooling requirements everything else has to satisfy.

The silicon layer: what's actually doing the work

Most AI data centers today are organized around GPUs (graphics processing units), originally built for rendering video game graphics but repurposed because their architecture — thousands of small cores executing the same operation in parallel — happens to be well suited to neural network math. A smaller but growing share of capacity runs on custom AI accelerators designed from scratch for machine learning workloads rather than adapted from graphics — see our comparison of GPUs, TPUs, and ASICs for how these approaches actually differ.

What matters for infrastructure design isn't the brand of chip but a few shared characteristics:

Chips are grouped into servers, servers into racks, and racks into "pods" or "clusters" — the actual unit that trains a large model. Because the chips inside a cluster are constantly exchanging data, the physical distance between them affects performance: cable lengths, switch placement, and even which chips sit in which rack are chosen to minimize communication delay. This is a major reason AI data centers are laid out differently from traditional ones — the network topology, not just floor space, drives the architecture.

Why density keeps climbing

Chip makers have been packing more computational power into each generation of accelerator, and each generation has drawn more electricity than the last. That trend, combined with the industry's preference for clustering as many chips as physically possible close together to minimize communication latency, has pushed rack power density up sharply — from a few kilowatts per rack a decade ago toward triple-digit kilowatts in the newest AI-optimized racks. (A more radical answer to the same clustering problem is wafer-scale computing, which fuses the chips themselves into one piece of silicon.) That single trend is the reason cooling architecture has had to change so fundamentally, which is covered below.

Power: getting electricity to the chip

Power is often the actual bottleneck on how large an AI data center can be, more so than land, construction time, or even chip supply. A facility housing tens of thousands of accelerators can require as much electricity as a mid-sized city — a demand curve the International Energy Agency now tracks as its own category — and that power has to arrive reliably, cleanly, and at a scale most local grids weren't built to deliver to a single customer.

The path electricity takes through the building looks roughly like this:

• Grid interconnection. A dedicated substation steps down high-voltage transmission power for use in the building. Getting this built — securing land, permits, and utility agreements — is frequently the longest lead-time item in a new facility, sometimes longer than the construction itself.
• Backup and continuity. Generators (often diesel or gas) and uninterruptible power supply (UPS) systems, typically battery-based, keep critical systems running through outages and bridge the gap before generators spin up.
• Distribution. Power is stepped down again and routed through switchgear to rows of racks, with redundant paths so a single failure doesn't take down a whole cluster mid-training-run.
• Delivery to the rack and chip. Power supplies convert to the voltages chips need, and increasingly this conversion is happening closer to the chip itself as densities rise, to cut down on losses over distance.

Why power is the real constraint

Three separate pressures collide here. First, individual chip power draw keeps rising with each hardware generation. Second, operators want to pack more chips per rack and more racks per building to keep clusters physically close together for performance reasons. Third, grid capacity in any given location is finite and slow to expand — new transmission lines and substations can take years to permit and build, far longer than it takes to manufacture and install chips.

The practical result is that the availability of grid power at a suitable location has become as much a gating factor for AI infrastructure buildout as chip supply. This is why operators increasingly look at sites near existing power infrastructure — former industrial sites, areas with nuclear or hydro capacity, or regions with underused grid connections — rather than choosing locations purely for land cost or connectivity, as was more common with traditional data centers. Some operators are going further still, designing facilities that can flex their own demand — a shift explored in our piece on grid-interactive data centers.

Cooling: getting the heat back out

Every watt of electricity a chip consumes eventually becomes heat that has to leave the building, or the chip throttles itself or fails. Air cooling — pushing chilled air across the equipment — was the default approach for decades of enterprise data centers, and it still works fine at low-to-moderate power densities. It runs into physical limits once rack power climbs into the tens or hundreds of kilowatts, because air simply can't carry heat away fast enough at that concentration without impractical airflow volumes and noise.

That's why AI data centers have been shifting toward liquid cooling, which moves heat far more efficiently because liquid has much higher heat capacity than air for a given volume.

Direct-to-chip liquid cooling has become the practical default for the highest-density AI racks: coolant is piped through a cold plate mounted directly on top of the chip, absorbing heat far more efficiently than air ever could, then carried away to be cooled and recirculated. Immersion cooling, where entire servers sit submerged in a non-conductive fluid, pushes density even further but adds operational complexity — maintenance, fluid handling, and component compatibility all become more involved than in an air-cooled hall.

Water, or not

A common misconception is that liquid cooling necessarily means large-scale water consumption. In practice, many liquid-cooled systems run as closed loops, where the same coolant recirculates and is cooled via heat exchangers rather than evaporated off. Evaporative cooling towers, which do consume water, remain common in many facilities because they're efficient at rejecting heat to the outside air, but the technology choice is a genuine trade-off between energy efficiency, water use, and climate — dry climates and water-scarce regions push operators toward different designs than humid, water-rich ones.

Benefits of Purpose-Built AI Data Centers

Designing a facility around one dominant workload brings trade-offs, but it also delivers capabilities that a general-purpose hall cannot match.

Clusters Large Enough for Frontier Training

Training large models requires thousands of accelerators exchanging data constantly. A building laid out around network topology, with short links and carefully placed switches, lets those chips behave as one machine. General-purpose facilities spread equipment for flexibility, which adds distance and latency between chips. Purpose-built layouts make it possible to run training jobs at a scale that would otherwise stall on communication overhead.

Far Higher Compute per Square Metre

Liquid cooling and redesigned power delivery allow racks drawing tens or hundreds of kilowatts. That density packs more computation into each hall, which shortens cable runs and reduces the land and building footprint needed for a given amount of capacity. For operators facing scarce sites with adequate power, getting more out of each square metre directly expands what a single campus can host.

Better Energy Efficiency at High Load

Liquid carries heat far more effectively than air, so direct-to-chip systems remove heat with less energy spent on fans and chillers at high densities. Converting power closer to the chip also cuts distribution losses. The result is that a larger share of the electricity entering the building goes into computation rather than overhead, which matters both for operating cost and for the sustainability profile of the workloads running there.

Reliability Designed for Long Runs

Training clusters run at near full load for weeks, so redundant power paths, spares strategy, and maintenance access are planned around that pattern. Facilities designed with this in mind reduce the chance that a single component failure interrupts a long job, and they support the checkpointing and failover practices that keep wasted compute to a minimum when something does fail.

Capacity That Others Can Rent

Most organisations will never build one of these facilities. Because cloud providers and colocation operators do, businesses can access high-end accelerators by the hour or through reserved commitments. The specialised building becomes shared infrastructure, letting smaller teams train, fine-tune, and serve models without designing power and cooling systems themselves.

AI Data Center Use Cases

The same physical building can serve quite different workloads. These are the main ones, and what each demands from the facility.

Training Large Models

Training is the workload that shaped AI data center design. A single run spreads a model across thousands of accelerators that exchange intermediate results continuously for weeks. The facility must deliver sustained full-load power, remove heat from dense racks without interruption, and keep chips physically close to minimise communication delay. When a single site cannot secure enough power, some organisations split training across geographically separate clusters linked by high-speed networks.

Serving Inference at Scale

Once trained, models answer requests from users and applications around the clock. Inference workloads are more variable than training, with demand following usage patterns, and they often need to run close to users for latency reasons. Operators balance dense accelerator racks with the need to place capacity in regions that match where requests and data originate, including residency requirements.

Fine-Tuning and Enterprise AI Workloads

Businesses adapting existing models to their own data typically rent capacity from cloud providers rather than building anything. These jobs are smaller than frontier training but still need accelerators and fast storage. For enterprises, the data center's role is to make that capacity available on reasonable timelines, which is why reservation planning and regional availability have become part of AI project planning.

Scientific and Engineering Computing

Simulation, climate modelling, and other research workloads share much of AI's appetite for parallel computation and fast interconnects. Increasingly they run on the same accelerator-based infrastructure, and some combine physics-based simulation with machine learning. Research institutions benefit from the same density and cooling advances developed for AI. They also compete for the same scarce capacity.

Flexible, Grid-Aware Operation

Some operators design facilities that can adjust their own demand in response to grid conditions, shifting deferrable work when power is scarce or expensive. That approach helps secure grid connections in constrained regions, turning the data center into a more cooperative participant in local power systems. For businesses renting capacity, this can show up as cheaper pricing for workloads that tolerate flexible scheduling.

Why this matters right now

None of this is abstract engineering trivia — the physical constraints of power and cooling are shaping decisions well outside the data center itself.

Model developers plan training runs around the compute they can actually get, and increasingly the limiting factor is not "can we buy enough chips" but "can we get them enough power, in one place, cooled adequately, on a reasonable timeline." That power-and-cooling math plays out differently across training and inference workloads, which have different scheduling profiles. That has pushed some organizations toward multi-site training approaches that split a single training run across geographically separate clusters, connected by high-speed networks, specifically because no single site can secure enough power fast enough. It has also pushed real estate and power-purchase decisions into the strategic core of AI companies in a way that would have been unusual for a software business a few years ago.

For cloud providers and colocation operators, the AI data center buildout has become a capital-intensive race with power availability, not chip availability, often the pacing item. Announcements of new campuses increasingly come paired with power agreements — sometimes multi-year commitments tied to specific generation sources — because the electricity contract can take longer to secure than the building.

For businesses consuming AI capacity rather than building it, the practical effect shows up as cost and availability: compute-intensive workloads can be capacity-constrained during periods of high demand, and pricing for the highest-end accelerators reflects how expensive it is to build and power the facilities that house them.

Practical implications for businesses and builders

Most organizations will never build their own AI data center, but the physical realities above still shape decisions they make.

• Cloud vs. on-premises AI infrastructure is increasingly a power-and-cooling decision, not just a cost decision. Retrofitting an existing office server room for high-density GPU racks is often impractical — the power delivery and cooling requirements exceed what the space was built for. This pushes many organizations toward colocation or cloud GPU capacity rather than in-house builds, even when cost modeling alone might favor owning hardware.
• Regional availability affects where workloads can run. Because AI-capable data center capacity is concentrated wherever power and land line up, the region where you can get GPU capacity may not match the region your data or users are in, which has knock-on effects for latency and data residency compliance.
• Procurement timelines have lengthened. Reserving significant GPU capacity, whether through cloud commitments or dedicated hardware, increasingly requires planning many months ahead, because both chip supply and the power/cooling infrastructure to house them have long lead times.
• Sustainability reporting now has to account for AI workloads specifically. Organizations with emissions or energy targets are finding that AI compute usage — whether run in-house or purchased from a cloud provider — is a fast-growing line item that older sustainability frameworks weren't built to track cleanly, a topic we cover in more depth in the energy cost of AI.
• Vendor selection increasingly involves questions about power sourcing and cooling design, not just price per GPU-hour, since these affect both reliability and the sustainability profile of the workload.

Common AI Infrastructure Planning Mistakes

Organisations that consume AI compute, rather than build it, still make costly planning errors rooted in the physical realities above.

Trying to Retrofit an Office Server Room

A server room built for a few kilowatts per rack cannot host modern GPU servers at full density. Teams that buy accelerators for an existing room discover that power circuits, cooling capacity, and floor loading all fall short. The hardware ends up throttled, partially installed, or moved to colocation at extra cost. Check power and cooling capacity before buying hardware, not after.

Assuming GPU Capacity Is Always Available on Demand

Cloud computing trained teams to expect instant capacity. High-end accelerators don't always work that way, especially during periods of heavy demand. Projects that plan to spin up large clusters the week they need them can face delays. Significant capacity often needs reserving months ahead.

Ignoring Where the Capacity Physically Sits

AI capacity concentrates where power and land are available, which may be far from users or from the region where data must stay. Teams that pick a provider on price alone can end up with latency problems or residency conflicts. Confirm which regions can host the workload before committing.

Comparing Providers Only on Price per GPU-Hour

The hourly price says little about reliability, interconnect quality between chips, cooling design, or power sourcing. A cheaper offer with weaker networking can make a distributed training job slower overall, and an opaque power mix can complicate sustainability reporting. Evaluate the full picture, not one number.

Running Long Jobs Without Robust Checkpointing

Hardware fails during multi-week runs, and an interruption without recent checkpoints can waste days of compute. Teams treating training like a short batch job, without checkpointing and restart procedures, absorb those losses repeatedly. Plan for failure as a normal event in long runs.

AI Compute Best Practices

• Decide cloud, colocation, or on-premises with power in mind. Treat power delivery and cooling as first-order inputs to the decision, alongside cost. If existing facilities can't support high-density racks, factor in the cost of upgrades or move the workload elsewhere.
• Reserve capacity early for planned projects. For training runs or launches that need substantial accelerator capacity, start reservation conversations months ahead and build that lead time into the project plan. Keep a fallback plan, such as a second provider or a smaller model, in case reservations fall through.
• Match regions to latency and residency needs. Map where your users and data are, confirm which provider regions offer the required accelerators, and resolve conflicts before architecture decisions are locked in. Revisit the map when you expand into new markets.
• Ask providers about power sourcing and cooling design. Include questions on energy mix, cooling approach, water use, and redundancy in vendor evaluations, since they affect reliability and sustainability reporting. Ask for documentation rather than accepting verbal assurances.
• Track AI compute in sustainability reporting. Measure energy use from AI workloads, whether in-house or purchased, as its own category so growth is visible rather than buried in general cloud spend. Tag workloads by team or product so the numbers lead to decisions.
• Checkpoint long-running jobs and test restarts. Save training state at intervals that balance overhead against potential loss, and rehearse recovery so a hardware failure costs hours, not days. Store checkpoints somewhere that survives the loss of the cluster itself.
• Separate training and inference planning. Plan bursty, large training jobs differently from steady inference serving, which may need capacity closer to users and different reliability targets.
• Revisit hardware efficiency with each generation. Compare performance per watt and per unit cost when new accelerators arrive, rather than assuming the newest option is always the most economical for your workload. Benchmark your own models on candidate hardware before switching.

Limitations and open questions

The current build-out approach has real constraints and unresolved tensions that are worth being clear-eyed about.

Grid capacity is a hard limit that money alone doesn't solve quickly. Even a well-funded operator can't shortcut the years-long process of permitting new transmission infrastructure or securing large blocks of grid capacity in a desirable location. This is arguably the single biggest structural bottleneck on how fast AI compute capacity can grow, more binding in many cases than chip manufacturing.

Liquid cooling adds operational complexity that the industry is still working through. Direct-to-chip and immersion systems introduce failure modes that didn't exist in air-cooled halls — coolant leaks, fluid compatibility with electronics, and maintenance procedures that require different skills than traditional data center operations. Standards and best practices, including thermal guidelines from bodies like ASHRAE, are still maturing for these newer approaches.

Efficiency gains and demand growth are racing each other. Chip makers continue to improve performance per watt with each generation, and data center operators continue to improve cooling and power-delivery efficiency. But total demand for AI compute has been growing fast enough that absolute power and cooling requirements keep climbing even as efficiency improves — it's an open question which trend wins out over the next several years.

Water and land use trade-offs don't have a universally right answer. Evaporative cooling is efficient but consumes water; closed-loop liquid systems save water but may be less energy-efficient in some climates; siting decisions that favor available power may conflict with siting decisions that favor water availability or land cost. These are genuine trade-offs, not solved problems, and different operators are making different bets.

Redundancy and reliability standards built for traditional data centers don't map cleanly onto AI clusters. A training run interrupted partway through can waste enormous amounts of compute time if checkpointing isn't handled well, which changes how operators think about acceptable downtime and failover compared to, say, a web application that can tolerate brief interruptions.

What to watch next

A few threads are worth tracking if you want to understand where AI infrastructure is heading:

1. Power-purchase and generation partnerships. Watch for AI infrastructure operators entering direct agreements with power generators — including newer generation sources — as a way to secure supply outside the constraints of local grid expansion timelines.
2. Cooling technology maturity. As direct-to-chip and immersion cooling become more standard rather than exceptional, expect clearer industry standards, more competitive vendor options, and gradually falling costs for high-density deployments.
3. Chip efficiency trajectories. Each new accelerator generation's performance-per-watt figure is a useful signal for whether the industry is bending the demand curve for power, or whether raw power draw keeps climbing in step with capability.
4. Multi-site and distributed training approaches. As single-site power becomes harder to secure at scale, techniques for training models across geographically distributed clusters are likely to become more common, with implications for network infrastructure between sites.
5. Regulatory and community response. Local permitting fights over new data center power and water use are already shaping where facilities can be built, and that dynamic is likely to intensify rather than fade as the number of large facilities grows.

Teams evaluating where and how to run demanding AI workloads — cloud, colocation, or hybrid — can get hands-on infrastructure guidance from Woyce Technologies.

FAQ

What is an AI data center, exactly?

It's a data center purpose-built around specialized accelerator chips (typically GPUs) running AI training and inference workloads, as opposed to the general-purpose servers found in a traditional enterprise data center. The defining differences are much higher power density per rack, heavier chip-to-chip networking requirements, and cooling systems designed for sustained, near-constant full-load operation.

Why do AI data centers need so much more power than traditional ones?

AI accelerator chips draw far more power per chip than general-purpose CPUs, and operators pack many of them densely together to minimize the communication delay between chips during training. A single AI-optimized rack can draw well over ten times what a traditional server rack draws, and clusters of thousands of chips can approach the power consumption of a small city.

Why is liquid cooling replacing air cooling in AI data centers?

Air can't remove heat fast enough once rack power density climbs into the tens or hundreds of kilowatts without impractical airflow volumes. Liquid has much higher heat capacity than air, so pumping coolant directly to a cold plate on the chip (or immersing servers entirely) removes heat far more efficiently, which is why liquid cooling has become standard for the highest-density AI hardware.

Is power or chip supply the bigger bottleneck for AI infrastructure growth?

Increasingly, power — specifically the time it takes to secure grid capacity and build out transmission and substation infrastructure at a given site. Chip manufacturing and delivery timelines are significant, but grid interconnection and power agreements have in many cases become the longer lead-time item. A factory can ramp chip output within a product cycle, while a new substation or transmission line can take years of permitting and construction. That gap is why operators now choose sites based on available power first, and why announcements of new campuses so often arrive alongside long-term power-purchase agreements.

Does liquid cooling mean AI data centers use huge amounts of water?

Not necessarily. Many liquid-cooled systems operate as closed loops that recirculate coolant rather than consuming it. Water use depends heavily on the specific cooling design — evaporative cooling towers do consume water, while closed-loop liquid systems generally don't — and operators make different choices depending on local climate and water availability.

How does this affect businesses that just consume AI services rather than build data centers?

Power and cooling constraints on the supply side show up downstream as capacity availability, pricing, and regional access to the highest-end compute. Businesses relying on cloud AI capacity may find that reserving significant GPU access requires longer planning horizons than they're used to for other cloud infrastructure. If AI capacity is central to a product roadmap, build that lead time and regional availability into plans early rather than assuming on-demand capacity will always be there.

What's the biggest unresolved challenge in AI data center design today?

Reconciling the pace of demand growth with the multi-year timelines required to secure grid power and build supporting infrastructure. Chip and cooling technology continue to improve efficiency, but total demand has been growing quickly enough that power availability, not technology, is the binding constraint in many regions. In practice, securing power has to start early in planning, and efficiency gains from new chips or cooling slow the growth in demand rather than reversing it.

Conclusion

The core problem an AI data center solves is physical: keeping thousands of power-hungry accelerators fed with electricity, connected over very short links, and cool enough to run at full load for weeks. Everything distinctive about these buildings follows from that workload, not from branding.

A few insights are worth carrying forward. Silicon choices come first and set the power and cooling budget for everything else. Rising rack density is what pushed the industry from air toward direct-to-chip and immersion cooling. And grid capacity, with its multi-year permitting timelines, now often paces growth more than chip supply does.

The caveats matter too. Efficiency gains per chip are real, but total demand has been rising faster, so absolute power use keeps climbing. Water use depends heavily on cooling design and climate, so blanket claims in either direction deserve skepticism. Reliability practices from traditional facilities also need rethinking when one interrupted training run can waste weeks of compute.

For most businesses, the practical next step is not building anything but planning around these limits: reserving GPU capacity early, checking which regions can actually host your workloads, and asking providers about power sourcing and cooling design. If you are working out the right mix of cloud, colocation, and hybrid infrastructure for AI workloads, our cloud architecture services can help you plan it.