Significantly higher demands on IT cooling

The new supercomputer is located in the basement of the headquarters along the research mile in Kaiserslautern, in a server room designed especially for this purpose, a medium voltage supply of 0.8 MW was installed to operate the corresponding periphery, which includes 1 petabyte of data storage just for the supercomputer. The extremely high computing power in such a small space generates excessive heat, placing increased demands on the IT cooling system. To be able to properly cool the supercomputer, DFKI needs an intelligent solution that can meet the requirements of precise, reliable, energy efficient and noise-reducing cooling of the DGX-2.

The main requirement is to provide strong cooling performance in a very small space with relatively low energy consumption. Also, it is important to keep the noise emission from outdoor units to a minimum so as not to disturb local residents. Another important factor is that cooling accounts for a large proportion of the energy consumption at the new data centre. This is why the overall concept was implemented keeping economic operation in mind, meaning that energy consumption should be minimised. To meet the various requirements of cooling the DGX-2, a special, intelligent air conditioning concept with energy efficient and powerful system solutions had to be created.

Greatest efficiency in smallest space

The i-NEXT series by RC IT Cooling is designed especially for high precision air conditioning of servers, utility rooms and computer centres. It provides the ideal ratio of performance to floor space required. It facilitates precise temperature and humidity control as well as humidifying and dehumidifying. Energy saving and noise-optimised EC high performance fans ensure controlled air flow. The fan speed automatically adapts to the thermal load identified by the sensors in the heat and cold aisle. The evolution+ controller controls the activation times of the compressor applying FIFO logic and regulates the generated cooling performance as a factor of the current heat intake. The data can also be transferred to a higher-order monitoring and remote maintenance system.

The climate chamber has two cooling circuits. One of the circuits is a water-cooled DX system. All of the components essential to operation are built into the unit and the controller. The second circuit is designed as a cold-water circuit for free-cooling operation. Only a dry cooler with supply pump has to be added on site to ensure energy efficient operation.

The indirect free-cooling system consists of a smart combination of various operating modes. When it is warm outside, only the DX system runs. As soon as the outdoor temperature falls slightly below the desired cold water flow temperature of the cold water circuit, free-cooling mode runs parallel to the DX system. When the outdoor temperature is sufficiently low, only the free-cooling circuit runs.

"Complete free-cooling" mode

When the outdoor temperature is low enough, cooling is achieved solely with outdoor air in the connected dry cooler, while the compressor is deactivated.
Chế độ "free-cooling hoàn toàn"

"Hybrid-free-cooling" mode

As soon as the outdoor temperature falls slightly below the desired cold water flow temperature of the cold water circuit, free-cooling mode runs parallel to the DX system. The air is precooled by the cold water heat exchanger in the unit. The compressors are activated as well to provide the cooling lacking for the complete room load.

"Direct evaporation (DX)" mode

Only when the outdoor temperatures are very high is exclusively the DX circuit is active and transporting away the heat. In this mode the dry cooler acts as a heat exchanger for the cooling water circuit.

Intelligent energy management

The two climate chambers are placed to the left and right at the ends of the server room. In the middle of the room there are five racks with the new computer and its peripheral IT components, with an aisle running around it. Server air conditioning is designed applying the down-flow principle to ensure optimum, energy-efficient cooling. Air intake into the climate chamber is from above, while the air outlet flows down and into a double bottom. Outlets in the double bottom on the front of the rack ensure ideal air distribution in the data centre. This creates a cold aisle on the air inlet side of the rack. The racks suck the cold air in and then blow the warm air out the back and into the so-called heat aisle. Then the air is sucked into the climate chamber currently operating, where it is cooled.

To precisely control the cold air supplied to the servers, a cold aisle housing was selected for this project. The heat and cold aisle along with the adjusted air flow ensure that the temperature remains constant. Working conditions become more stable and the entire air conditioning system is more efficient. Most of the time the precision climate chamber runs in hybrid or complete free-cooling mode. In standard mode cooling initially occurs with water via the free-cooling feature. If the heat load exceeds a certain reference point, a compressor is added to additionally cool the air intake through a coolant circuit. If the cooling requirements exceed an additional temperature reference point, the second compressor in the climate chamber switches on to ensure that the temperature level of 21 °C specified by the manufacturer is maintained.

Minimal noise emission in the neighbourhood

The generously dimensioned dry coolers ensure that the third important requirement of this project is met. The noise level was reduced by placing the climate chambers with EC fans, speed-controlled compressors and pump stations in the housing.

The heat exchangers outside have large heat exchange surfaces and bigger fans than the standard model. This means that the fan speeds are lower, resulting in a noise level of no more than 35 dB(A) at a distance of one meter.

Conclusion

Maintaining the proper climate for the new DGX-2 supercomputer at the DFKI poses a steep challenge in regard to IT cooling. First of all, the greater computing power requires powerful precision air conditioning in a limited space. Secondly, the current environmental protection discussion demands a rational allocation of energy. So only concepts and system solutions geared towards high-efficiency criteria can even be considered.

And thirdly, noise emission should be low enough that it does not disturb local residents. Applying know-how and the system solutions provided by RC Cooling, an ultramodern concept for cooling the new DFKI server room was achieved. Water-cooled climate chambers with direct evaporation and full inverter technology as well as the free-cooling feature are used. Energy saving and noise-optimised EC high performance fans ensure controlled air flow through a complete cold aisle housing. Generously dimensioned heat exchangers outside, with low fan speeds, reduce noise to a minimum.