Prazisionsklimatisierung Precision air conditioning For the new supercomputer at DFKI
1 Nov. 2022
Mitsubishi Electric LES provides specialist knowledge for shared success: Listening and understanding. Developing intelligent products. Providing skilled advice. Discovering trends. Shaping the future. Turning expertise into solutions.
Maximum cooling in even the smallest spaces Precision air conditioning for new supercomputer at DFKI
Maintaining the proper climate for the new DGX-2 supercomputer at the Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI, German Research Centre for Artificial Intelligence) poses a steep challenge in regard to IT cooling. On the one hand, the greater computing power requires powerful and precise cooling in a limited space. But on the other hand, the current environmental protection discussion demands a rational allocation of energy. So only concepts and system solutions geared towards high-efficiency criteria can even be considered. Applying the know-how and system solutions offered by RC IT Cooling – a Mitsubishi Electric brand – an ultramodern concept for cooling the new DFKI server room was developed.
The DFKI in Kaiserslautern is the leading application-oriented research facility in Germany in the field of innovative software technologies based on modern artificial intelligence methods. The competence centre focusses its research on deep learning and machine learning algorithms. To be able to expand this unique infrastructure ranging from basic research to industrial transfer of knowledge, DFKI has become the first institution in Europe to receive an NVIDIA DGX-2, currently considered the most powerful supercomputer in the world for modern AI processes. One of the research projects is the analysis of satellite images for the recognition and recording of the effects of natural disasters.
The AI supercomputer, with 2 petaFLOPS of performance, is based on an AI network fabric that delivers 2.5 TB per second of throughput. The network fabric of the DGX-2 integrates 16 NVIDIA Tesla V100 Tensor Core GPUs connected via NVIDIA NVSwitch. The innovative technology offers a 12-fold improvement over the previous generations and implementation of solutions that is five times faster than before. This facilitates flexible network options for creating extremely extensive deep learning computer clusters sharing infrastructure environments – in combination with scaling of the virtualisation speed as well as optimisation of user and workload isolation.
Significantly higher demands on IT cooling
The new supercomputer is located in the basement of the headquarters along the research mile in Kaiserslautern, in a server room designed especially for this purpose, a medium voltage supply of 0.8 MW was installed to operate the corresponding periphery, which includes 1 petabyte of data storage just for the supercomputer. The extremely high computing power in such a small space generates excessive heat, placing increased demands on the IT cooling system. To be able to properly cool the supercomputer, DFKI needs an intelligent solution that can meet the requirements of precise, reliable, energy efficient and noise-reducing cooling of the DGX-2.
The main requirement is to provide strong cooling performance in a very small space with relatively low energy consumption. Also, it is important to keep the noise emission from outdoor units to a minimum so as not to disturb local residents. Another important factor is that cooling accounts for a large proportion of the energy consumption at the new data centre. This is why the overall concept was implemented keeping economic operation in mind, meaning that energy consumption should be minimised. To meet the various requirements of cooling the DGX-2, a special, intelligent air conditioning concept with energy efficient and powerful system solutions had to be created.
Greatest efficiency in smallest space
The i-NEXT series by RC IT Cooling is designed especially for high precision air conditioning of servers, utility rooms and computer centres. It provides the ideal ratio of performance to floor space required. It facilitates precise temperature and humidity control as well as humidifying and dehumidifying. Energy saving and noise-optimised EC high performance fans ensure controlled air flow. The fan speed automatically adapts to the thermal load identified by the sensors in the heat and cold aisle. The evolution+ controller controls the activation times of the compressor applying FIFO logic and regulates the generated cooling performance as a factor of the current heat intake. The data can also be transferred to a higher-order monitoring and remote maintenance system.
The climate chamber has two cooling circuits. One of the circuits is a water-cooled DX system. All of the components essential to operation are built into the unit and the controller. The second circuit is designed as a cold-water circuit for free-cooling operation. Only a dry cooler with supply pump has to be added on site to ensure energy efficient operation.
The indirect free-cooling system consists of a smart combination of various operating modes. When it is warm outside, only the DX system runs. As soon as the outdoor temperature falls slightly below the desired cold water flow temperature of the cold water circuit, free-cooling mode runs parallel to the DX system. When the outdoor temperature is sufficiently low, only the free-cooling circuit runs.
"Complete free-cooling" mode
When the outdoor temperature is low enough, cooling is achieved solely with outdoor air in the connected dry cooler, while the compressor is deactivated. Chế độ "free-cooling hoàn toàn"
As soon as the outdoor temperature falls slightly below the desired cold water flow temperature of the cold water circuit, free-cooling mode runs parallel to the DX system. The air is precooled by the cold water heat exchanger in the unit. The compressors are activated as well to provide the cooling lacking for the complete room load.
"Direct evaporation (DX)" mode
Only when the outdoor temperatures are very high is exclusively the DX circuit is active and transporting away the heat. In this mode the dry cooler acts as a heat exchanger for the cooling water circuit.
Intelligent energy management
The two climate chambers are placed to the left and right at the ends of the server room. In the middle of the room there are five racks with the new computer and its peripheral IT components, with an aisle running around it. Server air conditioning is designed applying the down-flow principle to ensure optimum, energy-efficient cooling. Air intake into the climate chamber is from above, while the air outlet flows down and into a double bottom. Outlets in the double bottom on the front of the rack ensure ideal air distribution in the data centre. This creates a cold aisle on the air inlet side of the rack. The racks suck the cold air in and then blow the warm air out the back and into the so-called heat aisle. Then the air is sucked into the climate chamber currently operating, where it is cooled.
To precisely control the cold air supplied to the servers, a cold aisle housing was selected for this project. The heat and cold aisle along with the adjusted air flow ensure that the temperature remains constant. Working conditions become more stable and the entire air conditioning system is more efficient. Most of the time the precision climate chamber runs in hybrid or complete free-cooling mode. In standard mode cooling initially occurs with water via the free-cooling feature. If the heat load exceeds a certain reference point, a compressor is added to additionally cool the air intake through a coolant circuit. If the cooling requirements exceed an additional temperature reference point, the second compressor in the climate chamber switches on to ensure that the temperature level of 21 °C specified by the manufacturer is maintained.
Minimal noise emission in the neighbourhood
The generously dimensioned dry coolers ensure that the third important requirement of this project is met. The noise level was reduced by placing the climate chambers with EC fans, speed-controlled compressors and pump stations in the housing.
The heat exchangers outside have large heat exchange surfaces and bigger fans than the standard model. This means that the fan speeds are lower, resulting in a noise level of no more than 35 dB(A) at a distance of one meter.
Maintaining the proper climate for the new DGX-2 supercomputer at the DFKI poses a steep challenge in regard to IT cooling. First of all, the greater computing power requires powerful precision air conditioning in a limited space. Secondly, the current environmental protection discussion demands a rational allocation of energy. So only concepts and system solutions geared towards high-efficiency criteria can even be considered.
And thirdly, noise emission should be low enough that it does not disturb local residents. Applying know-how and the system solutions provided by RC Cooling, an ultramodern concept for cooling the new DFKI server room was achieved. Water-cooled climate chambers with direct evaporation and full inverter technology as well as the free-cooling feature are used. Energy saving and noise-optimised EC high performance fans ensure controlled air flow through a complete cold aisle housing. Generously dimensioned heat exchangers outside, with low fan speeds, reduce noise to a minimum.