AI infrastructure design shifts from individual servers to rack-scale systems

The scope of AI infrastructure design is expanding from individual servers to entire racks and data centers to manage complex requirements for power, cooling, and data movement. This shift toward "co-design" is driving memory companies like SK hynix to participate in customer system roadmaps from the early stages of development.

As AI technology accelerates, the perspective on AI infrastructure is undergoing a fundamental shift. While focus was previously placed on the performance of individual AI accelerators or servers, the core challenge has emerged as the design of entire systems where compute, memory, storage, network, power, and cooling operate together like a single organic entity.

According to 'AI Infrastructure Insight' released by SK hynix, the unit of AI infrastructure design is expanding beyond individual servers to the rack level, and further to the entire data center. Because performing large-scale AI training and inference requires more than a single accelerator, numerous accelerators must be closely connected to distribute tasks. Consequently, the 'Rack-scale System,' which manages resources within a rack as a single structure, is emerging as a representative design model.

AI infrastructure design scope widening from servers to racks

A Rack-scale System is a method of designing CPU, GPU, memory, and network, as well as power and cooling conditions, as an integrated unit at the rack level. A rack is evolving beyond a mere space to house servers into a system-level unit that integrally manages data movement, power supply, and thermal management.

As the scope of design widens, the variables to be considered have become more complex. As the scale of an AI cluster grows, communication efficiency between racks becomes a critical factor determining overall performance. Depending on the physical placement of racks, connection distances, and network structures, data transmission delays can occur. Furthermore, to handle the massive heat generated in high-density racks, cooling facilities and the physical structure of the data center must be designed in alignment from the initial design stage. A distribution structure for stable power supply is also an essential design element.

The appearance of infrastructure varies completely depending on the specific task being performed. While large-scale model training requires high throughput, memory bandwidth, and fast connections between accelerators, the inference stage prioritizes low latency and power and cost efficiency. As workloads such as 'Agentic AI'—which continuously reasons and repeatedly accesses external tools to achieve user goals—increase, the ability to manage data flow moving between memory, storage, and networks becomes a performance variable.

Paradigm shifting from 'post-assembly' to 'co-design'

As the technologies constituting infrastructure become closely interconnected, the traditional 'post-connection' method has faced limitations. It is difficult to meet the requirements of high-density AI systems by designing individual components separately and integrating them only at the final stage. Accordingly, the 'co-design' method, which considers the conditions of computing platforms and data center infrastructure together from the early stages of development, is spreading.

The scope of co-design now includes physical infrastructure areas such as power and cooling. In high-density AI systems, liquid cooling or thermal management methods are directly linked to the requirements of the computing system. This trend is leading to standardization discussions across the industry. This is the reason for movements to standardize the power structures and interfaces of next-generation data centers, centered around the global collaborative community OCP (Open Compute Project).

While standardization establishes the foundation for interoperability, companies carry out customized optimization based on those standards. In the memory field, while following the basic specifications set by JEDEC, highly advanced optimization work tailored to specific system requirements is carried out in parallel.

Role of memory companies like HBM expanding to 'early participation in customer roadmaps'

The change in design methods is fundamentally altering the roles and collaboration models of memory companies. As the scale and speed of data processed by accelerators increase, the required bandwidth, capacity, and heat dissipation conditions for memory have become different for every system. Accordingly, there is an increasing trend of determining memory specifications together from the platform conception stage.

A representative case is HBM (High Bandwidth Memory). Regarding next-generation HBM, discussions on 'custom HBM'—which directly reflects the requirements of customer systems, centered on the 'Base Die' responsible for data input/output under the stacked DRAM chips—are gaining momentum. This signifies that memory is evolving beyond the stage of supplying general-purpose products to becoming a partner deeply involved in the system design of customers.

In response to these changes, SK hynix is adopting a strategy to participate in the development process of customers' next-generation AI systems from the early stages. This involves preemptively identifying the system roadmaps of customers and defining the necessary memory conditions accordingly. Ultimately, the competition for future AI infrastructure is expected to be decided by how precisely the entire system structure is coordinated and how quickly companies can respond to the design directions of customers, moving beyond the performance of individual components.

Source: news.skhynix.co.kr