AI infrastructure design shifts toward rack-scale systems and co-design

The focus of AI technology competition is expanding from individual accelerators to rack-scale systems, where SK hynix reports that co-design is required to optimize memory, networks, and cooling as a single unit. This shift toward integrated design aims to address the complex data flow needs of emerging technologies like Agentic AI.

The axis of competition in AI technology is expanding beyond the performance of individual accelerators or models to the design and operation of the entire infrastructure supporting them. This is because, beyond simply securing fast computing devices, memory bandwidth, networks, storage, and the power and cooling systems required to maintain them must function as a single organic structure for AI to operate in actual industrial fields.

According to 'AI Infrastructure Insight' released by SK hynix, the core of next-generation AI infrastructure design lies in 'co-design,' which views the rack or the entire data center as a single unit to optimize all elements, moving beyond individual server units. This method optimizes from a total system perspective by considering the requirements of different technologies or systems from the early stages of development.

Design Scope Expanding from Servers to Rack Units

The design unit of AI infrastructure is rapidly moving from existing individual servers to 'rack-scale systems.' This is because large-scale AI training and inference cannot be processed by a single accelerator or server alone; instead, numerous accelerators must be closely connected to share tasks. Accordingly, methods are being introduced to connect multiple servers and accelerators within a rack via high-speed interconnects and to integrate design from memory configuration to power and cooling conditions at the rack level.

The scope of design does not stop there and extends to the entire data center. As the scale of AI clusters grows, communication efficiency between racks and physical placement have a decisive impact on system performance. The placement of racks, connection distances, and network structures determine data transmission latency, and cooling facilities to handle the massive heat generated in high-density racks, along with stable power distribution structures, must also be designed alongside the physical structure of the data center.

Customized Infrastructure Based on Workload Characteristics

Another criterion for infrastructure design is the nature of the 'workload' to be processed, as the same structure cannot be applied to all AI tasks. In the case of large-scale model training, high throughput, memory bandwidth, and fast connectivity between accelerators are top priorities because numerous accelerators must perform parallel computations for long periods. Conversely, in the inference stage, low latency, data access speed, and power and cost efficiency become the primary design conditions.

In particular, with the emergence of technologies such as 'Agentic AI,' which plans and performs tasks based on user goals, the requirements for infrastructure are becoming even more complex. Since Agentic AI must repeatedly access external tools and data during continuous reasoning processes, the ability to smoothly handle data flow moving between memory, storage, and networks becomes a variable that determines actual performance.

Changing Role of Memory Companies, Evolving into 'Custom HBM'

As infrastructure components become closely intertwined, the role of memory companies is also changing. While the previous method involved combining finished components after the fact, there is now a growing trend of determining memory specifications from the early stages to match the performance and power characteristics of accelerators. A representative example is High Bandwidth Memory (HBM). For next-generation HBM, discussions regarding 'custom HBM'—which reflects the requirements of customer systems centered around the base die beneath the stacked DRAM chips—are gaining momentum.

Following this trend, memory companies are expanding collaboration by identifying customer system roadmaps early and participating from the design stage to jointly define necessary memory conditions. SK hynix is also strengthening cooperation by participating in the development process of customers' next-generation AI systems from the early stages. The capability to preemptively understand the direction of customer systems and be deeply involved in the design is expected to be a differentiation point in the future AI infrastructure market.

Source: news.skhynix.co.kr