It's Not Computation Speed but 'Data Movement' That Determines AI Performance... Acceleration of Infrastructure Redesign

In the era of Artificial Intelligence (AI), the standard for performance competition is shifting from computational power to data management capability. As Large Language Models (LLM) and Agentic AI expand, how efficiently massive amounts of data are stored and delivered to computing devices has emerged as a key variable that determines system cost and performance.

According to the 'AI Ecosystem' series released by SK hynix, the focus of the current AI industry is moving from the model development stage to the stage of operating large-scale services. As the proportion of the inference stage, where user requests are repeated, grows larger than pre-training, the scale of data to be processed increases exponentially as questions get longer and contexts become more complex. Accordingly, 'infrastructure redesign,' which co-designs computing devices, memory, storage, and networks as a single system to minimize data movement, has become essential.

Data movement costs 2,000 times more than computation

Existing computing systems are designed around the processor (computing device). It is a structure where data moves through storage and memory to the computing device and then repeats the process of returning. While this latency has been mitigated through technologies such as cache layers or Out-of-Order Execution, the limitations are clear in the face of the explosion of AI workloads.

In modern systems, data movement incurs much higher costs than the computation itself. The energy required to read data once from DRAM can be anywhere from 150 to as much as 2,000 times greater than simple arithmetic operations. In actual research utilizing Google's neural network models, it was revealed that more than 90% of the system energy in large machine learning models is consumed by memory access and data movement rather than computation.

In particular, LLMs require a rapid increase in memory capacity and bandwidth due to attention operations and KV Cache (Key-Value Cache) management, which must reference previous context every time a token is generated. No matter how fast the computing device is, if the data does not arrive on time, the efficiency of the entire system plummets.

CXL and NVLink, high-speed interconnects widening data pathways

To solve the data movement bottleneck, 'Memory-Centric Computing' is emerging as an alternative. This is an approach that moves away from the existing method of bringing data to the computing device and instead uses the location and movement cost of data as a design standard, performing computations near the memory or optimizing memory placement.

In this process, high-speed interconnect technology that widens the data pathways between devices plays a key role. CXL (Compute Express Link) expands the connections between CPUs, AI accelerators, and memory devices to flexibly increase memory capacity and provides a foundation for 'memory pooling' where multiple devices can share memory. On the other hand, NVLink and NVSwitch focus on reducing bottlenecks by exchanging large amounts of data at high speeds between GPUs within a GPU cluster. Although the two technologies operate in different areas, their goal of improving system resource efficiency by reducing data movement is the same.

'Near-Memory Accelerators' that process directly near the data

If increasing connection speeds is not enough, a method of performing computations exactly where the data resides is required. This is called an 'interconnected near-memory accelerator architecture.' Instead of a central GPU or NPU pulling all the data, accelerators placed around the memory process the data directly and exchange only the necessary results with the central device.

The utility of such a structure is being proven through academic research. In past Tesseract design cases, when parallel graph processing was performed by connecting near-memory accelerators via high-speed interconnects, it achieved approximately 10 times the performance improvement and a similar level of energy saving compared to existing processor-centric designs.

Recently, research optimized for LLM inference has also been continuing. The CXL-based CENT design has drastically lowered the memory capacity and bandwidth burden of large-scale inference infrastructure by combining memory expansion structures with near-memory processing. Ultimately, the success of future AI infrastructure is expected to depend on how intelligently data and computing devices are placed and connected, moving beyond the speed competition of specific components.

Source: news.skhynix.co.kr