[Hardware]Qualcomm’s New AI Rack-Scale Solutions Actually Uses LPDDR Mobile Memory Onboard, Boldly Hoping to Take on NVIDIA and AMD

[-CosmiNNe]

Qualcomm has announced its latest AI chips, which are designed to scale up to a purpose-built rack-level AI inference solution, but interestingly, they employ mobile memory onboard.

Qualcomm's New AI Chips Take a 'Daring' Pivot Away From HBM To Target Efficient Inferencing Workloads
Qualcomm has come a long way from being a mobile-focused firm, and in recent years, the San Diego chipmaker has expanded into new segments, including consumer computing and AI infrastructure. Now, the firm has announced its newest AI200 and AI250 chip solutions, which are reportedly designed for rack-scale configurations. This not only marks the entry of a new player in a segment dominated by NVIDIA and AMD, but Qualcomm has managed to find a unique implementation by utilizing mobile-focused LPDDR memory.

Before we delve into the specifics of the newly announced AI chips, let's examine the use of LPDDR memory compared to the more traditional HBM solution. Qualcomm's new products offer up to 768 GB of LPDDR on the accelerator package, which is significantly higher than the industry's HBM capacity. The main reason this venture looks attractive is that it reduces data-movement energy and cost, a key advantage that the firm calls a "near-memory" approach. Here are the traditional improvements the firm gets by employing LPPDR over HBM:

Power Efficiency (Lower Draw per Bit)
Cheaper relative to modern-day HBM modules
High Memory Density (Ideal for inferencing)
Thermal Efficiency due to less heat dissipation versus HBM
While this implementation sounds optimistic, Qualcomm's rack-scale solutions still fall short when compared to mainstream options from NVIDIA/AMD, simply because avoiding HBM use results in lower memory bandwidth, higher latency due to a narrow interface, and, most importantly, utilizing an immature memory stack in 24/7 high-heat server environments. However, the intention of the San Diego firm here is to provide companies with a capable inferencing option, and the use of LPDDR certainly achieves this goal, but it does limit these rack-scale configurations to a specific application.

Apart from this, the AI200 and AI250 chip solutions feature direct liquid cooling, PCIe/Ethernet protocols, and a rack-level power consumption of 160 kW, which is a pretty low figure for a modern-day solution. More importantly, the chips onboard employ the firm's Hexagon NPUs, which are widely expanding in terms of inferencing capabilities, supporting advanced data formats as well as inference-focused features.

Interestingly, the pivot towards bringing capable inferencing solutions to the market is being done by a lot of compute providers, with one of the more recent examples being Intel with its 'Crescent Island' solution and NVIDIA introducing a new Rubin CPX AI chip. Qualcomm apparently recognizes that the inferencing segment is gaining market spotlight, which is why the AI200 and AI250 chip solutions are a sensible approach here. However, for modern training or large workloads, these racks would probably be the last choice.

It's exciting to see competition emerging in the AI space, and by the looks of it, retailers took the announcements with quite some optimism.

https://wccftech.com/qualcomm-new-ai-rack-scale-solution-actually-uses-lpddr-mobile-memory-onboard/