AMD (AMD 3.20%) discussed plenty of new and upcoming products at CES, including data center GPUs meant to compete with Nvidia, its Helios rack-scale AI platform, and new PC CPUs. While the high-powered MI440X GPU and the upcoming MI500 Series GPUs got much of the attention, there was one product that is far more interesting, and it points to a significant evolution for the AI industry.
Image source: AMD.
AI is going local
Running AI inference in the cloud is expensive, and spending is exploding even as token costs decline. Running an AI model performing at the level of GPT-3.5 has dropped in price over the past two years 280-fold, according to a report from Stanford . However, AI agents and reasoning models, which produce better results and can perform more complex tasks, generally require far more tokens.
Small, focused, and cheap large language models are often enough for many types of tasks. As hardware improves and AI models become more efficient, it becomes feasible to run increasingly sophisticated AI workloads on local devices rather than in the cloud.
Deloitte has recently outlined a framework for enterprises to determine how and where to run AI inference workloads. For highly variable workloads that require top-tier AI models or experimentation, cloud hyperscalers are the most suitable option. For consistent and predictable workloads with sensitive data requirements that require minimal latency, on-premises hardware makes the most sense. And for real-time processing with smaller models, edge devices, including PCs, are the answer.
AMD's Ryzen AI Halo platform, announced at CES and available in the second quarter, targets that final use case. The Ryzen AI Halo is a mini-PC with a 16-core CPU, 128GB of unified memory, an integrated AI processor, and a graphics chip that adds additional AI firepower. All told, the device delivers up to 126 TOPS of AI processing power. The Ryzen AI Halo isn't designed for end users, but rather for those developing AI applications.
The amount of memory included makes it possible to run fairly large AI models on the Ryzen AI Halo, although nothing close to the top-tier models offered by OpenAI, Anthropic, or other AI providers. Still, various open-source AI models that are capable of complex tasks will be able to run on this machine.
This points to a very different future for AI
The Ryzen AI Halo won't be a big seller. It's a development platform, not a consumer device, and will be expensive. However, it's an important product for AMD because it provides a preview of how the AI industry is likely to evolve.
It doesn't make much sense to assume that all AI inference workloads are going to run in the cloud indefinitely. As devices include more memory and processing power, and as AI models become more efficient, it's inevitable that some AI inferencing can be done locally.
Today, code assistants like Claude Code are changing how programming is done. How long will it be before a laptop can run an AI model powerful enough to be used for a coding assistant with similar capabilities? Three years? Maybe four? It's impossible to say, but it seems like an inevitability.
NASDAQ: AMD
Key Data Points
AMD is well-positioned for this shift. The company's Ryzen AI 400 series CPUs deliver 60 TOPS of AI performance, and systems start shipping this month. Meanwhile, the Ryzen AI Max+ line of CPUs, which are used in the Ryzen AI Halo, supports 128GB of memory and can run an AI model with 128 billion parameters.
AI PCs aren't very useful yet. They don't have enough AI processing power, and they'll need far more memory. The ongoing memory chip shortage poses a problem for now. But eventually, the idea of using a cloud AI service for many tasks on a PC will seem crazy. Running AI locally incurs no ongoing costs, keeps data on the device, and can dramatically reduce latency.
AMD's Ryzen AI Halo is a niche product, but devices powerful enough to run capable AI models locally won't be niche forever. AMD is battling Nvidia in the data center, but it's also positioning itself to be a winner in the next phase of the AI revolution.
