Intel has released containerised, pre-tuned LLM serving for Xeon CPUs. It only targets 4th Gen Xeon (Sapphire Rapids) and newer, so the generation of an existing server fleet now decides whether CPU inference is a supported option or a do-it-yourself project.
What was announced
- Intel Inference Microservices are container images that each serve one large language model behind an OpenAI-compatible API, on port 8000 by default, with chat completions, completions, models, health and Prometheus metrics endpoints.
- At startup the container detects processor generation, available cores, NUMA layout and whether AMX, AVX-512, AVX2 and VNNI are present, then applies a configuration Intel has tuned and validated for that model on that silicon. The serving engine in this release is vLLM, running at BF16.
- Supported platforms are 4th Gen Xeon (Sapphire Rapids), 5th Gen Xeon (Emerald Rapids) and Xeon 6 with both performance cores (Granite Rapids) and efficient cores (Sierra Forest). Other Xeon CPUs are reported as generic.
- Model-specific images refuse to start on a host with no matching profile. Only the base image, which can serve any Hugging Face model, falls back to a general configuration without model-specific tuning.
- Intel's sizing guidance is roughly 2GB of memory per billion parameters at BF16 before KV cache: 32GB for 2B to 9B models, 48GB for 10B to 16B, 64GB for 17B to 24B and 96GB for 25B to 30B, with mixture-of-experts models sized on total, not active, parameters. It also suggests at least 50GB of disk for the model cache.
- Deployment is Docker Engine 20.10 or later on a single host or Kubernetes 1.24 or later. A Helm chart, a Kubernetes Operator, more engines and more precisions are planned, and Intel says the same interface will extend to its GPUs, including the Crescent Island datacentre inference GPU.
- The first images are tagged version 0.1.0, and tool calling and structured output are passed through to vLLM rather than validated.
The ETON view
This is the most practical thing Intel has done for CPU inference in a while, because it removes the week of tuning that usually stops a small model ever reaching production on a general-purpose server. For enterprises and hosting providers with spare Xeon capacity, an internal assistant, document classifier or RAG front end on a 4B to 30B model becomes a scheduling decision rather than a GPU purchase.
The catch is the generation line. Only Sapphire Rapids and newer are supported, which in practice means Dell PowerEdge 16G and 17G, HPE ProLiant Gen11 and Gen12 and Lenovo ThinkSystem V3 and V4 class servers. The large installed base of 2nd and 3rd Gen Xeon Scalable in 14G and 15G PowerEdge, Gen10 and Gen10 Plus ProLiant and ThinkSystem V1 and V2 servers is not on the list, and the tuned model images will not start there. That makes this a sensible point to plan which hosts get refreshed for AI work and which stay on conventional duties.
Memory is the second check. Intel's own sizing puts a 30B model at 96GB before long contexts or many concurrent users, so on a dual-socket server the RDIMM population, channel balance across both NUMA nodes and headroom for KV cache matter more than core count. Treat CPU serving as right for low-concurrency, small-model work; once concurrency or model size climbs, a single PCIe inference GPU in the same chassis is usually the cheaper route to acceptable latency.
We can supply Sapphire Rapids, Emerald Rapids and Xeon 6 servers from Dell, HPE, Lenovo and Supermicro, matched RDIMM upgrades for existing hosts, and PCIe GPUs for when CPU inference runs out of road, with like-for-like quotes on price and lead time.
Related infrastructure
Category: AI & GPU · Vendor: Intel · Technology: Xeon 4th/5th/6th Gen, Intel AMX, vLLM, CPU LLM inference, OpenAI-compatible API, Kubernetes · Last verified: 18 Oct 2026 · ~3 min read
Sourcing this kind of infrastructure? Talk to ETON about availability, lead time and pricing.
