I

Inference Chips

Inference chips are processors designed to run already-trained AI models in production, answering each request quickly and at a low cost per token, rather than to train new models.

What is an inference chip?

An inference chip is a processor built to run a trained AI model, not to train one. Inference is the work done every time a model answers a prompt, so these chips are tuned for low latency per request, high throughput and a low cost per token. Examples include Groq’s LPU, Cerebras’s wafer-scale engine, AWS Inferentia and Google’s TPUs.

What it is

Inference chips are processors purpose-built for the runtime side of AI workloads. Where training a frontier model requires massive parallel compute over weeks (NVIDIA H100 and B200 GPUs dominate this), serving the trained model to users in production has very different requirements: low latency per request, high throughput on small-batch decoding, predictable cost per token, and tolerance for the long tail of model architectures that have already been trained. Vendors in this space include Groq (LPU — language processing unit, built around static dataflow for high tokens-per-second on transformer decode), Cerebras (wafer-scale chips that fit large models entirely on-die), SambaNova, AWS Inferentia/Trainium, Google TPU v5e and v6, Etched (a transformer-specific ASIC), and a growing roster of startups. Some vendors target both training and inference; the "inference chips" framing specifically highlights the runtime-economics use case.

Why it matters

Inference cost is the dominant ongoing operating cost of a deployed AI product. A frontier model trained once might cost $100M, but the serving infrastructure that runs it for a million users a day can cost more than that within twelve months. Inference chips matter because they shift the unit economics: a 5–10× improvement in tokens-per-dollar at inference compounds across every customer interaction for years. The category also matters strategically — if specialized inference silicon delivers materially better economics on the long-tail of agentic workloads (many short calls, often interrupted by tool use), the cloud-vendor matrix that dominated 2023–2025 (NVIDIA on AWS / Azure / GCP) starts splintering. For agent operations leaders, inference chip choice flows directly into the model selection and routing decisions that the capability registry makes — a model that's 4× cheaper on Groq than on a generic GPU may be the right pick even if it's a half-step behind on quality.

Key components

  • GPUs vs purpose-built — general-purpose NVIDIA dominant in training; inference-specific silicon increasingly competitive at runtime
  • Tokens per second — the headline metric for inference chip benchmarks (Groq famously hit 500+ tokens/sec on Llama 70B)
  • Tokens per dollar — the unit-economics metric that matters for production deployments
  • Vendor landscape — Groq, Cerebras, SambaNova, AWS Inferentia, Google TPU, Etched, and others
  • Interaction with routing — inference chip choice affects which models are economical for which workloads in a vendor-neutral stack

How it connects

Agentforce agents make dozens of model calls per workflow — summarizing cases, drafting emails, routing leads — so the chip powering those calls directly affects both response speed inside Salesforce and the per-interaction cost that shows up on your AI budget. If you're working with a model provider that runs on specialized inference hardware, you may get materially faster Einstein or third-party model responses without changing anything in your Salesforce configuration.

Good to know

You rarely choose an inference chip directly — you choose a model provider or API endpoint, and the chip is their infrastructure decision. The practical lever for most Salesforce teams is comparing cost-per-thousand-tokens and response latency across providers before committing a model to a high-volume Agentforce flow.

Frequently asked questions

What is the difference between an inference chip and a GPU?

A GPU is a general-purpose parallel processor that handles both training and inference, and it dominates training. An inference chip is specialised for serving a trained model, trading flexibility for faster responses and lower cost per token. Some chips, such as Google’s TPUs, are used for both.

Why do inference chips matter for AI costs?

Most of what a business pays for AI in production is inference, charged per token. Faster, cheaper inference hardware lowers that price and makes real-time uses, such as voice agents and multi-step agent workflows, practical.

Need Help Implementing This?

We specialize in putting AI and Agentforce to work for Salesforce customers. Let's talk about your use case.

Book Intro Call