Managed inference and serverless GPU
Call a model or run a function and pay per token, per call or per second, without renting a machine.
Definition, not a sourced statistic
What it is
Managed inference runs models for you behind an API, so you send a request and get an answer without managing any server. Serverless GPU platforms run your own code or container on accelerators only while it works and scale down when idle. Billing is by token, by call or by short slices of time.
Who it suits
- Applications that serve a model to users and want to scale with traffic.
- Developers who would rather call an API than manage machines or quota.
- Uneven or bursty work where paying for idle machines would waste money.
Common contract shapes
- Per token or per call. Pay for what each request uses, with no machine to rent.
- Per second of execution. Your container runs on accelerators only while it works.
- Dedicated deployment. A model deployed on capacity set aside for you, billed by time.
- Reserved throughput. A committed amount of capacity or traffic for a term.
Providers use their own names for these shapes, so compare what is promised, not what it is called. The contract shapes guide explains each one.
What to ask
- Egress and data transfer. Are responses, logs or stored outputs charged for transfer?
- Service levels. What latency and availability are promised, are there rate limits, and how are they raised?
- Data residency and access. Where are requests processed, are prompts and outputs stored or used for training, and can you switch that off?
- Security certifications. Which certifications cover the inference service itself, and is there a trust centre or report to read?
- Export controls and sanctions. Are some models or regions restricted for your country or use, and who is responsible for the model licence?
- Exit. Can you run the same open model elsewhere with the same results, or are you tied to a proprietary format or API?
- What happens to a cold start, the delay when an idle model or function wakes up?
The full list is on the printable buyer's checklist. For rules on chips and where they can go, see export controls and compute thresholds.
On this site
Entries listed here because their own summary or checked facts on this site fit this type. An entry can fit more than one type, and a provider missing here may still offer it. This is a list, not a ranking, and in name order. We link to providers for prices.
Providers
- Baseten: An inference platform for deploying open-source and custom models, with managed cloud, self-hosted and hybrid options.
- Cerebras: A maker of wafer-scale AI systems that also sells cloud inference, dedicated capacity and on-premises systems.
- Cloudflare Workers AI: A serverless service that runs open-source machine learning models on GPUs across Cloudflare's network, with no servers to manage.
- Fireworks AI: A training and inference platform for open models, with serverless, on-demand and reserved deployment tiers.
- Groq (GroqCloud): An inference provider built on its own Language Processing Unit chips, offering GroqCloud for developers.
- Hugging Face Inference Providers: A unified API that routes requests for hosted models to a range of partner inference providers. Dedicated Inference Endpoints are a separate service.
- Koyeb: A serverless platform for deploying containers and AI workloads on GPUs, CPUs and accelerators across distributed locations.
- Modal: A serverless platform for running AI inference, training, sandboxes and functions on GPUs and CPUs spread across multiple clouds.
- Nebius: An AI cloud offering GPU compute with InfiniBand networking, plus managed inference and MLOps tooling.
- Replicate: A platform and API for running, fine-tuning and deploying machine learning models, including community-published open models.
- RunPod: A GPU platform offering pods, serverless GPU endpoints and multi-node clusters from a shared catalogue in many regions.
- SambaNova: An AI hardware and inference company that runs its own RDU chips, sold as racks and offered as a cloud inference service.
- Together AI: A platform for running open models through an inference API, with fine-tuning and dedicated GPU clusters for larger jobs.
- Verda: A cloud platform for AI workloads, offering GPU instances, InfiniBand clusters and serverless GPU containers. Storage options include block storage and a shared filesystem.
Programmes
- Hugging Face ZeroGPU Global
Related
- Glossary: Training and inference, SLA (service level agreement), Data residency.
- Guide: How to get GPUs this week
- Provider directory
- Compare offer types
Other offer types: Hyperscale cloud instances · GPU clouds and bare metal · Marketplaces and aggregators · Decentralised and community networks · Regional and sovereign clouds · Chip-maker and accelerator clouds · National and institutional HPC, and free or funded programmes · Colocation, wholesale, powered shell and build-to-suit · On-premises hardware, and leasing or financing it · Capacity brokers, reserved contracts, secondary and spot markets · Edge and telecom sites, and converted sites
Definitions and questions written in our own words, checked on 3 October 2026 and due for review by 3 January 2027. Sourced facts stay on the provider, programme and policy pages this page links to. This is general information to help you ask better questions. It is not legal, tax or procurement advice. For export controls, sanctions and contracts, check the primary rules and talk to a qualified adviser. Something wrong? Tell us.