How Fireworks AI maps to Runpod
Fireworks is a model platform: you pick a model ID and Fireworks runs the inference stack. Runpod is a compute platform with a hosted model catalog on top. On Runpod, you either call a model Runpod already hosts, or you run an open-source inference server (vLLM) on GPUs you configure.What’s different
What works differently
You choose the model weights and the serving flags. A Fireworks model ID hides the checkpoint, precision, chat template, and tool-call parser. On a vLLM endpoint, you pick the Hugging Face repo and set those yourself with environment variables. Output can differ from Fireworks for the same model family if the quantization or chat template differs, so compare outputs before you cut over. Billing moves from tokens to time for your own endpoints. Fireworks Serverless bills input, cached input, and generated tokens. A Runpod Serverless endpoint bills per second while a worker runs, which includes the cold start, request processing, and the idle timeout. Public Endpoints bill per use (per token for text models). A workload with steady traffic and high GPU utilization behaves differently from a spiky, low-volume one, so estimate from your own request logs. A cold endpoint queues requests instead of rejecting them. A Fireworks deployment scaled to zero returns503 with DEPLOYMENT_SCALING_UP and doesn’t queue the request, so clients retry. A Runpod queue-based endpoint holds the request in its queue while a worker starts. The trap is the synchronous path: /runsync and the /openai/v1 route sit behind an edge timeout of about 100 seconds, so a first request that waits on a large model load can return a 524. Use /run plus /status for the first request, raise the idle timeout, or keep an active worker.
Scale-down happens much sooner by default. Fireworks’ default scale-to-zero window is 1 hour (minimum 5 minutes). Runpod’s console default idle timeout is 5 seconds, and the maximum is 3,600 seconds. Set it deliberately. Workers are billed while they wait.
Autoscaling signals are different. Fireworks scales on load targets such as concurrent_requests or tokens_generated_per_second. Runpod scales on queue delay (seconds a request waits) or request count (queued plus in-progress requests divided by a target). The vLLM worker’s MAX_CONCURRENCY (default 30) sets how many requests one worker takes at once.
Idle resources are cleaned up differently. Fireworks deletes a deployment with min replicas of 0 after 7 days without traffic. Runpod reduces an idle endpoint’s max workers to 2 after 3 days and to 0 after 7 days, and keeps the endpoint. Raise max workers to use it again.
Model names change. accounts/fireworks/models/llama-v3p1-8b-instruct becomes the Hugging Face repo the worker serves, for example meta-llama/Llama-3.1-8B-Instruct. Set OPENAI_SERVED_MODEL_NAME_OVERRIDE to serve it under a name of your choice.
Reasoning output uses a different field. Fireworks returns reasoning in reasoning_content. vLLM v0.32.0 (the version the current vLLM worker release is built on) returns it in reasoning. Read both fields in your client. Reasoning also needs REASONING_PARSER set on the worker.
Fireworks request extensions don’t carry over. Remove service_tier, context_length_exceeded_behavior, perf_metrics_in_response, and the x-session-affinity header unless you’ve confirmed the target accepts them. Fireworks lowers max_tokens when a request would exceed the context window. On Runpod, set max_tokens explicitly. The Runpod vLLM docs list a default of 16.
Token usage in streams. Fireworks includes usage in the last streamed chunk by default. If your code reads it, test that path against your Runpod endpoint.
Rate limits are per endpoint, not per model. Fireworks Serverless enforces adaptive tokens-per-minute limits per model. Runpod enforces request-rate limits per endpoint and operation (see rate limits). Throughput on your own endpoint is bounded by your workers.
What has no direct equivalent
Going the other way, Runpod gives you things Fireworks deployments don’t: any container image and any inference engine, SSH access to Pods, network volumes shared between training and serving, a job queue with webhooks and
/status polling, and a batch jobs API while Fireworks’ Batch API is paused.
Requirements
- A Runpod account with credits and a Runpod API key.
firectlsigned in to the Fireworks account you’re migrating (firectl whoamishows the account).- A Hugging Face access token if any model is gated or private.
- Your Fireworks request logs or billing export, to size GPUs and estimate cost.
Step 1: Take inventory of what you run on Fireworks AI
List every resource in the account. Add--output json (or -o json) to capture the full configuration.
replicaStats.readyReplicaCount shows how many replicas are serving.
Serverless usage has no resource to list, so find it in code. This command finds SDK imports, base URLs, model IDs, and environment variables.
/chat/completions, /completions, /embeddings, /rerank, /responses, or the Anthropic-compatible /v1/messages), and which features it uses: streaming, tools, response_format, vision input, reasoning_effort, or service_tier. Gateways such as LiteLLM reference Fireworks as fireworks_ai/MODEL, so check their config files too.
Fireworks’ docs say the Batch API is paused and that stored Batch input datasets and job outputs will be permanently deleted. If you have Batch data there, download it now with
firectl dataset download DATASET_ID --output-dir ./data.Step 2: Set up Runpod
Create an API key on the API keys page, then install the CLI and export the key.runpodctl doctor stores the key and verifies the setup. You need runpodctl 2.4.0 or later for --model-reference. Check with runpodctl version.
Inference clients keep using the OpenAI SDK, so there’s nothing else to install for them. If you also want to build custom workers in Python, install the Runpod SDK with pip install runpod (see the SDK overview). To let a coding agent manage resources, connect the Runpod MCP server.
Step 3: Choose where each workload lands
Use this table to place each item from your inventory.Models that Runpod hosts
Build your model map from the Public Endpoints model reference, which lists image, video, audio (text-to-speech), and text models with their prices. These are the text models in the model reference today. Confirm that a model is live in the Hub before you map a call site to it.
The OpenAI-compatible base URL for a Public Endpoint is
https://api.runpod.ai/v2/SLUG/openai/v1. Every Public Endpoint also accepts the native routes: https://api.runpod.ai/v2/SLUG/runsync and /run.
For Fireworks model IDs that match these, such as accounts/fireworks/models/kimi-k3, compare the model card on each side before you assume they’re the same checkpoint and precision. For everything else, the general answer is your own vLLM endpoint: Fireworks’ serverless catalog is larger than Runpod’s Public Endpoints catalog.
GPU mapping
Fireworks accelerator types translate to Runpod GPU pools (for Serverless) or GPU type IDs (for Pods) as follows. IDs come from the GPU types reference.
You aren’t limited to the hardware Fireworks used. A model that Fireworks served on an 80 GB card at FP8 may fit a 48 GB pool (
AMPERE_48, ADA_48_PRO) or a 24 GB pool (AMPERE_24, ADA_24) depending on its size and quantization. Pick the GPU that fits the model and your latency target, and list more than one pool so workers have a fallback. Stock varies by data center, so check it before you commit to a pool.
Step 4: Move your images and code
Most Fireworks users have no container images to move, because Fireworks runs the inference server. On Runpod, the prebuilt vLLM worker fills that role, so this step is about translating serving configuration into environment variables. The vLLM worker turns anyvllm serve flag into an environment variable in upper snake case (--max-model-len becomes MAX_MODEL_LEN). VLLM_EXTRA_ARGS passes flags that have no variable.
Check the model’s Hugging Face README and the model-specific settings for the parser and tokenizer values your model family needs.
If the model needs a vLLM version newer than the Hub worker ships, use the upstream
vllm/vllm-openai image on a load balancing endpoint instead (see Step 6). Any image you build yourself must be linux/amd64.
If you run workloads vLLM doesn’t serve (embeddings with a different server, rerankers, image models), you write a worker. A queue worker is a Python handler started with runpod.serverless.start({"handler": handler}). See handler functions and deploy a worker.
Step 5: Move your data and models
Base models from the Fireworks library
If you deploy a Fireworks-hosted model (accounts/fireworks/models/NAME), don’t move weights. Find the matching Hugging Face repo and reference it. Runpod caches the model on worker hosts.
Custom models and fine-tuned weights
Download each custom model and adapter from Fireworks.firectl model download too if you aren’t sure, or ask Fireworks which base was used.
To serve a fine-tune as a single model, merge the adapter into the base. This follows the PEFT recipe in Fireworks’ own docs.
merge.py
- Hugging Face repo (recommended)
- Network volume
- Bake into the image
Push the merged model (or the adapter) to a private Hugging Face repo. The endpoint then uses cached models, which gives the fastest cold starts and no data-center pinning.Create the repository first.
hf upload creates a missing repository for you, and a repository created that way isn’t private unless you ask for it.Reference it in Step 6 with --model-reference https://huggingface.co/YOUR_ORG/my-model:main and an HF_TOKEN that can read the repo.LoRA adapters without merging
To keep adapters separate (the equivalent of Fireworks multi-LoRA), push each adapter to Hugging Face and load it by name.LORA_MODULES takes a JSON list.
model to the adapter name (support-adapter) instead of Fireworks’ MODEL#DEPLOYMENT form. Set MAX_LORA_RANK to at least your adapter’s rank (the default is 16), and MAX_LORAS to the number of adapters used in one batch.
Datasets and training
Fireworks training datasets use the OpenAI chat format (messages with role and content), which most open-source trainers read directly. Download them, then train on a Pod that writes to a network volume.
Trainer, or Axolotl), write checkpoints under /workspace, then delete the Pod with runpodctl pod delete POD_ID. The volume keeps the adapter. See connect to a Pod and the fine-tuning overview.
Runpod has no managed counterpart to Fireworks’ Serverless Training, reinforcement fine-tuning, or DPO jobs. You bring the training loop, and a Pod bills per second until you stop or delete it.
Step 6: Deploy on Runpod
Public Endpoint
There’s nothing to deploy. Confirm the model works for your prompts in the Hub playground, then go to Step 7.vLLM queue endpoint from the Hub
This is the default target for on-demand deployments and for serverless models Runpod doesn’t host. The before and after below deploy Llama 3.1 8B Instruct with scale to zero and up to three replicas.- Before: Fireworks AI
- After: Runpod
--hub-idtakes the listing ID thatrunpodctl hub search vllmreturns. ReplaceHUB_IDwith the ID from your own search output.--gpu-idaccepts a pool ID or a GPU type ID. Always pass it. Without it, the endpoint uses the Hub release’s default pools, which may be larger GPUs than you need.--model-referenceattaches the cached model. It takes the full Hugging Face URL with a ref (:main, a tag, or a commit), and it’s GPU only. Each endpoint supports one cached model. KeepMODEL_NAMEpointed at the same repo.--idle-timeout 600mirrors the 10-minute scale-to-zero window. Lower it to pay for less idle time, or raise it (up to 3,600) to avoid cold starts.--scale-by delay --scale-threshold 4adds workers when requests wait longer than 4 seconds. Request count scaling (--scale-by requests) is the closer match to aconcurrent_requestsload target and is the recommended scaler for LLM workloads, but REST API v2 rejects an idle timeout on a queue-based endpoint that scales on request count. This example keeps the long idle timeout, so it scales on queue delay. If you prefer request count scaling, drop the idle timeout and use active workers to stay warm.--workers-minis the number of active workers. Set it to 1 or more to mirror--min-replica-countabove 0. Active workers bill continuously.HF_TOKENreferences a Runpod secret namedhf_token. Llama models are gated, so create the secret first. Public models don’t need a token.--envapplies with--hub-id. With--template-id, environment variables come from the template.
vLLM load balancing endpoint (alternative)
Use this when you need a specificvllm/vllm-openai tag or want requests to reach vLLM directly with no queue. Create it with REST API v2. The args string holds the vllm serve arguments, and the three environment variables tell Runpod where the health check lives.
https://ENDPOINT_ID.api.runpod.ai/v1, and the gateway checks your Runpod API key, so don’t also set a vLLM API key. This image is a CUDA 13 build, which is why the request sets minCudaVersion. Weights download to the container disk, so size disk for the model plus headroom. Load balancing endpoints don’t have a queue, /run, webhooks, or batch jobs.
Translate the rest of your deployment settings
Step 7: Update the code that calls Fireworks AI
For chat and text completions, three values change: the base URL, the API key, and the model name.
Put the base URL and model name behind configuration so you can switch back during cutover.
Python (OpenAI SDK)
- Before: Fireworks AI
- After: Runpod vLLM endpoint
- After: Runpod Public Endpoint
max_tokens. Qwen3 is a reasoning model, so a small max_tokens can end the response inside its thinking block.
Python (Fireworks SDK)
The Fireworks SDK (from fireworks import Fireworks) exposes the same chat.completions.create call shape, so inference code moves to the OpenAI SDK with a changed import and constructor.
- Before: Fireworks SDK
- After: OpenAI SDK on Runpod
fireworks-ai with openai in your dependencies once no code imports it. Code that uses the Fireworks SDK for platform automation (client.deployments.create(...), datasets, training jobs) moves to runpodctl or REST API v2. See the API map.
TypeScript
- Before: Fireworks AI
- After: Runpod
@runpod/ai-sdk-provider package works with Public Endpoints and with your own endpoint IDs.
curl
- Before: Fireworks AI
- After: Runpod
Tool calling
The request and response format stays the OpenAItools format. On your own endpoint, it works only when the worker has ENABLE_AUTO_TOOL_CHOICE=true and a TOOL_CALL_PARSER that matches the model, as set in Step 6.
tool_calls comes back empty and the call appears as text in content, the parser doesn’t match the model’s tool format.
Feature parity for the OpenAI-compatible surface
Each cell reflects what the platform’s docs state. “Check the model” means the docs don’t settle it, so test before you rely on it.Batch and offline inference
Fireworks’ Batch API used JSONL files withcustom_id and body per line. On a Runpod queue endpoint, each request is a job. The vLLM worker’s native input takes messages and a sampling_params object. This script submits a Fireworks-format batch file as Runpod jobs and records which job belongs to which custom_id.
submit_batch.py
GET https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID, or receive it at the webhook URL. Async results are kept for 30 minutes after completion, so collect them promptly or use webhooks. /run accepts 1,000 requests per 10 seconds per endpoint, so pace large files. For very large offline workloads, batch jobs run on dedicated batch workers and take a JSON array of the same {"input": ...} objects.
Image and audio models
Public Endpoints for image, video, and text-to-speech models use the native job format, not the OpenAI format. A request isPOST https://api.runpod.ai/v2/SLUG/runsync (or /run) with a body of {"input": {...}}, and the result is under output (for example output.image_url).
Control-plane automation
Scripts and CI jobs that callfirectl or the Fireworks REST API move to runpodctl or REST API v2. See the CLI command map and the API map.
Step 8: Verify, cut over, and shut down
1
Send a real request from outside
Use the same prompts your application sends, from the same network. For a cold endpoint, submit the first request asynchronously so the model load doesn’t hit the synchronous timeout.When the job is
COMPLETED, a worker is warm and the OpenAI route responds normally.2
Compare outputs and latency
Replay a sample of production prompts against both platforms. Compare output quality, tool-call correctness, JSON validity, time to first token, and tokens per second. Differences usually trace to quantization, chat template, or parser settings.
3
Measure cost on real traffic
Run representative load, then read worker time from the endpoint’s Metrics tab or
runpodctl billing serverless. Tune the idle timeout, active workers, and GPU pool from that data.4
Run both in parallel and shift traffic
Switch the base URL and model name through configuration for a small share of traffic, watch error rates and latency, then increase the share. Keep the Fireworks path callable until you’re done.
5
Shut down Fireworks AI
After traffic has moved and you’ve downloaded any weights and datasets you need, stop the resources that bill.Fireworks bills reserved capacity until the reservation ends whether or not you use it, so talk to your Fireworks account manager about any reservation. Serverless usage stops billing when you stop sending requests.
runpodctl serverless delete ENDPOINT_ID, and check for leftover network volumes with runpodctl network-volume list.
Reference
CLI command map
API map
Note the two Runpod hosts:
api.runpod.ai serves requests to endpoints, and api.runpod.io serves the management API. Both take Authorization: Bearer RUNPOD_API_KEY.
Environment variable map
These are client-side variables, since Fireworks doesn’t run your containers.Status map
Request-level states on a Runpod queue endpoint are
IN_QUEUE, IN_PROGRESS, COMPLETED, FAILED, CANCELLED, and TIMED_OUT. Fireworks chat completion requests are synchronous and have no job state.
Migrate with a coding agent
Install the Runpod skills and MCP server in your agent. MCP sign-in uses OAuth (“Sign in with Runpod”), so you don’t need an API key to start.- Claude Code
- Other agents
Troubleshooting
The first request returns a 524 or times out. The synchronous routes have an edge timeout of about 100 seconds, and the first request to a cold worker waits for the model to load. Send the first request with/run and poll /status, raise the idle timeout so workers stay warm between requests, or set one active worker.
“Invalid model” or a 404 on the model name. You sent the Fireworks ID. The model value must be the Hugging Face repo the worker serves or your OPENAI_SERVED_MODEL_NAME_OVERRIDE value. List what the endpoint serves with client.models.list().
Responses are cut off after a few tokens. Fireworks adjusts max_tokens to the context window for you. The Runpod vLLM docs list a default max_tokens of 16. Set it on every request.
Tool calls come back as plain text. The worker is missing ENABLE_AUTO_TOOL_CHOICE=true, or TOOL_CALL_PARSER doesn’t match the model’s tool format. See the model-specific settings.
The reasoning trace disappeared. Your client reads reasoning_content. vLLM v0.32.0 returns reasoning. Read both, and set REASONING_PARSER on the worker. Without a parser, thinking text arrives inside content.
Workers stay throttled or never start. The selected pool has no free GPUs in the allowed data centers, or the endpoint’s CUDA constraint excludes the available hosts. Add a second GPU pool, remove data center restrictions, and check stock with runpodctl gpu list. A network volume restricts the endpoint to one data center.
The worker crashes while loading the model. Usually out of memory. vLLM reserves VRAM for the KV cache on top of the weights. Lower MAX_MODEL_LEN, use a quantized repo, or move to a larger pool. Gated models fail to download without a valid HF_TOKEN.
Output quality differs from Fireworks. Check that the Hugging Face repo matches the checkpoint and precision Fireworks served, and that the chat template is the model’s own. For fine-tunes, confirm you merged the adapter into the exact base it was trained on.
The bill is higher than expected at low traffic. A long idle timeout or an active worker bills for time with no requests. Lower the idle timeout, set active workers to 0, or move low-volume models to a Public Endpoint if one is listed.
429 responses. On Runpod, a 429 is the endpoint’s request-rate limit, not a sign that workers are saturated. Queue-based endpoints hold excess requests in the queue, so watch queue delay and raise max workers.
Next steps
vLLM on Serverless
Deploy and configure OpenAI-compatible LLM endpoints.
Public Endpoints
Browse the models Runpod hosts and their parameters.
Endpoint settings
Tune workers, idle timeout, and autoscaling.
Cached models
Cut cold starts by caching Hugging Face models on hosts.