Skip to main content
This guide is for teams that run production inference on Baseten and want to run it on Runpod. A Truss model with a Model class becomes a Serverless queue endpoint with a handler, a custom Docker server becomes a load balancing endpoint or the Runpod vLLM worker, Model APIs calls move to Public Endpoints or your own vLLM endpoint, Chains become several endpoints behind an orchestrator, and Training Jobs move to Pods. At the end you’ll have each workload deployed on Runpod, your clients pointed at the new URLs, and your Baseten deployments deactivated.
You can hand this migration to a coding agent. See Migrate with a coding agent for the setup commands and a prompt.

How Baseten maps to Runpod

What’s different

What works differently

  • You own the image build. baseten model push uploads a Truss and Baseten builds the image. On Runpod you build and push a linux/amd64 image, deploy from a GitHub repository that contains a Dockerfile, or use the Runpod SDK, which packages your code without a container build.
  • The request body is wrapped. A Truss predict receives the JSON body as is. A Runpod queue endpoint expects {"input": {...}} and your handler reads job["input"].
  • The response is wrapped. /predict returns your model’s output directly. /runsync returns a job object, and your output is under the output key.
  • Load at import time instead of in load(). Code at the top of handler.py runs once when the worker starts. That’s where the work from __init__ and load() goes.
  • Secrets are environment variables. Replace kwargs["secrets"]["hf_access_token"] and reads of /secrets/hf_access_token with os.environ.
  • Sync requests have a shorter wait window. Baseten’s docs give a default 1,200 second predict timeout for sync requests. Runpod’s /runsync waits 90 seconds by default and up to 5 minutes with ?wait=. For anything longer, including requests that might land on a cold start, use /run with /status or a webhook.
  • Async results are stored. Baseten delivers async output only to your webhook and doesn’t store it. Runpod keeps /run results for 30 minutes and returns them from /status/JOB_ID, with or without a webhook.
  • Streaming on queue endpoints is polled. A Truss generator streams over the open HTTP response. A Runpod generator handler publishes chunks that clients read from /stream/JOB_ID. For a streamed HTTP response (for example server-sent events), use a load balancing endpoint or the vLLM worker’s OpenAI-compatible route.
  • Autoscaling has fewer knobs. Runpod scales a queue endpoint by queue delay (add workers when requests wait longer than a threshold, 4 seconds by default) or by request count (ceil((queued + in progress) / scaler value)). Set the request count scaler value to your Baseten concurrency target. To keep headroom like a target utilization below 100%, use a lower scaler value. Runpod has no autoscaling window, max scale-down rate, or schedule settings.
  • Idle timeout and request count scaling don’t combine on queue endpoints. REST API v2 rejects an idle timeout on a queue endpoint that scales by request count. Pick queue delay scaling when you need a long idle timeout.
  • Promotion is an image update. Instead of promoting a deployment into an environment, you update the endpoint’s image tag. Runpod replaces workers progressively: workers without a job are replaced right away, and busy workers finish their job first.
  • Cancel works on running jobs. Baseten cancels only QUEUED async requests. Runpod’s /cancel/JOB_ID stops queued and in-progress jobs.
  • Health checks. A docker_server declares readiness and liveness endpoints. A load balancing worker has one health path (/ping by default, configurable with HEALTH_CHECK_PATH). Return 200 when ready and 204 while initializing.
  • Ports. Baseten reserves port 8080 and sets PORT=8080. On a load balancing endpoint, PORT and PORT_HEALTH both default to 80, and you set both if your server listens elsewhere.
  • Load balancing timeouts. A load balancing endpoint allows 5.5 minutes of processing per request and a 30 MB payload. Longer requests belong on a queue endpoint.
  • Payload limits on queue endpoints. 10 MB for /run and 20 MB for /runsync. Send large inputs as URLs.
  • Idle endpoints scale themselves down. After 3 days without requests, Runpod reduces an endpoint’s max workers to 2, and after 7 days to 0. Raise max workers again before you send traffic to an endpoint that sat unused.

What has no direct equivalent

Going the other way, these Runpod features don’t have a counterpart in the Baseten docs this guide was checked against: Pods that keep running as long-lived, general-purpose GPU machines outside of a deployment or training job (Baseten’s docs describe SSH into running deployments and training jobs), writable shared network volumes (Baseten’s BDN volumes are immutable and mounted read-only), and job results you can fetch from /status after the fact.

Requirements

  • A Runpod account with credits and an API key.
  • Access to your Baseten workspace, with the baseten CLI authenticated (baseten auth login or BASETEN_API_KEY).
  • Docker and a container registry account (for example Docker Hub), unless you migrate with the Runpod SDK only.
  • Python 3.10 to 3.14 for the Runpod SDK.
  • The source of each Truss (config.yaml, model/, data/, packages/). If you don’t have it locally, Step 1 shows how to download it.

Step 1: Take inventory of what you run on Baseten

List every model, then every deployment and environment for each model. The baseten CLI prints JSON with --output json, which is easier to diff later.
For each deployment, record the instance type, region, min and max replicas, scale-down delay, concurrency target, and which environment points at it. If the download fails (a push can disable archive download), fall back to your repository. Then list the resources that aren’t models:
Secret values can’t be exported from Baseten. Collect them from wherever you keep the originals before you start. Next, find every Truss in your repositories and every place your code calls Baseten:
The second command catches inference URLs, Model APIs base URLs, the legacy Api-Key header, Chains code, CI jobs that push models, and Terraform.

Step 2: Set up Runpod

Create an API key in the console (see Manage API keys), then install the tools.
runpodctl doctor walks you through first-time setup, including the API key. rp login stores credentials in ~/.runpod/config.toml, and the SDK also reads RUNPOD_API_KEY. On macOS you can install the CLI with brew install runpod/runpodctl/runpodctl. If you manage infrastructure as code, add the Terraform provider. To let an agent manage resources, connect the MCP server.

Step 3: Choose where each workload lands

GPU mapping

Serverless endpoints select GPUs by pool. Pick the pool by the VRAM your model needs, not by name. To request several GPUs per worker (for example H100:2), set the GPU count. A pool can hold a different GPU model than the one you ran on Baseten, so rerun your latency benchmarks. GPU stock varies by data center and over time. Check what’s available before you commit to a pool with runpodctl gpu list, and list more than one pool where your model fits both. For the full list of GPU IDs and pools, see GPU types. CPU-only instances map to CPU instance IDs in the form cpu{generation}{type}-{vCPU}-{memory GB}:

Step 4: Move your images and code

Runpod runs linux/amd64 images. If you build on an Apple Silicon machine, pass --platform linux/amd64. Use versioned tags instead of :latest, because workers can keep serving a cached copy of a mutable tag.

Translate a Truss model to a handler

This is a complete Truss for a text generation model. config.yaml declares the environment and model/model.py holds the Model class.
On Runpod, the same model is three files. The handler loads the pipeline at module scope, which replaces __init__ and load(), and handler(job) replaces predict(model_input).
What matters in the translation:
  • The base image already contains PyTorch 2.8 and CUDA 12.8, so torch isn’t in requirements.txt. If your Truss pins a different torch, choose a base image that matches it, or start from an nvidia/cuda image and install it yourself. See Create a Dockerfile.
  • The secret arrives as the HF_TOKEN environment variable. You wire it to a Runpod secret when you create the endpoint in Step 6.
  • preprocess and postprocess don’t exist as separate hooks. Call them in order inside the handler: postprocess(predict(preprocess(job["input"]))).
  • An exception raised in the handler marks the job FAILED. To fail with your own message, return {"error": "..."}.
  • Test before you build: python handler.py --test_input '{"input": {"prompt": "Hello"}}'.

Field-by-field map of config.yaml

GPU endpoints have no separate vCPU or memory field. You choose a GPU pool and the container disk size. If your Truss asks for a lot of system RAM next to a small GPU, check the pool’s hosts with runpodctl gpu list before you settle on it.

Streaming

A Truss predict that returns a generator becomes a handler that yields.
Clients read the chunks from /stream/JOB_ID. With return_aggregate_stream set, the full output is also available from /run (through /status) and /runsync, as a list of the yielded chunks instead of one string. See Streaming handlers.

Concurrency

runtime.predict_concurrency with an async def predict becomes an async handler plus a concurrency_modifier.
The function can return a different number over time to adapt to load. See Build a concurrent handler.

Skip the Dockerfile with the Runpod SDK

The Runpod SDK is the closest match to the Truss workflow: you declare hardware and dependencies in Python, and rp flash deploy packages and deploys the code. This file replaces config.yaml, model/model.py, the Dockerfile, and the endpoint settings from the example above.
main.py
Run it against temporary endpoints with rp flash dev main.py, then deploy with rp flash deploy. A few things to know:
  • model=weights stages the Hugging Face weights on the host before the worker starts, and transformers finds them through the standard cache. This replaces weights or model_cache in config.yaml.
  • Imports of worker-only packages go inside the function, so your machine doesn’t need them. torch is preinstalled on the built-in GPU images.
  • The SDK docs describe a once-per-worker setup hook, @runpod.init, for @app.api classes. For an @app.queue function, this example keeps the pipeline in a module-level variable. Confirm in rp flash dev that a second call skips the load. If it doesn’t, move the model to an @app.api class with @runpod.init, which mirrors load() and predict() directly.
  • Environments map to rp flash deploy --env staging and rp flash deploy --env production. Each environment gets its own endpoints. See Apps and environments.
For every decorator argument, see Resource parameters.

Custom servers

A docker_server Truss already is a container with an HTTP server, so the image moves as is. What changes is the wiring around it.
config.yaml
You have two options on Runpod:
  • vLLM worker (recommended for vLLM). Deploy Runpod’s vLLM worker from the Hub and set MODEL_NAME. You get cached models, the queue, and an OpenAI-compatible route at https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1. Any vllm serve flag can be set as an environment variable in upper snake case, for example MAX_MODEL_LEN=8192. See vLLM environment variables.
  • Load balancing endpoint (any server). Run the same image and start command. Set PORT and PORT_HEALTH to your server port, set HEALTH_CHECK_PATH to your readiness route, and expose the port. Requests to https://ENDPOINT_ID.api.runpod.ai/v1/chat/completions reach /v1/chat/completions on your server. There’s no /predict alias and no /sync/ prefix.
Step 6 has the deploy commands for both. If your image depends on Baseten’s runtime contract, check these before you move it:

Chains

A Chain deploys several Chainlets together and lets one call another through run_remote. On Runpod, each Chainlet becomes its own endpoint with its own hardware and scaling, and the entrypoint becomes a small CPU endpoint that calls the others.
Deploy rand_int and hello_world as two endpoints (the orchestrator on a CPU instance), store a Runpod API key as a secret, and pass it to the orchestrator as ORCHESTRATOR_API_KEY along with RAND_INT_ENDPOINT_ID. Secret names can’t start with runpod, so don’t name the secret RUNPOD_API_KEY. What you give up compared with Chains: typed stubs between steps, a single truss chains push for the whole pipeline, and Chainlet-level retries declared in chains.depends. What stays the same: each step scales separately on its own hardware. Keep the orchestrator’s wait under the /runsync window, or switch the inner calls to /run and poll /status.

Step 5: Move your data and models

Hugging Face weights. If your Truss uses weights with an hf:// source, or model_cache with a Hugging Face repo_id, use a cached model. Runpod tries to start workers on hosts that already hold the model and doesn’t bill worker time for the download. Cached models are mounted at /runpod-volume/huggingface-cache/hub/ in the Hugging Face cache layout, so the path changes from your Truss mount_location or /app/model_cache/VOLUME_FOLDER. Use Hugging Face models has a helper that resolves the snapshot path. Each endpoint takes one cached model. Weights in S3, GCS, R2, or Azure, and BDN volumes. Put them on a network volume, which mounts at /runpod-volume on Serverless workers. A network volume lives in one data center, and an endpoint that attaches it runs only there, so choose a data center that stocks the GPU pool you need.
The S3-compatible API uses its own access key and secret, which you create in the console, and is available in the data centers listed in S3-compatible API. To pull a BDN volume down first, run baseten volume pull bdn:NAMESPACE/VOLUME ./weights. For large transfers, start a Pod with the volume attached and copy from your bucket inside the data center instead of through your laptop. Small files. Anything in the Truss data/ directory, or fetched by external_data, can be copied or downloaded into the image in the Dockerfile. This keeps the worker self-contained but increases image size. Weights without a data center pin. A global volume (beta) isn’t tied to a data center and attaches to GPU endpoints at /runpod-volume. It bills for storage and requests and has no S3-compatible API, so load it from a Pod. Training checkpoints. Baseten stores checkpoints from Training Jobs. Download the ones you need before you shut down, and write new ones to a network volume on Runpod.

Step 6: Deploy on Runpod

Queue endpoint from your image

First build and push the image from Step 4, and create the secret the handler reads.
On Baseten, one command deploys the Truss and sets autoscaling:
On Runpod, create a Serverless template for the image, then an endpoint that uses it.
TEMPLATE_ID comes from the output of the first command. HF_HUB_CACHE points transformers at the cached model that --model-reference attaches. Without it, the handler downloads the weights again on every cold start. Each Serverless template binds to one endpoint, so create one template per endpoint. For a private image, run runpodctl registry create and add --registry-auth-id to the template.
The autoscaling flags map as follows:

Queue endpoint from the Runpod SDK

If you wrote main.py in Step 4, there’s no template or image to create:
rp flash deploy prints a link to each endpoint. To remove them later, run rp flash undeploy --app text-generator --env production.

vLLM worker

Find the vLLM worker in the Hub and create an endpoint from it:
For a gated model, also set HF_TOKEN from a secret. The endpoint answers Runpod-native requests on /run and /runsync, and OpenAI SDK requests at https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1. See Get started with vLLM.

Load balancing endpoint

runpodctl serverless create creates queue endpoints. Create a load balancing endpoint with REST API v2 (or the console, by choosing Load Balancer as the endpoint type). This is the docker_server example from Step 4:
cmd is the argument list passed to the image’s entrypoint, so check what your image’s ENTRYPOINT is and set entrypoint too if you need to replace it. Load balancing endpoints scale by request count only. Set requestCount to the concurrency target you used on Baseten. The response includes the endpoint ID, and your server is then reachable at https://ENDPOINT_ID.api.runpod.ai.

Pods for training

A Baseten Training Job packages your code and runs it on a GPU. On Runpod, the SDK equivalent is @app.task, which starts a Pod for the call and terminates it when the function returns:
train.py
Run it with rp flash dev train.py --once. For interactive work, create a Pod from the console or with runpodctl pod create and connect over SSH. See Pods.

Step 7: Update the code that calls Baseten

Sync requests

The URL, the body wrapper, and the response wrapper change. Both platforms accept Authorization: Bearer. If your code still sends Baseten’s legacy Authorization: Api-Key ... header, change the scheme to Bearer.
The Runpod response is a job object with id, status, output, delayTime, and executionTime. The first output key is Runpod’s, and the second is the key the handler returned. An HTTP 200 doesn’t mean the job succeeded, so check status.

Async requests and webhooks

Update your webhook receiver too. Baseten posts {"request_id", "model_id", "deployment_id", "type", "time", "data", "errors"} and signs it with X-BASETEN-SIGNATURE. Runpod posts the job object (the same shape /status returns) and expects a 200, retrying up to 2 more times with a 10 second delay. Read output instead of data, id instead of request_id, and replace the signature check as described in What has no direct equivalent. The webhook field is optional: you can poll /status alone. Per-request options move as follows: max_time_in_queue_seconds becomes policy.ttl (milliseconds, covering queue plus run time), and priority: 1 or 2 becomes policy.lowPriority: true. See Send API requests.

Streaming

On a queue endpoint, submit with /run, then read chunks with the Runpod SDK’s endpoint client:
This replaces a requests.post(..., stream=True) loop over a Baseten /predict response. If a browser or another service consumes the HTTP stream directly, serve it from a load balancing endpoint instead.

OpenAI-compatible clients

For Model APIs, and for deployments you called through /environments/production/sync/v1, only the base URL, API key, and model name change.
Each Public Endpoint has its own base URL that contains the model’s endpoint ID, unlike Baseten’s single base URL with a model slug per request. The example uses Qwen3 32B AWQ (qwen3-32b-awq, model Qwen/Qwen3-32B-AWQ), which documents an OpenAI-compatible route. The model reference lists the text models Runpod hosts, and the catalog is smaller than Baseten’s Model APIs, so check it for the model you call. For any other Model APIs slug, deploy that model on a vLLM endpoint. The vLLM worker serves /chat/completions, /completions, and /models. If you use Baseten’s Responses or Anthropic Messages protocols, move those calls to chat completions or serve the protocol from your own server on a load balancing endpoint. For a load balancing endpoint that runs an OpenAI-compatible server, the base URL is https://ENDPOINT_ID.api.runpod.ai/v1.

Control-plane automation

Replace CI steps and scripts that call the baseten or truss CLIs or the Management API. The common ones: Updating the image creates a release, and workers move to it as a rolling release. To roll a new image, patch the endpoint itself:
The full maps are in the Reference section.

Step 8: Verify, cut over, and shut down

1

Send a real request from outside

Call the new endpoint from a machine outside Runpod and check the job status and output.
The first request after a deploy includes a cold start. Send a second one to see warm latency.
2

Compare outputs and latency

Replay a fixed set of production inputs against both platforms with the same seed and sampling settings, and diff the outputs. In the Runpod job object, delayTime is queue plus cold start time and executionTime is handler time, both in milliseconds. Compare executionTime with your Baseten inference latency, and tune active workers and idle timeout against delayTime.
3

Run both in parallel and shift traffic

Keep the Baseten deployment active and send a small share of traffic to Runpod from your client or gateway. Raise max workers before each increase. Watch the endpoint’s Metrics tab for delay time, cold starts, and failed requests.
4

Set a cost guard on Runpod

Max workers caps how far an endpoint scales, and the execution timeout stops runaway jobs. Keep active workers at 0 unless you need warm capacity. Check spend with runpodctl billing --help or the REST v2 /v2/billing/serverless route.
5

Shut down Baseten

When traffic has been at zero for long enough to trust it, deactivate first (reversible), then delete.
Deleting a model or volume can’t be undone. Deactivate instead of only scaling to zero, because a scaled-to-zero deployment stays live and any stray request starts a replica. Finally, delete the Baseten API keys your services used (baseten org api-key list) and remove BASETEN_API_KEY from your secret stores.

Migrate your Terraform

Baseten publishes the basetenlabs/baseten provider with a baseten_model resource (which can push a Truss directory and manage environment autoscaling) and a baseten_secret resource. The Runpod provider, runpod/runpod at ~> 2.0, is built on REST API v2.
Both providers read the API key from the environment (BASETEN_API_KEY and RUNPOD_API_KEY). A new image tag in image rolls the endpoint. timeout is in milliseconds and idle_timeout in seconds. For every argument, see the provider reference.

Reference

CLI command map

API map

Request and response field map

Environment variable map

Status map

Baseten deployment statuses map more loosely: Active is an endpoint with a running worker, Scaled to zero is an endpoint with no running workers, Waking up and Loading model are a worker that is initializing, and Inactive is an endpoint with max workers set to 0.

Migrate with a coding agent

Install the Runpod plugin or skills and connect the MCP server.
The MCP server signs in with OAuth (“Sign in with Runpod”), so you don’t need an API key to start. See Agent skills and MCP servers for the supported agents and client IDs. Then give the agent this prompt from the root of your repository:

Troubleshooting

The handler raises KeyError on the input. The client is still sending the bare model input. Wrap it as {"input": {...}}. If the handler receives a nested input inside job["input"], the client wrapped it twice. The client gets a 200 but no result. /runsync returned before the job finished, or the code reads the top level of the response. Check status, read output, and switch long requests to /run plus /status. The model reloads on every request. The load is inside handler (a direct port of predict that calls self.load()-style code). Move it to module scope. The secret is empty. Code still reads kwargs["secrets"] or /secrets/NAME. Read the environment variable, and check that its value on the endpoint is {{ RUNPOD_SECRET_name }} with the exact secret name. Workers pick up a changed secret the next time they start. The worker never becomes ready on a load balancing endpoint, and requests return 502. The server listens on a port the load balancer doesn’t check. Set PORT and PORT_HEALTH to the server’s port, expose that port, and make sure HEALTH_CHECK_PATH returns 200. A server ported from Baseten that reads PORT and expects 8080 needs the variable set explicitly. The worker is stopped during startup with a large model. Initialization exceeded RUNPOD_INIT_TIMEOUT (420 seconds by default). Raise it in the endpoint’s environment variables, and use a cached model so the download doesn’t count against startup. Baseten’s default startup threshold is 30 minutes, so a model that loaded there can time out here. Jobs sit in the queue with no workers. The GPU pool has no stock in the endpoint’s data centers, the endpoint is pinned to one data center by a network volume, or max workers is 0 because the endpoint went unused for 7 days. Add a second pool, widen the data centers, or raise max workers. exec format error in the worker logs. The image was built for ARM. Rebuild with --platform linux/amd64. The new image isn’t running after a push. The endpoint still references the old tag, or you pushed over a mutable tag such as :latest. Push a new version tag and patch the endpoint’s image. The webhook receiver rejects Runpod’s callbacks. It still verifies X-BASETEN-SIGNATURE and parses data. Update it for the Runpod job payload.

Next steps

Handler functions

Streaming, async, and concurrent handlers, error handling, and payload limits.

Endpoint settings

Workers, scaling, timeouts, FlashBoot, and data centers.

Runpod SDK

Define and deploy endpoints from Python with runpod.App.

Cached models

Start workers on hosts that already hold your Hugging Face model.
Last modified on November 14, 2026