Model class becomes a Serverless queue endpoint with a handler, a custom Docker server becomes a load balancing endpoint or the Runpod vLLM worker, Model APIs calls move to Public Endpoints or your own vLLM endpoint, Chains become several endpoints behind an orchestrator, and Training Jobs move to Pods. At the end you’ll have each workload deployed on Runpod, your clients pointed at the new URLs, and your Baseten deployments deactivated.
How Baseten maps to Runpod
What’s different
What works differently
- You own the image build.
baseten model pushuploads a Truss and Baseten builds the image. On Runpod you build and push alinux/amd64image, deploy from a GitHub repository that contains a Dockerfile, or use the Runpod SDK, which packages your code without a container build. - The request body is wrapped. A Truss
predictreceives the JSON body as is. A Runpod queue endpoint expects{"input": {...}}and your handler readsjob["input"]. - The response is wrapped.
/predictreturns your model’s output directly./runsyncreturns a job object, and your output is under theoutputkey. - Load at import time instead of in
load(). Code at the top ofhandler.pyruns once when the worker starts. That’s where the work from__init__andload()goes. - Secrets are environment variables. Replace
kwargs["secrets"]["hf_access_token"]and reads of/secrets/hf_access_tokenwithos.environ. - Sync requests have a shorter wait window. Baseten’s docs give a default 1,200 second predict timeout for sync requests. Runpod’s
/runsyncwaits 90 seconds by default and up to 5 minutes with?wait=. For anything longer, including requests that might land on a cold start, use/runwith/statusor a webhook. - Async results are stored. Baseten delivers async output only to your webhook and doesn’t store it. Runpod keeps
/runresults for 30 minutes and returns them from/status/JOB_ID, with or without a webhook. - Streaming on queue endpoints is polled. A Truss generator streams over the open HTTP response. A Runpod generator handler publishes chunks that clients read from
/stream/JOB_ID. For a streamed HTTP response (for example server-sent events), use a load balancing endpoint or the vLLM worker’s OpenAI-compatible route. - Autoscaling has fewer knobs. Runpod scales a queue endpoint by queue delay (add workers when requests wait longer than a threshold, 4 seconds by default) or by request count (
ceil((queued + in progress) / scaler value)). Set the request count scaler value to your Baseten concurrency target. To keep headroom like a target utilization below 100%, use a lower scaler value. Runpod has no autoscaling window, max scale-down rate, or schedule settings. - Idle timeout and request count scaling don’t combine on queue endpoints. REST API v2 rejects an idle timeout on a queue endpoint that scales by request count. Pick queue delay scaling when you need a long idle timeout.
- Promotion is an image update. Instead of promoting a deployment into an environment, you update the endpoint’s image tag. Runpod replaces workers progressively: workers without a job are replaced right away, and busy workers finish their job first.
- Cancel works on running jobs. Baseten cancels only
QUEUEDasync requests. Runpod’s/cancel/JOB_IDstops queued and in-progress jobs. - Health checks. A
docker_serverdeclares readiness and liveness endpoints. A load balancing worker has one health path (/pingby default, configurable withHEALTH_CHECK_PATH). Return200when ready and204while initializing. - Ports. Baseten reserves port 8080 and sets
PORT=8080. On a load balancing endpoint,PORTandPORT_HEALTHboth default to80, and you set both if your server listens elsewhere. - Load balancing timeouts. A load balancing endpoint allows 5.5 minutes of processing per request and a 30 MB payload. Longer requests belong on a queue endpoint.
- Payload limits on queue endpoints. 10 MB for
/runand 20 MB for/runsync. Send large inputs as URLs. - Idle endpoints scale themselves down. After 3 days without requests, Runpod reduces an endpoint’s max workers to 2, and after 7 days to 0. Raise max workers again before you send traffic to an endpoint that sat unused.
What has no direct equivalent
Going the other way, these Runpod features don’t have a counterpart in the Baseten docs this guide was checked against: Pods that keep running as long-lived, general-purpose GPU machines outside of a deployment or training job (Baseten’s docs describe SSH into running deployments and training jobs), writable shared network volumes (Baseten’s BDN volumes are immutable and mounted read-only), and job results you can fetch from
/status after the fact.
Requirements
- A Runpod account with credits and an API key.
- Access to your Baseten workspace, with the
basetenCLI authenticated (baseten auth loginorBASETEN_API_KEY). - Docker and a container registry account (for example Docker Hub), unless you migrate with the Runpod SDK only.
- Python 3.10 to 3.14 for the Runpod SDK.
- The source of each Truss (
config.yaml,model/,data/,packages/). If you don’t have it locally, Step 1 shows how to download it.
Step 1: Take inventory of what you run on Baseten
List every model, then every deployment and environment for each model. Thebaseten CLI prints JSON with --output json, which is easier to diff later.
Api-Key header, Chains code, CI jobs that push models, and Terraform.
Step 2: Set up Runpod
Create an API key in the console (see Manage API keys), then install the tools.runpodctl doctor walks you through first-time setup, including the API key. rp login stores credentials in ~/.runpod/config.toml, and the SDK also reads RUNPOD_API_KEY. On macOS you can install the CLI with brew install runpod/runpodctl/runpodctl.
If you manage infrastructure as code, add the Terraform provider. To let an agent manage resources, connect the MCP server.
Step 3: Choose where each workload lands
GPU mapping
Serverless endpoints select GPUs by pool. Pick the pool by the VRAM your model needs, not by name. To request several GPUs per worker (for exampleH100:2), set the GPU count.
A pool can hold a different GPU model than the one you ran on Baseten, so rerun your latency benchmarks. GPU stock varies by data center and over time. Check what’s available before you commit to a pool with
runpodctl gpu list, and list more than one pool where your model fits both. For the full list of GPU IDs and pools, see GPU types.
CPU-only instances map to CPU instance IDs in the form cpu{generation}{type}-{vCPU}-{memory GB}:
Step 4: Move your images and code
Runpod runslinux/amd64 images. If you build on an Apple Silicon machine, pass --platform linux/amd64. Use versioned tags instead of :latest, because workers can keep serving a cached copy of a mutable tag.
Translate a Truss model to a handler
This is a complete Truss for a text generation model.config.yaml declares the environment and model/model.py holds the Model class.
__init__ and load(), and handler(job) replaces predict(model_input).
- The base image already contains PyTorch 2.8 and CUDA 12.8, so
torchisn’t inrequirements.txt. If your Truss pins a differenttorch, choose a base image that matches it, or start from annvidia/cudaimage and install it yourself. See Create a Dockerfile. - The secret arrives as the
HF_TOKENenvironment variable. You wire it to a Runpod secret when you create the endpoint in Step 6. preprocessandpostprocessdon’t exist as separate hooks. Call them in order inside the handler:postprocess(predict(preprocess(job["input"]))).- An exception raised in the handler marks the job
FAILED. To fail with your own message, return{"error": "..."}. - Test before you build:
python handler.py --test_input '{"input": {"prompt": "Hello"}}'.
Field-by-field map of config.yaml
GPU endpoints have no separate vCPU or memory field. You choose a GPU pool and the container disk size. If your Truss asks for a lot of system RAM next to a small GPU, check the pool’s hosts with
runpodctl gpu list before you settle on it.
Streaming
A Trusspredict that returns a generator becomes a handler that yields.
/stream/JOB_ID. With return_aggregate_stream set, the full output is also available from /run (through /status) and /runsync, as a list of the yielded chunks instead of one string. See Streaming handlers.
Concurrency
runtime.predict_concurrency with an async def predict becomes an async handler plus a concurrency_modifier.
Skip the Dockerfile with the Runpod SDK
The Runpod SDK is the closest match to the Truss workflow: you declare hardware and dependencies in Python, andrp flash deploy packages and deploys the code. This file replaces config.yaml, model/model.py, the Dockerfile, and the endpoint settings from the example above.
main.py
rp flash dev main.py, then deploy with rp flash deploy. A few things to know:
model=weightsstages the Hugging Face weights on the host before the worker starts, andtransformersfinds them through the standard cache. This replacesweightsormodel_cacheinconfig.yaml.- Imports of worker-only packages go inside the function, so your machine doesn’t need them.
torchis preinstalled on the built-in GPU images. - The SDK docs describe a once-per-worker setup hook,
@runpod.init, for@app.apiclasses. For an@app.queuefunction, this example keeps the pipeline in a module-level variable. Confirm inrp flash devthat a second call skips the load. If it doesn’t, move the model to an@app.apiclass with@runpod.init, which mirrorsload()andpredict()directly. - Environments map to
rp flash deploy --env stagingandrp flash deploy --env production. Each environment gets its own endpoints. See Apps and environments.
Custom servers
Adocker_server Truss already is a container with an HTTP server, so the image moves as is. What changes is the wiring around it.
config.yaml
- vLLM worker (recommended for vLLM). Deploy Runpod’s vLLM worker from the Hub and set
MODEL_NAME. You get cached models, the queue, and an OpenAI-compatible route athttps://api.runpod.ai/v2/ENDPOINT_ID/openai/v1. Anyvllm serveflag can be set as an environment variable in upper snake case, for exampleMAX_MODEL_LEN=8192. See vLLM environment variables. - Load balancing endpoint (any server). Run the same image and start command. Set
PORTandPORT_HEALTHto your server port, setHEALTH_CHECK_PATHto your readiness route, and expose the port. Requests tohttps://ENDPOINT_ID.api.runpod.ai/v1/chat/completionsreach/v1/chat/completionson your server. There’s no/predictalias and no/sync/prefix.
Chains
A Chain deploys several Chainlets together and lets one call another throughrun_remote. On Runpod, each Chainlet becomes its own endpoint with its own hardware and scaling, and the entrypoint becomes a small CPU endpoint that calls the others.
rand_int and hello_world as two endpoints (the orchestrator on a CPU instance), store a Runpod API key as a secret, and pass it to the orchestrator as ORCHESTRATOR_API_KEY along with RAND_INT_ENDPOINT_ID. Secret names can’t start with runpod, so don’t name the secret RUNPOD_API_KEY.
What you give up compared with Chains: typed stubs between steps, a single truss chains push for the whole pipeline, and Chainlet-level retries declared in chains.depends. What stays the same: each step scales separately on its own hardware. Keep the orchestrator’s wait under the /runsync window, or switch the inner calls to /run and poll /status.
Step 5: Move your data and models
Hugging Face weights. If your Truss usesweights with an hf:// source, or model_cache with a Hugging Face repo_id, use a cached model. Runpod tries to start workers on hosts that already hold the model and doesn’t bill worker time for the download. Cached models are mounted at /runpod-volume/huggingface-cache/hub/ in the Hugging Face cache layout, so the path changes from your Truss mount_location or /app/model_cache/VOLUME_FOLDER. Use Hugging Face models has a helper that resolves the snapshot path. Each endpoint takes one cached model.
Weights in S3, GCS, R2, or Azure, and BDN volumes. Put them on a network volume, which mounts at /runpod-volume on Serverless workers. A network volume lives in one data center, and an endpoint that attaches it runs only there, so choose a data center that stocks the GPU pool you need.
baseten volume pull bdn:NAMESPACE/VOLUME ./weights. For large transfers, start a Pod with the volume attached and copy from your bucket inside the data center instead of through your laptop.
Small files. Anything in the Truss data/ directory, or fetched by external_data, can be copied or downloaded into the image in the Dockerfile. This keeps the worker self-contained but increases image size.
Weights without a data center pin. A global volume (beta) isn’t tied to a data center and attaches to GPU endpoints at /runpod-volume. It bills for storage and requests and has no S3-compatible API, so load it from a Pod.
Training checkpoints. Baseten stores checkpoints from Training Jobs. Download the ones you need before you shut down, and write new ones to a network volume on Runpod.
Step 6: Deploy on Runpod
Queue endpoint from your image
First build and push the image from Step 4, and create the secret the handler reads.- runpodctl
- REST API v2
- Console
TEMPLATE_ID comes from the output of the first command. HF_HUB_CACHE points transformers at the cached model that --model-reference attaches. Without it, the handler downloads the weights again on every cold start. Each Serverless template binds to one endpoint, so create one template per endpoint. For a private image, run runpodctl registry create and add --registry-auth-id to the template.Queue endpoint from the Runpod SDK
If you wrotemain.py in Step 4, there’s no template or image to create:
rp flash deploy prints a link to each endpoint. To remove them later, run rp flash undeploy --app text-generator --env production.
vLLM worker
Find the vLLM worker in the Hub and create an endpoint from it:HF_TOKEN from a secret. The endpoint answers Runpod-native requests on /run and /runsync, and OpenAI SDK requests at https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1. See Get started with vLLM.
Load balancing endpoint
runpodctl serverless create creates queue endpoints. Create a load balancing endpoint with REST API v2 (or the console, by choosing Load Balancer as the endpoint type). This is the docker_server example from Step 4:
cmd is the argument list passed to the image’s entrypoint, so check what your image’s ENTRYPOINT is and set entrypoint too if you need to replace it. Load balancing endpoints scale by request count only. Set requestCount to the concurrency target you used on Baseten. The response includes the endpoint ID, and your server is then reachable at https://ENDPOINT_ID.api.runpod.ai.
Pods for training
A Baseten Training Job packages your code and runs it on a GPU. On Runpod, the SDK equivalent is@app.task, which starts a Pod for the call and terminates it when the function returns:
train.py
rp flash dev train.py --once. For interactive work, create a Pod from the console or with runpodctl pod create and connect over SSH. See Pods.
Step 7: Update the code that calls Baseten
Sync requests
The URL, the body wrapper, and the response wrapper change. Both platforms acceptAuthorization: Bearer. If your code still sends Baseten’s legacy Authorization: Api-Key ... header, change the scheme to Bearer.
id, status, output, delayTime, and executionTime. The first output key is Runpod’s, and the second is the key the handler returned. An HTTP 200 doesn’t mean the job succeeded, so check status.
Async requests and webhooks
{"request_id", "model_id", "deployment_id", "type", "time", "data", "errors"} and signs it with X-BASETEN-SIGNATURE. Runpod posts the job object (the same shape /status returns) and expects a 200, retrying up to 2 more times with a 10 second delay. Read output instead of data, id instead of request_id, and replace the signature check as described in What has no direct equivalent. The webhook field is optional: you can poll /status alone. Per-request options move as follows: max_time_in_queue_seconds becomes policy.ttl (milliseconds, covering queue plus run time), and priority: 1 or 2 becomes policy.lowPriority: true. See Send API requests.
Streaming
On a queue endpoint, submit with/run, then read chunks with the Runpod SDK’s endpoint client:
requests.post(..., stream=True) loop over a Baseten /predict response. If a browser or another service consumes the HTTP stream directly, serve it from a load balancing endpoint instead.
OpenAI-compatible clients
For Model APIs, and for deployments you called through/environments/production/sync/v1, only the base URL, API key, and model name change.
qwen3-32b-awq, model Qwen/Qwen3-32B-AWQ), which documents an OpenAI-compatible route. The model reference lists the text models Runpod hosts, and the catalog is smaller than Baseten’s Model APIs, so check it for the model you call. For any other Model APIs slug, deploy that model on a vLLM endpoint. The vLLM worker serves /chat/completions, /completions, and /models. If you use Baseten’s Responses or Anthropic Messages protocols, move those calls to chat completions or serve the protocol from your own server on a load balancing endpoint. For a load balancing endpoint that runs an OpenAI-compatible server, the base URL is https://ENDPOINT_ID.api.runpod.ai/v1.
Control-plane automation
Replace CI steps and scripts that call thebaseten or truss CLIs or the Management API. The common ones:
Updating the image creates a release, and workers move to it as a rolling release. To roll a new image, patch the endpoint itself:
Step 8: Verify, cut over, and shut down
1
Send a real request from outside
Call the new endpoint from a machine outside Runpod and check the job status and output.The first request after a deploy includes a cold start. Send a second one to see warm latency.
2
Compare outputs and latency
Replay a fixed set of production inputs against both platforms with the same seed and sampling settings, and diff the outputs. In the Runpod job object,
delayTime is queue plus cold start time and executionTime is handler time, both in milliseconds. Compare executionTime with your Baseten inference latency, and tune active workers and idle timeout against delayTime.3
Run both in parallel and shift traffic
Keep the Baseten deployment active and send a small share of traffic to Runpod from your client or gateway. Raise max workers before each increase. Watch the endpoint’s Metrics tab for delay time, cold starts, and failed requests.
4
Set a cost guard on Runpod
Max workers caps how far an endpoint scales, and the execution timeout stops runaway jobs. Keep active workers at 0 unless you need warm capacity. Check spend with
runpodctl billing --help or the REST v2 /v2/billing/serverless route.5
Shut down Baseten
When traffic has been at zero for long enough to trust it, deactivate first (reversible), then delete.Deleting a model or volume can’t be undone. Deactivate instead of only scaling to zero, because a scaled-to-zero deployment stays live and any stray request starts a replica. Finally, delete the Baseten API keys your services used (
baseten org api-key list) and remove BASETEN_API_KEY from your secret stores.Migrate your Terraform
Baseten publishes thebasetenlabs/baseten provider with a baseten_model resource (which can push a Truss directory and manage environment autoscaling) and a baseten_secret resource. The Runpod provider, runpod/runpod at ~> 2.0, is built on REST API v2.
BASETEN_API_KEY and RUNPOD_API_KEY). A new image tag in image rolls the endpoint. timeout is in milliseconds and idle_timeout in seconds. For every argument, see the provider reference.
Reference
CLI command map
API map
Request and response field map
Environment variable map
Status map
Baseten deployment statuses map more loosely: Active is an endpoint with a running worker, Scaled to zero is an endpoint with no running workers, Waking up and Loading model are a worker that is initializing, and Inactive is an endpoint with max workers set to 0.
Migrate with a coding agent
Install the Runpod plugin or skills and connect the MCP server.- Claude Code
- Other agents
Troubleshooting
The handler raisesKeyError on the input. The client is still sending the bare model input. Wrap it as {"input": {...}}. If the handler receives a nested input inside job["input"], the client wrapped it twice.
The client gets a 200 but no result. /runsync returned before the job finished, or the code reads the top level of the response. Check status, read output, and switch long requests to /run plus /status.
The model reloads on every request. The load is inside handler (a direct port of predict that calls self.load()-style code). Move it to module scope.
The secret is empty. Code still reads kwargs["secrets"] or /secrets/NAME. Read the environment variable, and check that its value on the endpoint is {{ RUNPOD_SECRET_name }} with the exact secret name. Workers pick up a changed secret the next time they start.
The worker never becomes ready on a load balancing endpoint, and requests return 502. The server listens on a port the load balancer doesn’t check. Set PORT and PORT_HEALTH to the server’s port, expose that port, and make sure HEALTH_CHECK_PATH returns 200. A server ported from Baseten that reads PORT and expects 8080 needs the variable set explicitly.
The worker is stopped during startup with a large model. Initialization exceeded RUNPOD_INIT_TIMEOUT (420 seconds by default). Raise it in the endpoint’s environment variables, and use a cached model so the download doesn’t count against startup. Baseten’s default startup threshold is 30 minutes, so a model that loaded there can time out here.
Jobs sit in the queue with no workers. The GPU pool has no stock in the endpoint’s data centers, the endpoint is pinned to one data center by a network volume, or max workers is 0 because the endpoint went unused for 7 days. Add a second pool, widen the data centers, or raise max workers.
exec format error in the worker logs. The image was built for ARM. Rebuild with --platform linux/amd64.
The new image isn’t running after a push. The endpoint still references the old tag, or you pushed over a mutable tag such as :latest. Push a new version tag and patch the endpoint’s image.
The webhook receiver rejects Runpod’s callbacks. It still verifies X-BASETEN-SIGNATURE and parses data. Update it for the Runpod job payload.
Next steps
Handler functions
Streaming, async, and concurrent handlers, error handling, and payload limits.
Endpoint settings
Workers, scaling, timeouts, FlashBoot, and data centers.
Runpod SDK
Define and deploy endpoints from Python with
runpod.App.Cached models
Start workers on hosts that already hold your Hugging Face model.