How Together AI maps to Runpod
What’s different
What works differently
You deploy the model before you call it. On Together serverless, a model string is enough. On Runpod, unless the model is a Public Endpoint, you create a Serverless endpoint first and call its endpoint ID. Plan for one endpoint per model you use. The base URL carries the endpoint. Together uses one base URL (https://api.together.ai/v1, or https://api-inference.together.ai/v1 for dedicated endpoints) and selects the model with the model field. A Runpod vLLM endpoint has its own base URL, https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1, and model must be the Hugging Face ID you deployed or the name you set with OPENAI_SERVED_MODEL_NAME_OVERRIDE. Code that switches models by changing one string needs a small routing table from model to base URL.
Billing moves from tokens to worker seconds. A self-hosted endpoint bills for every second a worker runs: loading the model, processing requests, and waiting out the idle timeout. Token counts don’t appear on the bill. Whether that works in your favor depends on utilization: a worker that batches many concurrent requests spreads its cost across all of them, and a worker that wakes up for one short request pays for the cold start too. Measure with your own traffic before you commit. Public Endpoints keep usage-based pricing.
Cold starts are yours to manage. Together serverless models are always loaded. A Runpod endpoint with zero active workers scales to zero, and the first request after that waits for a worker to start and load the weights. A synchronous request that waits on a cold worker can exceed the gateway’s time limit and fail with a 524. You control this with active workers (always warm, always billed), the idle timeout, cached models, and FlashBoot. See Step 6.
Model features are engine settings. Together’s catalog marks each model for function calling and structured outputs. On a vLLM endpoint you turn tool calling on with ENABLE_AUTO_TOOL_CHOICE and TOOL_CALL_PARSER, set a reasoning parser with REASONING_PARSER, and set context length with MAX_MODEL_LEN. The right values depend on the model. See Configure vLLM to work with your model.
You pick the quantization. Together’s catalog lists the quantization each serverless model runs at. On Runpod, you get whatever precision the Hugging Face repository you deploy contains, so outputs can differ slightly from Together’s for the same model family. Compare outputs on your own evaluation set.
Rate limits are per endpoint. Together applies dynamic limits and may return 429 or 503 under load. Runpod applies documented per-endpoint limits to each operation, and real throughput is bounded by your max workers and each worker’s concurrency (MAX_CONCURRENCY on the vLLM worker).
Async jobs replace long HTTP calls. Runpod queue-based endpoints have a native job API (/run, /status, /stream, /cancel) with webhooks, alongside the OpenAI-compatible route. Together’s video API and container queue API follow a similar submit-and-poll pattern, so those translate directly.
What has no direct equivalent
Requirements
- A Runpod account with credits and a Runpod API key.
- Access to your Together AI projects, with the Together CLI 3.0 or later (
uv tool install "together[cli]" --upgrade) andTOGETHER_API_KEYset for each project you’re migrating. - A Hugging Face account and access token if you use gated models or need a place to store fine-tuned weights.
- Docker, if you run custom containers (dedicated container inference).
- An evaluation set of real prompts and expected outputs. You’ll use it in Step 8 to compare the two platforms.
Step 1: Take inventory of what you run on Together AI
Together API keys are scoped to a project, so repeat this step for each project. Start by confirming which project the CLI is pointed at.List billable resources
Run the following commands and save the JSON. Each one covers a different Together product.minReplicas and maxReplicas, the scaling metric and target, the inactivity timeout, the placement regions, and the endpoint string (PROJECT_SLUG/ENDPOINT_NAME) your code passes as model.
Endpoints created on Together’s older v1 API don’t appear in CLI 3.0. List them with the v1 REST API.
pyproject.toml that Jig deploys from. The [tool.jig.image] and [tool.jig.deploy] tables hold the start command, GPU type and count, CPU, memory, storage, port, replica bounds, environment variables, and volume mounts.
For each cluster, record the GPU type, GPU count, cluster type (KUBERNETES or SLURM), region, billing type, driver and CUDA versions, and the shared volumes attached.
Find serverless usage
Serverless models aren’t resources you can list, so the inventory comes from your code and your bill. Together’s cost analytics page in the console breaks spend down by project and API key. In your codebase, search for the SDKs, hostnames, and environment variables.api.together.xyz hostname in case older code still uses it. Also check frameworks that wrap Together behind their own provider name, such as LangChain, LiteLLM, and the Vercel AI SDK. For every call site, record the model string, the API (chat, completions, images, video, audio, rerank, batch), and the features in use: streaming, tools, response_format, image inputs, and logprobs.
Step 2: Set up Runpod
Create an API key in the Runpod console and export it. See Manage API keys for restricted and read-only keys.runpodctl to create and manage resources from your terminal, and the Runpod SDK if you’ll call endpoints from Python or write your own workers.
runpodctl doctor walks through first-time setup of your API key and SSH key. For agent-driven work, connect the Runpod MCP server. If you manage infrastructure as code, the Terraform provider (runpod/runpod, ~> 2.0) covers endpoints, templates, network volumes, Pods, and clusters.
Step 3: Choose where each workload lands
Use this table for every item in your inventory.Model map for Public Endpoints
These are the Together serverless model strings that have a Public Endpoint in the same family, based on the Runpod model reference and model pages. Versions and parameters differ between the two platforms, so read the linked page before you switch. Every Together model not in this table lands on your own endpoint.
Runpod’s text catalog also includes IBM Granite 4.0, and its audio catalog has Chatterbox Turbo and Minimax Speech 02 HD for text-to-speech. None of Together’s serverless text-to-speech or speech-to-text models has a Public Endpoint with the same model. The catalog changes, so check the Hub for the current list.
GPU map
Together’s dedicated hardware is listed below against the Runpod GPU type ID you pass to--gpu-id and the Serverless pool that contains it, from the GPU types reference.
You don’t have to match Together’s hardware. Together’s dedicated hardware list starts at H100, and many models fit on smaller cards: the vLLM worker docs suggest 16 to 24 GB for models under 10B parameters and 32 to 48 GB for 10B to 30B. Choose the GPU that fits the model and context length you serve, using the VRAM estimates. Stock varies by data center and changes through the day, so check what’s available before you deploy and list more than one pool on production endpoints.
Step 4: Move your images and code
If you only call Together’s model APIs, you have no images to move. Skip to Step 5.Sprocket workers (dedicated container inference)
A Sprocket worker implementssetup(), which runs once at startup, and predict(args), which runs once per job and receives the job’s payload. A Runpod queue-based worker does the same two things with module-level code and a handler function that receives the request’s input.
Here’s the worker from Together’s container quickstart.
hello_world.py
handler.py
setup() moves to module level so it loads once per worker. The return value becomes the job’s output. If predict() raises an exception, Sprocket marks the job failed, and an exception in a Runpod handler does the same. Four Sprocket behaviors need explicit code on Runpod:
- Progress updates. Replace
emit_info(...)withrunpod.serverless.progress_update(job, ...). - Input files. Sprocket downloads any HTTPS URL in the payload and hands your code a local path. A Runpod handler receives the URL as sent, so download it in the handler.
- Output files. Sprocket uploads a returned
FileOutput(...)to Together storage and puts its URL in the result. A Runpod handler uploads the file to your own storage and returns the URL. - HTTP server mode. A Sprocket worker that serves
predict_pathdirectly becomes a load balancing endpoint: your own HTTP server with a/pinghealth check.
Container image
Jig builds the image frompyproject.toml and pushes it to Together’s registry. On Runpod you build and push it yourself, and the image must be linux/amd64. A minimal Dockerfile for the handler above:
Dockerfile
tg beta jig dockerfile prints the Dockerfile Jig generates, which is a useful starting point for the system packages your worker needs.
Jig configuration
Translate the[tool.jig.deploy] table using the configuration field map. Environment variables in [tool.jig.deploy.environment_variables] become template environment variables, Jig secrets become Runpod secrets, and Jig volume mounts become a network volume (mounted at /runpod-volume on Serverless) or files baked into the image. Remove any code that reads TOGETHER_DEPLOYMENT_NAME or other Together runtime variables. Inside a Runpod worker, RUNPOD_ENDPOINT_ID identifies the endpoint.
Prebuilt vLLM images
If you deploy stock vLLM through Jig, you don’t need to port the image. Use the vLLM worker from the Hub in Step 6, and move yourvllm serve flags to environment variables: each flag becomes an upper snake case variable (--max-model-len 8192 becomes MAX_MODEL_LEN=8192), and VLLM_EXTRA_ARGS passes anything else through.
Step 5: Move your data and models
Base models
If you serve a stock open model, there’s nothing to copy. The vLLM worker pulls the model from Hugging Face, and cached models keep a copy on Runpod hosts so workers don’t download it on every cold start. Find the Hugging Face repository that corresponds to your Together model string. Together’s IDs follow the sameorganization/model convention but aren’t always the Hugging Face repository name, and Together may serve a quantized build. For gated models, accept the license on Hugging Face and create an access token.
Fine-tunes
Together lets you download the weights of a completed fine-tuning job. Choose the checkpoint type that matches how you’ll serve it:merged gives you self-contained weights, and adapter gives you only the LoRA adapter. Full fine-tunes download as the full model.
.tar.zst archive. Decompress it to get a standard Hugging Face model directory.
tg models download MODEL_ID ./local-dir, or go back to the original source.
Then put the weights where a Runpod worker can load them. You have two documented options.
- Hugging Face (recommended)
- Network volume
Create a private Hugging Face repository, then push the directory to it. This works with cached models, so cold starts don’t re-download the weights.Create the repository first.
hf upload creates a missing repository for you, and a repository created that way isn’t private unless you ask for it.In Step 6, set MODEL_NAME to YOUR_HF_USER/my-finetune and add HF_TOKEN to the endpoint so workers can read the private repository.A LoRA adapter served through the vLLM worker’s
LORA_MODULES setting is loaded from Hugging Face. If you can’t publish the adapter there, serve the merged weights instead.tg files retrieve-content FILE_ID --output ./data), and fine-tuning expects JSONL or Parquet, which you can reuse as is. To train on Runpod, start a GPU Pod with a network volume, run your training script, and write the output to the volume so it outlives the Pod.
/workspace on the Pod and at /runpod-volume on Serverless workers, so an adapter written to /workspace/outputs during training is at /runpod-volume/outputs when you serve it. Delete the Pod when training finishes. For a guided setup with Axolotl, see Fine-tune a model. Hyperparameters don’t carry over automatically: Together’s job settings (epochs, learning rate, LoRA rank) are inputs to your own script now.
Batch files and cluster data
Together keeps batch input, output, and error files for 7 days. Download any results you still need before you shut down (tg batches download BATCH_ID --output ./out).
For GPU clusters, copy the contents of each shared volume to a Runpod network volume. From a node that mounts the shared volume (at /data on Slurm clusters), sync to the network volume through the S3-compatible API with the same aws s3 sync command shown above. Data on Together’s local NVMe disks is ephemeral, so only the shared volume needs to move.
Step 6: Deploy on Runpod
Public Endpoints
There’s nothing to deploy. Confirm your account has credits, then go to Step 7.A vLLM endpoint for an open model
This is the landing spot for most Together serverless models and all dedicated endpoints. The example servesQwen/Qwen3-8B with tool calling enabled. Replace the model, GPU pool, and vLLM settings with the ones for your model from Configure vLLM to work with your model.
- runpodctl
- Console
Look up the Hub listing ID for the vLLM worker, then create the endpoint from it.
--hub-id pulls the worker image, container disk size, and default environment from the Hub release, and --env overrides individual values. --gpu-id accepts a pool ID or a GPU type ID; set it explicitly so the endpoint runs on the hardware you sized for. --model-reference attaches the Hugging Face model as a cached model: the :main suffix is the branch, tag, or revision, and the flag requires a GPU endpoint. The command prints the endpoint ID. For a gated or private model, store the token as a secret named hf_token and add --env HF_TOKEN='{{ RUNPOD_SECRET_hf_token }}', so the token itself never appears in the command or the endpoint configuration./run route so a slow first load can’t time out, then use the OpenAI-compatible route once a worker is warm.
Match a dedicated endpoint’s scaling
A Together deployment’s replica bounds map onto worker counts. Set active workers to your oldminReplicas and max workers to your old maxReplicas.
Before, on Together:
runpodctl builds name these flags --scaler-type and --scaler-value on serverless update. Run runpodctl serverless update --help to see which your build accepts.
With --workers-min 1, one worker stays loaded and bills continuously, the same trade you made with minReplicas. Request count scaling adds workers based on queued plus in-progress requests divided by the scaler value, and it’s the recommended scaler for LLM workloads. Runpod has no inactivity timeout because it isn’t needed: with --workers-min 0, workers stop after the idle timeout and start again on the next request. To cut cold starts without paying for an always-on worker, keep the model cached, leave FlashBoot on, and raise the idle timeout (--idle-timeout, in seconds) so a worker stays warm between bursts. REST API v2 rejects an idle timeout on a queue-based endpoint that scales on request count, so pair a long idle timeout with queue delay scaling.
The same change through REST API v2:
A deployment that needed several GPUs on Together (for example, a config with a GPU count above one) needs
--gpu-count on Runpod. On multi-GPU workers, the vLLM worker sets tensor parallelism to the number of visible GPUs.Fine-tuned models and LoRA adapters
For merged weights, deploy exactly as above withMODEL_NAME pointing at your Hugging Face repository (and --model-reference set to the same repository), or at the path on an attached network volume (--network-volume-id NETWORK_VOLUME_ID with MODEL_NAME=/runpod-volume/my-finetune).
For a LoRA adapter, deploy the base model and attach the adapter with the vLLM worker’s LoRA settings.
MAX_LORA_RANK must be at least the rank you trained with (the default is 16). After the endpoint is up, call client.models.list() to see the names it serves, and pass the adapter’s name as model in requests.
Custom workers
For a ported Sprocket worker, create a Serverless template from your image, then an endpoint from the template.gpu_type = "none"), replace --gpu-id with --compute-type CPU. Environment variables for a template-based endpoint come from the template (runpodctl template create --env), not from serverless create --env.
Image, video, and audio workers
For models that aren’t Public Endpoints, deploy a prebuilt worker from the Hub. Search for one, check its release and GPU pool, and deploy it the same way as the vLLM worker.Instant Clusters
Deploy a cluster from the Instant Clusters page in the console: choose the cluster type (Slurm, PyTorch, or Ray), GPU type, node count, region, and network volume. See Slurm Clusters for the Slurm flow.sbatch and srun scripts from a Together Slurm cluster carry over, with paths changed from /data to wherever you mount the network volume. Kubernetes manifests don’t carry over: convert each job to a Slurm script or a container start command. Clusters are also available through REST API v2 (POST /v2/clusters) and Terraform (runpod_cluster).
Step 7: Update the code that calls Together AI
Chat completions through the OpenAI SDK
If you already call Together through an OpenAI client, change the API key, the base URL, and the model name.- Python
- TypeScript
- cURL
Before:After:
max_tokens on every request. The vLLM worker’s documented default is 16 tokens, which truncates most answers. The longer client timeout covers a request that arrives while a worker is starting. To keep your existing model strings in client code, set OPENAI_SERVED_MODEL_NAME_OVERRIDE on the endpoint to the name your code already sends.
For a model in the Public Endpoints map, the change is the same, with the endpoint’s slug in the base URL.
The together SDKs
Runpod doesn’t document pointing the together or together-ai packages at a Runpod base URL, so replace them with the OpenAI SDK for chat and completions. The method names and response shapes are the same for these calls.
- Python
- TypeScript
Before:After:
Together(base_url="https://api-inference.together.ai") and the endpoint string (PROJECT_SLUG/ENDPOINT_NAME) with the Runpod base URL and the served model name. Calls to client.images, client.videos, client.audio, client.rerank, client.batches, client.files, client.fine_tuning, and client.endpoints have no drop-in replacement and are covered below.
If you use the Vercel AI SDK, replace the Together provider with @runpod/ai-sdk-provider, which works with Public Endpoints and with your own endpoint IDs.
Feature parity for the OpenAI-compatible surface
Tool calling and structured output requests use the same OpenAI request fields on both platforms (
tools, tool_choice, response_format), so client code doesn’t change once the endpoint is configured for the model. Together’s logprobs shape differs from OpenAI’s, so code that parses it needs a rewrite either way.
Images, video, and audio
These calls move from Together’s OpenAI-style media routes to Runpod’s job API. The request body is wrapped ininput, and the result is under output.
Before, with the together SDK:
output.result, as most image and video models do, and the Flux models return it in output.image_url. Input names such as size, width, height, or aspect_ratio also vary by model. Output URLs expire after 7 days, so download anything you need to keep. Together’s video API already uses submit and poll (client.videos.create, then client.videos.retrieve until completed). On Runpod, submit with /run, poll /status/JOB_ID until COMPLETED, and read the URL from output.
Batch jobs
Together’s batch API takes a JSONL file where each line has acustom_id and a body. Runpod batch jobs take a JSON array where each element is a normal request with an input field, and they run on your endpoint’s dedicated batch workers.
Before:
messages or prompt, with sampling parameters under sampling_params), not the OpenAI body. There’s no custom_id field, so carry your own identifier inside input if your handler echoes it, or map results by the request IDs the API returns. Batch jobs are available on queue-based endpoints only. For smaller volumes, submit each request to /run with a webhook URL and let Runpod call you when each job finishes.
Container queue jobs
Together’s queue API and Runpod’s job API line up closely. Before:payload becomes input, request_id becomes id, and outputs becomes output. Runpod has no numeric job priority. The documented lowPriority policy flag only keeps a job from triggering worker scaling, so if you rely on Together’s priority ordering, run a separate endpoint for high-priority traffic. Use /runsync to wait for the result in one call, and add a top-level webhook field to be notified instead of polling. From Python, runpod.Endpoint("ENDPOINT_ID").run_sync({...}) wraps the same calls.
Control-plane automation
Replacetg commands in scripts and CI with runpodctl or REST API v2, using the CLI command map and API map. In CI, set RUNPOD_API_KEY as a secret, the same way you set TOGETHER_API_KEY.
Step 8: Verify, cut over, and shut down
Verify
Send real requests from outside Runpod, with the same client library and network path your application uses.- Run your evaluation set against both platforms and compare outputs. Expect differences where the model version or quantization differs.
- Test every feature from your inventory against the new endpoint: streaming, tool calls,
response_format, image inputs, and long prompts near your context limit. - Measure latency for a warm worker and for a cold start. If the cold start is too slow for your use, set active workers or raise the idle timeout.
- Run a load test at your peak concurrency and watch for
429responses and queue delay. Raise max workers orMAX_CONCURRENCYif requests queue. - Check the job counts and worker state.
Cut over
Run both platforms in parallel behind a configuration flag that selects the base URL, API key, and model name. Shift a small share of traffic to Runpod, compare error rates and latency, and increase the share in steps. Keep the Together path deployable until you’ve run at full traffic through at least one peak period. While both run, you pay for both.Set a cost guard on Runpod
Max workers is the ceiling on what an endpoint can spend per hour, so set it deliberately. Keep active workers at zero on anything that doesn’t need to be warm, keep the execution timeout enabled so a stuck job can’t run indefinitely, and delete test endpoints. Runpod also applies an account-wide spend limit per hour, and scales down endpoints that receive no requests for several days.Shut down Together AI
Serverless models stop costing money when you stop calling them. Everything else needs an explicit action. Deleting is permanent, so confirm each resource against your inventory first.tg endpoints ls, tg clusters list, and tg clusters storage list show nothing running. Reserved clusters are charged for the full reservation, so deleting one early doesn’t change what you’ve already paid. Finally, revoke the Together API keys your application no longer uses, in each project’s API key settings, and remove TOGETHER_API_KEY from your secret stores. If you have auto-recharge on, turn it off.
Reference
CLI command map
API map
Note the two Runpod hosts:
api.runpod.ai serves requests to endpoints, and api.runpod.io is the management API.
Configuration field map
Environment variable map
Status map
See Job states for the full list on Runpod.
Migrate with a coding agent
A coding agent can run the inventory, propose the mapping, and make the code changes. Install the Runpod skills and the MCP server first.- Claude Code
- Other agents
The plugin installs the skills and connects the hosted MCP server.
Troubleshooting
The first request returns a 524 or times out. The request waited on a worker that was still starting and loading the model, and the synchronous route gave up. Send the first request with/run and poll /status, raise your client timeout, and for latency-sensitive traffic set active workers to 1 or more. Check worker state with the /health route.
Responses stop after a few words. The vLLM worker’s documented default for max_tokens is 16, so code that never set max_tokens gets truncated output. Set it on every request.
The API returns an invalid model error. model must match the Hugging Face ID you deployed, or the value of OPENAI_SERVED_MODEL_NAME_OVERRIDE. Together model strings and endpoint strings (PROJECT_SLUG/ENDPOINT_NAME) don’t resolve on Runpod. Call client.models.list() to see what the endpoint serves.
Requests return 401. The client is still sending the Together key. Use your Runpod API key as the bearer token. If the key is restricted, confirm it has access to this endpoint.
Tool calls come back as plain text. Tool calling isn’t enabled for the model, or the parser doesn’t match the model’s format. Set ENABLE_AUTO_TOOL_CHOICE=true and the TOOL_CALL_PARSER for your model family from Configure vLLM to work with your model.
Workers fail to start or run out of memory. The model plus its KV cache doesn’t fit on the GPU you chose. Together selected hardware for you, and on Runpod this is your decision. Lower MAX_MODEL_LEN, lower GPU_MEMORY_UTILIZATION, use a quantized repository, or move to a larger GPU pool. For models that take longer than the default 420 seconds to load, raise RUNPOD_INIT_TIMEOUT.
The endpoint stays at zero workers. The GPU pool may be out of stock in the endpoint’s data centers, or the endpoint’s CUDA version constraint may exclude the machines that have stock. Add more pools, remove data center restrictions, and check the endpoint’s CUDA settings. A cached model that isn’t on any host yet also delays the first worker while it downloads, which shows up as a job sitting in the queue.
A gated model fails to download. Add HF_TOKEN to the endpoint with a token that has accepted the model’s license. Together handled model access for you. On Runpod the worker pulls from Hugging Face with your credentials.
Reasoning text is missing from streamed responses. The field name differs between serving stacks: Together documents reasoning or reasoning_content depending on the model, and vLLM’s field depends on its version. Read both fields in your client, and set REASONING_PARSER on the endpoint.
The bill is higher than expected at low traffic. You’re paying for worker seconds, including model load on each cold start and the idle window after each request. Check whether active workers is set above zero, how long the idle timeout is, and how often workers cold start. For a model that Runpod hosts, a Public Endpoint may fit low, bursty traffic better than your own endpoint.
Next steps
vLLM on Serverless
Configure, deploy, and call OpenAI-compatible LLM endpoints.
Endpoint configurations
Tune active workers, scaling, timeouts, and FlashBoot.
Public Endpoints model reference
Browse the hosted models and their parameters.
Cached models
Cut cold starts by caching Hugging Face models on Runpod hosts.