Batch Jobs
Run a container on a GPU until it exits, and let small jobs share a GPU by memory with the GPU Memory Scheduler.
A job is a container and a command that runs on one GPU until it exits. You submit it, GPU.ai starts a machine for it, and the machine is shut down a few minutes after your last job finishes.
Jobs are part of the GPU memory scheduler. You can submit them with the gpu CLI, the API, or the console under Jobs.
Your first job
You need an API key (gpu login, or create one under API Keys in the console) and a balance: the machine a job runs on is billed to your account, and a job from an account with no balance is refused with 402 insufficient_balance. Add credits under Billing first.
This job prints the GPU it was given and exits. It uses a public image, so it runs as written.
gpu jobs submit --name hello-gpu \
--image nvidia/cuda:12.4.1-base-ubuntu22.04 \
--gpu-type rtx_a6000 \
-- nvidia-smi
gpu jobs get <id> --wait 15m # the id is printed by submit
gpu jobs logs <id>
The first job starts a machine, which usually takes 3 to 10 minutes. The machine is shut down a few minutes after your last job finishes, so the whole run costs a few cents.
Jobs run on a subset of GPU.ai's suppliers, so not every type in gpu gpu-types list can run a job. rtx_a6000, a100_80gb, h100_pcie, l40s and rtx_6000_ada are the types most often available. A job asking for a type with no machine available waits in the queue with the reason in status_reason for up to 24 hours, then fails. Cancel it with gpu jobs cancel and pick another type.
Submit a job
This example is a template for your own work. Replace the image and the command with yours.
gpu jobs submit \
--name nightly-eval \
--image ghcr.io/acme/eval:1.4 \
--gpu-type a100_80gb \
--memory-hint-mb 9000 \
--compute-hint-pct 30 \
--allow-sharing --allow-restart --max-wait 10m \
-- python eval.py --suite full
The same request over the API:
curl -X POST https://api.gpu.ai/v1/jobs \
-H "Authorization: Bearer $GPUAI_API_KEY" -H "Content-Type: application/json" \
-d '{
"name": "nightly-eval",
"image": "ghcr.io/acme/eval:1.4",
"command": ["python", "eval.py", "--suite", "full"],
"gpu_type": "a100_80gb",
"memory_hint_mb": 9000,
"compute_hint_pct": 30,
"allow_sharing": true,
"allow_restart": true,
"max_wait_seconds": 600
}'
| Field | Meaning |
|---|---|
image | Container image. It must be public, or pullable by the machine without credentials. |
command | The program and its arguments. It replaces the image's entrypoint. |
env | Environment variables. Values are stored to run the job and are never returned by the API. |
gpu_type or min_vram_gb | The GPU model, or the least GPU memory you accept. One is required. |
memory_hint_mb | The peak GPU memory you expect the job to use. Optional. |
compute_hint_pct | The share of time you expect the job to keep the GPU busy, 1 to 100. Optional. |
timeout_seconds | The longest the job may run. Default 3600, maximum 259200. |
max_wait_seconds | How long the job may wait for space on a machine you already have. Default 0. |
allow_sharing | The job may share a GPU with your own other jobs. Default false. |
allow_restart | The job may be stopped and started again. Default false. |
checkpoint_path | Where the checkpoint directory is mounted. Default /gpuai/checkpoint. |
max_retries | Retries after running out of GPU memory, 0 to 5. Default 2. |
max_price_per_hour | The highest hourly machine price you accept. Optional. |
secrets | Names of organization secrets to deliver as environment variables of the same names. The values are not stored with the job. Optional. |
Follow it with gpu jobs get , gpu jobs logs , and gpu jobs telemetry . Cancel with gpu jobs cancel .
To hand the job an API key or token, store it once as an organization secret and name it with --secret NAME (or "secrets": ["NAME"] in the API). The job's container gets it as the environment variable NAME. Unlike env, the value is never stored with the job.
The three opt-ins
A job gets a whole GPU unless you say otherwise. Each opt-in gives up something specific.
- Sharing (
allow_sharing). The job may run on a GPU beside your own other jobs. It never shares a GPU with another customer. A job that keeps the GPU's compute busy all the time would run slower beside neighbors, so jobs share a GPU only while their compute adds up to 100% or less (see below). - Restart (
allow_restart). The job may be stopped and started again. Sharing requires it, because the way a job that outgrows its memory is handled is to stop it and run it again with more. - Waiting (
max_wait_seconds). When nothing fits right now, the job waits up to this long for space on a machine you already have, in place of starting another machine straight away.
Hard isolation
isolation: "hard" (--isolation hard) runs the job in a GPU slice of its own (NVIDIA MIG) on an A100 or H100: its own memory, its own compute and its own fault domain, so no other job on the card can slow it, take its memory or crash it. The slice is the smallest that holds the memory hint or the job's history; without either, or when no slice is large enough, the job gets a whole GPU. A slice has a share of the GPU's compute in proportion to its size. It needs gpu_type set to a model with slices (a100_80gb or h100_pcie) and cannot be combined with allow_sharing. The job response shows the slice in mig_profile. Where hard isolation is not available, the request is refused with a 422.
How much memory a job is given
A shared job is given a reservation of GPU memory. It comes from, in order:
- Your hint. The reservation is the hint plus 15% plus 512 MiB.
- History. With no hint, the job is sized from earlier finished runs: first your runs under the same name (digits are ignored, so
eval-17andeval-18are the same job), then your runs of the same image, then runs of that image by at least three customers. Three earlier runs are needed. The reservation is the highest recent peak plus 20% plus 1024 MiB. - Neither. The job gets a whole GPU.
reservation_mb, whole_gpu, and prediction_source.
The reservation is not a hard cap. Nothing stops a container from asking the GPU for more. The reservation is passed to the job in GPUAI_MEMORY_LIMIT_MB so it can limit itself, and the machine watches memory every two seconds: when a GPU is nearly full, a job that is over its reservation is stopped (an eviction) and retried.
How much compute a job is counted for
Sharing a GPU frees memory, not compute. Two jobs that each keep a GPU busy all the time take as long side by side as one after the other, and the batch is slower overall once the switching between them is counted. So a job joins a GPU that already has jobs only when its memory fits, the compute duties of all the jobs on it add up to 100% or less, and sharing is predicted to get more work done than running them one at a time (at least 10% more) without slowing any of them by more than a quarter. A job's duty is the share of time it keeps the GPU busy:
- Your hint,
compute_hint_pct. - History. With no hint, the duty measured on earlier finished runs, looked up the same way as memory (same name, then same image). One earlier run is enough to be considered. The estimate is the runs' mean plus a safety margin that shrinks as the job is measured more: a long run counts for more than a short one, and many consistent runs for more than one.
- Neither. The job counts as 100%: it gets a GPU to itself the first time, is measured, and can share from then on.
predicted_compute_pct and where it came from in compute_prediction_source, and what was measured on its last run in util_mean_pct (mean GPU utilization) and busy_pct (share of samples at 50% or more). A job whose utilization comes in on/off bursts has the two close together; a steady job has one near 0 or 100.
How it is measured: the share of time the GPU was busy over the part of the run that used it (start-up and shutdown are left out), sampled four times a second. Where the GPU reports utilization per process, a job's own figure is used even while it shares (compute_source: process on the attempt); otherwise it is taken while the job is the only one on its GPU (solo). A job that shared its GPU for its whole run without per-process figures gets the whole GPU's figure, which over-states it (shared_upper_bound); it is used only until a better measurement exists.
What to expect: packing helps jobs that leave the GPU idle much of the time, such as evaluations, sweeps over small models, preprocessing, and training that waits on its input. A job that keeps the GPU busy will mostly run alone and lose nothing. When jobs with bursty use do share, their bursts sometimes collide, and each runs somewhat slower than it would alone while the batch as a whole finishes sooner.
Keep your job from taking the whole card
Many frameworks reserve most of the GPU's memory at start, whatever the job needs. A job that does that cannot share. Each job is started with these settings unless it sets them itself:
| Variable | Value | Effect |
|---|---|---|
PYTORCH_CUDA_ALLOC_CONF | expandable_segments:True | PyTorch grows its memory pool in place instead of in large fixed blocks |
TF_FORCE_GPU_ALLOW_GROWTH | true | TensorFlow allocates as it goes instead of taking the whole GPU |
XLA_PYTHON_CLIENT_PREALLOCATE | false | JAX does not take 75% of the GPU at start |
GPUAI_MEMORY_LIMIT_MB always holds the job's reservation, so the job can size itself:
- PyTorch needs nothing; it takes memory as it uses it. To make it refuse to grow past the reservation, use
torch.cuda.set_per_process_memory_fractionas in the example below. - TensorFlow and JAX are covered by the defaults above. JAX can also be held to a fraction with
XLA_PYTHON_CLIENT_MEM_FRACTION. - Inference servers reserve a fixed share of the GPU at start and need their own setting. For vLLM, pass
--gpu-memory-utilizationas the reservation over the GPU's total, for example--gpu-memory-utilization $(python3 -c "import os; print(round(int(os.environ['GPUAI_MEMORY_LIMIT_MB'])/int(os.environ['GPUAI_GPU_TOTAL_MB']), 2))"). Without it a vLLM server takes 90% of the card, and sizing from history will learn that 90%.
When a job runs out of memory
A job that exits with a GPU out-of-memory error, or is evicted, is run again automatically when it allows restart:
- the first time, with 1.5 times its reservation or its observed peak, whichever is larger
- the second time, with a whole GPU
- with a whole GPU already, it fails, because there is nothing larger to give it on that GPU model
max_retries times. Any other failure, such as a non-zero exit from your own code, is not retried. The job response counts retries, oom_exits, and evictions, and attempts lists each run.
The checkpoint directory
Every job gets a directory that survives a restart on the same machine. Write your progress there and read it back at start.
| Variable | Value |
|---|---|
GPUAI_CHECKPOINT_DIR | The checkpoint directory inside the container |
GPUAI_ATTEMPT | 1 for the first run, 2 for the first retry, and so on |
GPUAI_MEMORY_LIMIT_MB | The job's GPU memory reservation |
GPUAI_GPU_TOTAL_MB | The GPU's total memory |
GPUAI_GPU_SHARED | 1 when the GPU is shared with your other jobs, 0 when the job has it alone |
GPUAI_JOB_ID | The job id |
import os, torch
ckpt = os.path.join(os.environ["GPUAI_CHECKPOINT_DIR"], "state.pt")
start = 0
if os.path.exists(ckpt):
state = torch.load(ckpt)
model.load_state_dict(state["model"])
optimizer.load_state_dict(state["optimizer"])
start = state["step"]
# Optional: have PyTorch itself refuse to grow past the reservation.
limit = int(os.environ["GPUAI_MEMORY_LIMIT_MB"])
total = int(os.environ["GPUAI_GPU_TOTAL_MB"])
if os.environ.get("GPUAI_GPU_SHARED") == "1":
torch.cuda.set_per_process_memory_fraction(min(1.0, limit / total))
for step in range(start, total_steps):
train_one_step()
if step % 100 == 0:
tmp = ckpt + ".tmp"
torch.save({"model": model.state_dict(), "optimizer": optimizer.state_dict(), "step": step + 1}, tmp)
os.replace(tmp, ckpt)
The directory is on the machine's disk. If the machine itself is lost, the job is queued again on another machine and starts with an empty directory. For checkpoints that must survive that, write them to your own storage.
Billing
The machines that run your jobs are ordinary instances in your account. They appear in your instance list, and they are billed the normal way, by the hour at the listed price, from the moment the machine is ready until it is shut down. That is the bill of record.
Each job also shows gpu_seconds, the time it ran on a GPU, and attributed_cost_usd, its share of that GPU's price: while several of your jobs share a GPU, each is attributed an equal share of the GPU's price for that time. A job that shared a GPU therefore shows a lower cost than the same job on a GPU alone. Machine time with no job running (startup, and the few minutes a machine is kept after its last job) is billed but not attributed to any job, so the attributed costs add up to a little less than the machine bill.
Limits
- One GPU per job.
- Sharing is only between jobs from the same account.
- Images must be public or pullable by the machine without credentials.
- The memory reservation is enforced by stopping a job after the fact, not by a hard cap, unless the machine runs in MPS mode (an operator setting, off by default), where it is a hard per-job limit.
- The checkpoint directory survives restarts on the same machine only.
- Job output is kept up to about 8 MB per attempt.