CH·02CLI reference

gpu endpoints create

Create an endpoint

Create an endpoint

Synopsis

Creates an endpoint named NAME (lowercase letters, digits, and hyphens). Nothing starts until the first request, unless --min-replicas keeps machines warm. A cold start takes a few minutes while the image and the model download.

For a gated model, store a Hugging Face token first (gpu secrets set HF_TOKEN) and pass --hf-token-secret HF_TOKEN.

The output shows the endpoint's URL and an example curl.

gpu endpoints create NAME --model HF_ID --gpu GPU_TYPE [flags]

Examples

  gpu endpoints create qwen-small --model Qwen/Qwen2.5-0.5B-Instruct --gpu rtx_4090
  gpu endpoints create llama --model google/gemma-2-9b-it --gpu l40s \
    --hf-token-secret HF_TOKEN --max-replicas 2 --idle-timeout 600

Options

      --gpu string               GPU type for each machine (24 GB or more), such as rtx_4090; see gpu gpu-types
  -h, --help                     help for create
      --hf-token-secret string   Name of a secret holding a Hugging Face token, delivered as HF_TOKEN
      --idle-timeout int         Seconds without requests before a machine is released (at least 60) (default 300)
      --max-price float          Highest hourly price per machine, in USD
      --max-replicas int         Most machines under load (at most 4) (default 1)
      --min-replicas int         Machines kept warm even with no requests
      --model string             Hugging Face model id, such as Qwen/Qwen2.5-0.5B-Instruct
      --target-concurrency int   Requests in flight per machine before another starts (default 16)

Options inherited from parent commands

      --api-base string   API base URL (override with GPUAI_API_BASE env) (default "https://api.gpu.ai/v1")
      --debug             Enable debug logging to stderr
  -o, --output string     Output format: table|json (default table on TTY, json otherwise)

SEE ALSO

  • gpu endpoints - Serve an open model as an OpenAI-compatible endpoint

← The gpu CLI