CH·02CLI reference

gpu endpoints

Serve an open model as an OpenAI-compatible endpoint

Serve an open model as an OpenAI-compatible endpoint

Synopsis

An endpoint serves a Hugging Face model with vLLM on a GPU type you pick, behind a stable OpenAI-compatible URL. Point any OpenAI client's base_url at it and use a GPU.ai API key.

Machines start on the first request, scale out under load up to --max-replicas, and are released after --idle-timeout seconds without requests (down to --min-replicas, zero by default). You pay for machine time while a machine is up, at the instance price.

Endpoints are off by default. On a deployment that does not offer them, these commands fail with this server has no such endpoint. See the endpoints guide.

Examples

# Create, then call it
gpu endpoints create qwen-small --model Qwen/Qwen2.5-0.5B-Instruct --gpu rtx_4090
gpu endpoints get qwen-small

# Delete it (its machines are released)
gpu endpoints delete qwen-small --yes

Options

  -h, --help   help for endpoints

Options inherited from parent commands

      --api-base string   API base URL (override with GPUAI_API_BASE env) (default "https://api.gpu.ai/v1")
      --debug             Enable debug logging to stderr
  -o, --output string     Output format: table|json (default table on TTY, json otherwise)

SEE ALSO

← The gpu CLI