gpu.aiDocs
UPDATED 2026.11.18READ 5 MINEDIT ON GITHUB →
CH·24Inference

Model endpoints.

Pick a Hugging Face model and a GPU. GPU.ai runs it with vLLM and gives you a URL that any OpenAI client can call with a GPU.ai API key.

§ 23.1Create an endpoint

In the console, open Endpoints and choose New endpoint. From the CLI or the API:

SHELL
gpu endpoints create qwen-small --model Qwen/Qwen2.5-0.5B-Instruct --gpu rtx_4090
gpu endpoints list
API
curl -X POST https://api.gpu.ai/v1/endpoints \
  -H "Authorization: Bearer $GPUAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "qwen-small", "model": "Qwen/Qwen2.5-0.5B-Instruct", "gpu_type": "rtx_4090"}'

The response has the endpoint's url. Nothing starts and nothing is billed until the first request.

§ 23.2Call it

SHELL
curl -N https://api.gpu.ai/v1/endpoints/ENDPOINT_ID/openai/v1/chat/completions \
  -H "Authorization: Bearer $GPUAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "stream": true,
       "messages": [{"role": "user", "content": "Say hello in one sentence."}]}'
PYTHON
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.gpu.ai/v1/endpoints/ENDPOINT_ID/openai/v1",
                api_key=os.environ["GPUAI_API_KEY"])
reply = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(reply.choices[0].message.content)

Endpoints serve chat/completions, completions, and models, with streaming. The model field is always set to the endpoint's model, so any value works.

§ 23.3Scaling and cold starts

With the default minimum of 0 machines, an endpoint scales to zero. The next request waits while a machine starts, pulls the vLLM image, and downloads the model, which takes a few minutes. If it waits longer than 10 minutes the request is answered 503 with a Retry-After header. Under load the endpoint adds machines, up to the maximum, and a machine with no request for the idle period is released, down to the minimum. Set a minimum of 1 to keep a machine warm.

§ 23.4Billing

Each machine is billed for its time from your GPU.ai balance, like an instance, from when vLLM is serving until the machine is released. Requests themselves are not billed. The machines also appear in your instance list, named after the endpoint.

§ 23.5Gated models

Save a Hugging Face token as a secret, then name it on the endpoint. Machines receive it as HF_TOKEN.

SHELL
gpu secrets set HF_TOKEN
gpu endpoints create llama --model google/gemma-2-9b-it --gpu l40s --hf-token-secret HF_TOKEN