Functions.

§ 21.1What it is¶

You install one Python package, put a decorator on a function, and call it. The call runs on a GPU machine that GPU.ai starts for that function. The machine stays ready (“warm”) for an idle period after each call, so the next call skips start-up. When the idle period passes with no calls, the machine is released, and a function nobody calls costs nothing.

Functions is a separate product from the Inference API. The Inference API serves hosted models and bills per token. Functions runs your own code on a machine and bills for the machine's time.

§ 21.2Quickstart¶

Functions is part of the GPU.ai Python SDK, gpuai-sdk (Python 3.10 or newer), and is imported from gpuai_sdk.functions. Install the package and set your API key. Create the key in the console under API Keys. A key with full access works. A restricted key needs the functions scope with write access.

pip install gpuai-sdk
export GPUAI_API_KEY=gpuai_...

Then decorate a function and call it with .remote(). This one asks for any GPU with at least 16 GB of VRAM and reports what it got.

from gpuai_sdk.functions import function


@function(min_vram_gb=16, idle_timeout=300)
def gpu_info():
    import subprocess

    out = subprocess.run(
        ["nvidia-smi", "--query-gpu=name,memory.total", "--format=csv,noheader"],
        capture_output=True, text=True, check=True,
    )
    return out.stdout.strip()


if __name__ == "__main__":
    print("First call (starts a machine):", gpu_info.remote())
    print("Second call (machine is warm):", gpu_info.remote())

§21.2.1Cold and warm calls¶

The first call is a cold start. GPU.ai starts a machine, installs your pip requirements, and then runs the function. In our tests (about 40 cold calls across 12 GPU types) the middle case was 2 to 4 minutes: about a minute on a host that already has the GPU.ai image, a minute more when pip installs PyTorch, and 5 to 8 minutes on a virtual machine or a host that first downloads the image. A machine that is not ready after 10 minutes is replaced on a different supplier, so the slowest cold calls took about 18 minutes. The second call lands on the same machine while it is still warm and adds about 2 seconds to the time the function itself runs. The machine stays warm for idle_timeout seconds after each call (300 by default), and you pay for that time. Set idle_timeout=0 to release the machine as soon as a call returns.

§ 21.3Three ways to call¶

result = f.remote(1, 2)        # run on GPU.ai and wait for the result
handle = f.spawn(1, 2)         # run on GPU.ai in the background
result = f(1, 2)               # run on this computer, as if undecorated

.spawn() returns a handle straight away. Use it to check on the call, read its output, wait for the result, or stop it.

handle.status()                # queued, setting_up, running, succeeded, failed, timed_out, cancelled
handle.result(timeout=60)      # wait up to 60 seconds for the result
handle.logs()                  # everything the call printed so far
handle.cancel()                # stop it
handle.info()["cost_usd"]      # what the call cost, once it has finished

§ 21.4Examples¶

Three complete programs. Each runs as written once GPUAI_API_KEY is set.

§21.4.1Run a model on a GPU¶

Loads a small sentiment model with PyTorch and classifies a few sentences. The model is loaded on the first call a machine serves and stays in GPU memory, so later calls on that warm machine skip the load. torch is listed in pip so the example works on any Python version. On Python 3.10 it is already installed on the machine, so nothing is downloaded for it.

from gpuai_sdk.functions import function

MODEL = "distilbert-base-uncased-finetuned-sst-2-english"
_loaded = {}  # filled on the machine, on the first call it serves


@function(
    min_vram_gb=16,              # any GPU with at least 16 GB; or name one: gpu="a100_80gb"
    pip=["torch", "transformers>=4.44"],  # torch is already on the machine for Python 3.10
    timeout=600,
    idle_timeout=300,
)
def classify(texts):
    import torch
    from transformers import AutoModelForSequenceClassification, AutoTokenizer

    if not torch.cuda.is_available():
        raise RuntimeError("no GPU visible to PyTorch")

    # A machine keeps the function loaded between calls, so what is stored
    # here is still in GPU memory for the next call that lands on it.
    if "model" not in _loaded:
        _loaded["tokenizer"] = AutoTokenizer.from_pretrained(MODEL)
        _loaded["model"] = AutoModelForSequenceClassification.from_pretrained(MODEL).to("cuda").eval()
    tokenizer, model = _loaded["tokenizer"], _loaded["model"]

    batch = tokenizer(texts, padding=True, truncation=True, return_tensors="pt").to("cuda")
    with torch.no_grad():
        probs = torch.softmax(model(**batch).logits, dim=-1)
    labels = [model.config.id2label[i] for i in probs.argmax(dim=-1).tolist()]
    return {
        "device": torch.cuda.get_device_name(0),
        "predictions": [
            {"text": t, "label": label, "score": round(float(p.max()), 4)}
            for t, label, p in zip(texts, labels, probs)
        ],
    }


if __name__ == "__main__":
    out = classify.remote(["I love this GPU.", "The queue was far too long."])
    print("Ran on:", out["device"])
    for p in out["predictions"]:
        print(f"  {p['label']:8} {p['score']:.4f}  {p['text']}")

§21.4.2Use a pip dependency¶

The function needs numpy. Listing it in pip=[...] is all it takes. GPU.ai installs it on the machine before the first call and keeps it for later ones. Nothing is installed on your computer beyond gpuai-sdk itself.

from gpuai_sdk.functions import function


@function(min_vram_gb=8, pip=["numpy>=1.26"], timeout=300)
def column_stats(rows):
    import numpy as np

    data = np.asarray(rows, dtype=np.float64)
    return {
        "numpy": np.__version__,
        "mean": data.mean(axis=0).round(4).tolist(),
        "std": data.std(axis=0).round(4).tolist(),
    }


if __name__ == "__main__":
    stats = column_stats.remote([[1, 10], [2, 20], [3, 30], [4, 40]])
    print("numpy on the machine:", stats["numpy"])
    print("mean:", stats["mean"])   # [2.5, 25.0]
    print("std: ", stats["std"])    # [1.118, 11.1803]

§21.4.3Errors, timeouts, background calls, and cancel¶

What the caller sees when things do not go to plan. An exception raised inside the function reaches the caller with its type, message, and remote traceback. A call that runs past its timeout is stopped. A background call can be watched and cancelled.

import time

from gpuai_sdk.functions import RemoteCancelled, RemoteError, RemoteTimeout, function


@function(min_vram_gb=8, timeout=120)
def divide(a, b):
    return a / b


@function(min_vram_gb=8, timeout=5)
def too_slow():
    import time

    print("starting a 60 second job with a 5 second timeout")
    time.sleep(60)
    return "never returned"


@function(min_vram_gb=8, timeout=600)
def long_job(seconds):
    import time

    for i in range(seconds):
        print(f"step {i + 1} of {seconds}")
        time.sleep(1)
    return "finished"


if __name__ == "__main__":
    # 1. An error inside the function reaches the caller.
    print("divide(1, 0):")
    try:
        divide.remote(1, 0)
    except RemoteError as e:
        print("  type:   ", e.type)       # ZeroDivisionError
        print("  message:", e.message)    # division by zero
        print("  remote traceback ends with:", e.traceback.strip().splitlines()[-1])

    # 2. A call past its timeout is stopped, and stops costing run time.
    print("too_slow():")
    try:
        too_slow.remote()
    except RemoteTimeout as e:
        print("  stopped:", e.message)

    # 3. A background call: watch it, read its output, cancel it.
    print("long_job(300) in the background:")
    handle = long_job.spawn(300)
    while handle.status() == "queued":
        time.sleep(2)
    time.sleep(5)
    print("  status:", handle.status())
    print("  output so far:", handle.logs().strip().splitlines()[-1:])
    handle.cancel()
    try:
        handle.result(timeout=60)
    except RemoteCancelled:
        print("  cancelled; final status:", handle.status())
        print("  cost of the cancelled call: $%.4f" % (handle.info().get("cost_usd") or 0))

§ 21.5Settings¶

Every setting is an argument to @function(...). Change the code or any setting and the next call uploads a new version.

§ 21.6How errors reach the caller¶

If the function raises, .remote() raises RemoteError on your side, with the remote exception's type, message, and traceback. For a background call, handle.result() raises the same way.

from gpuai_sdk.functions import RemoteError, RemoteTimeout

try:
    f.remote()
except RemoteTimeout:
    ...                        # the call ran past its timeout and was stopped
except RemoteError as e:
    print(e.type, e.message)
    print(e.traceback)

Catch RemoteTimeout and RemoteCancelled before RemoteError, since both are subclasses of it. The same type, message, and traceback are shown on the run's page in the console, where a timed out run has the type Timeout and a cancelled run has the type Cancelled. RemoteError also carries the run's run_id, which you can open in the console or pass to the CLI.

A few other types describe problems around the function rather than inside it. PythonVersionMismatch, SerializationError, DeserializationError, and ResultTooLarge point at packaging and size limits. ProcessExited means the Python process running the function ended before it returned a result.

§ 21.7Billing¶

A function is billed for the time its machines run, from start to release. That includes start-up and the idle period after each call. The rate is the machine's normal hourly price. A function with no machine costs nothing.

• Where it appears. Each machine shows up on the Billing usage page as an instance named fn-<function name>-..., labelled with the function it ran.
• Per-run cost. Each run also reports its own cost_usd, which is the time it held the machine at the machine's hourly price. Read it from handle.info() or on the run's page in the console.
• Why the two differ. Run costs add up to less than the machine time you are billed, because machine time between calls belongs to no run. The function's page in the console shows both for the last 24 hours.
• Controlling it. Lower idle_timeout to pay for less idle time at the price of more cold starts. Use max_price_per_hour to cap the hourly price and max_concurrency to cap how many machines run at once.

§ 21.8How your code gets there¶

The function is pickled by value with cloudpickle and uploaded the first time you call it. There is no image to build and no deploy step. A few rules follow from that.

• Python version. The machine runs your function with the same Python minor version as your program. If that is Python 3.10, the Python of the GPU.ai base image, the image's own packages (PyTorch with CUDA) are available without listing them. If it does not, GPU.ai installs your Python version on the machine, and then every package the function imports, PyTorch included, must be in pip=[...].
• Imports. Import third-party packages inside the function, and list them in pip. The machine does not have the packages installed on your computer.
• Your own modules. Helpers from your other files are not shipped automatically. Pass them as include=[my_module].
• Size. Arguments are limited to 4 MB and results to 16 MB once pickled. Pass large data by reference, as a URL or an object-store key.

§ 21.9In the console and the CLI¶

The Functions page lists each function with its status, its machines, and its cost for the last 24 hours. Open a function to see its settings and runs, and open a run to see its timings, cost, error, and live output. You can cancel a run and delete a function there. Deleting a function releases its machines and cancels its queued runs. Creating and calling a function happens from your code.

The gpu CLI does the same from a terminal. Wherever a command takes a function, pass its name or its id.

# What are my functions doing, and what did they cost today?
gpu functions list

# One function's settings, machines, and cost
gpu functions get classify

# The last runs of one function
gpu functions runs classify

# Why did a run fail?
gpu functions run classify <run-id>

# Watch a run's output live
gpu functions logs classify <run-id> --follow

# Stop a run, or delete a function
gpu functions cancel classify <run-id>
gpu functions delete classify

§ 21.10Not supported yet¶

• Public web endpoints. A function is called from Python with your API key. It does not get a URL of its own.
• Always-on autoscaled inference. Machines start when calls arrive and are released when idle. For a hosted model behind a stable endpoint, use the Inference API.
• Multi-machine calls. One call runs on one machine.
• Languages other than Python.
• A start-up time guarantee. A cold start usually takes 2 to 4 minutes and can take up to about 8, depending on the GPU and the host. A machine that is not ready after 10 minutes is released and another is started on a different supplier; if that one is not ready either, the call fails with a LaunchError and nothing is charged. Keep a machine warm with idle_timeout when latency matters.