how it will work
How it fits into GPU.ai
vLLM One-Click Endpoints is a planned way to serve an open model behind an OpenAI-compatible API with one click. It will run vLLM, the open source inference engine, on top of Serverless Functions, so you get an endpoint without setting up a machine or a server.
- Your appany OpenAI client
- vLLM endpointstarted in one click
- Serverless Functionsa GPU on GPU.ai
Pick a model, get an endpoint
Choose a model and a GPU type in the console, and GPU.ai will start vLLM and give you the endpoint's address.
OpenAI-compatible
Existing OpenAI client code is meant to work by changing the base URL.
Built on Serverless Functions
Endpoints will run as functions, with the same warm machines, idle timeout, and machine time billing.
vLLM is an open source project. GPU.ai is not affiliated with or endorsed by its maintainers.