vLLM

Use a vLLM OpenAI-compatible server with NeoStack AI.

vLLM is a good choice when you have a GPU workstation or server and want higher-throughput local or LAN inference.

Start vLLM

Run vLLM with the model you want to serve:

vllm serve Qwen/Qwen2.5-Coder-7B-Instruct

By default, vLLM serves an OpenAI-compatible API on:

http://localhost:8000/v1

Add It To NeoStack AI

1. Open Settings > Chat & Agents > Chat Providers.
2. Click + Add Custom Provider.
3. Name it vLLM.
4. Set the base URL:

http://localhost:8000/v1

5. Turn off Require an API key unless your server requires one.
6. Leave Auto-discover models from /models endpoint enabled.
7. Start a chat with Local & BYOK Chat and pick the vLLM model.

Notes

• vLLM implements OpenAI-compatible Chat Completions.
• Remote vLLM servers should be protected with auth or a private network.
• Tool-call quality depends on the served model.