Xinference

What is Xinference?

Xorbits Inference (Xinference) serves language, embedding, rerank, speech, image and multimodal models from a built-in catalog or your own model files. It picks an engine per model (Transformers, vLLM, SGLang, llama.cpp, MLX), exposes an OpenAI-compatible REST API plus a web UI, and can spread models across several workers. Dify, FastGPT, RAGFlow and MaxKB can use it as a model provider.

Where it fits in a 10–200 person business

• One internal endpoint for the chat, embedding and rerank models behind a RAG or knowledge-base app
• Serve speech-to-text, text-to-speech or image models next to LLMs on the same GPU server
• Give teams per-user API keys and an audit trail for self-hosted model access

These use cases are our editorial suggestions, not claims made by the project.

Can a business use Xinference? (license)

This is a plain-English summary of the license text, not legal advice. Third-party models you run with it have their own licenses.

Data & compliance: does Xinference phone home?

Checked against the project's own docs and repository on Oct 26, 2026. "Not documented" means we could not find the answer in official sources; it does not mean the feature is missing or that no data is sent. Third-party services you connect (model APIs, vector databases, search engines) have their own data policies. Compare all apps.

What hardware does Xinference need?

The project does not publish a minimum CPU/RAM figure, so we leave the official value blank rather than guess.

GPU: The default Docker image requires an NVIDIA GPU (host CUDA 13.0 or newer, driver 580 or newer, NVIDIA Container Toolkit). For CPU, use the image tag with a -cpu suffix; pip installs on Apple Silicon can use the MLX engine.

No server CPU, RAM or disk figures are published; requirements depend on the models and engines you launch, and each engine keeps its own OS, hardware and driver requirements.

Source: inference.readthedocs.io/en/latest/getting_started/using_docker_image.html

How much does it cost to host Xinference per month?

How to deploy Xinference

Method: Docker

# pick a release tag from https://github.com/xorbitsai/inference/releases
docker run -v </on/your/host>:</on/the/container> -e XINFERENCE_HOME=</on/the/container> \
  -p 9998:9997 --gpus all xprobe/xinference:v<your_version> xinference-local -H 0.0.0.0

From the official instructions: inference.readthedocs.io/en/latest/getting_started/using_docker_image.html. Always check the source for the current version before you run it.

Known gotchas when self-hosting Xinference

• Since v3.0 authentication is on by default and no admin exists until you create one with the setup token printed in the server startup log. Recreating the container without its auth database needs the new instance's token. source
• Since v3.0 the GPU image no longer bundles vLLM or SGLang; engines are installed from PyPI into a per-model virtual environment at first launch. Offline or air-gapped hosts need the private PyPI mirror from the Docker Compose offline profile. source
• The image holds no models and downloads them into the container. Mount a host directory and set XINFERENCE_HOME, or the downloads are lost when the container is recreated. source
• The GPU image cannot start without --gpus, and -H 0.0.0.0 is required or the port is not reachable from the host. Multi-GPU setups need a larger --shm-size. source

Alternatives in Local model runtimes

• Ollama The simplest way to download and run open-weight LLMs locally behind an API. Compare Ollama vs Xinference
• llama.cpp Lightweight C/C++ engine that runs quantized open models on CPU or GPU, with an OpenAI-compatible server. Compare llama.cpp vs Xinference
• vLLM High-throughput LLM inference server for GPUs with an OpenAI-compatible API. Compare vLLM vs Xinference
• LocalAI OpenAI-compatible API for LLMs, vision, voice and images on any hardware. Compare LocalAI vs Xinference
• GPUStack GPU cluster manager that deploys vLLM, SGLang and other engines behind an OpenAI-compatible API. Compare GPUStack vs Xinference

Sources

• GitHub repository: github.com/xorbitsai/inference
• Official documentation, Hardware requirements, Deploy instructions, Gotcha 2, Gotcha 3, Gotcha 4: inference.readthedocs.io/en/latest/getting_started/using_docker_image.html
• License text: github.com/xorbitsai/inference/blob/main/LICENSE
• Gotcha 1: inference.readthedocs.io/en/latest/user_guide/auth_system.html
• Hetzner Cloud pricing (cost estimate): www.hetzner.com/cloud/regular-performance
• GitHub stars, commits and releases: GitHub REST API, fetched Oct 26, 2026