llama.cpp

What is llama.cpp?

llama.cpp is the inference engine underneath many local-AI tools. Its llama-server runs a quantized (GGUF) model and exposes OpenAI-compatible chat, responses and embeddings endpoints plus a built-in web UI, with parallel requests for several users. It runs on plain x86 or ARM CPUs and accelerates on NVIDIA (CUDA), AMD (ROCm/HIP), Intel (SYCL), Vulkan and Apple Metal.

Where it fits in a 10–200 person business

• Serve one open model to the whole office from a small server, with no Python stack
• Local model back-end for chat UIs and RAG tools that speak the OpenAI API
• Run quantized models on CPU-only or older hardware

These use cases are our editorial suggestions, not claims made by the project.

Can a business use llama.cpp? (license)

This is a plain-English summary of the license text, not legal advice. Third-party models you run with it have their own licenses.

Data & compliance: does llama.cpp phone home?

Checked against the project's own docs and repository on Oct 22, 2026. "Not documented" means we could not find the answer in official sources; it does not mean the feature is missing or that no data is sent. Third-party services you connect (model APIs, vector databases, search engines) have their own data policies. Compare all apps.

What hardware does llama.cpp need?

The project does not publish a minimum CPU/RAM figure, so we leave the official value blank rather than guess.

Our measurement measured: 437 MiB RAM at idle, 489 MiB while working (llama.cpp build 11804 (c7fc50941), Oct 22, 2026). Full test results.

GPU: Optional. Runs on CPU (AVX/AVX2/AVX512/AMX, ARM NEON) and supports CPU+GPU hybrid inference for models larger than VRAM. Docker images exist for CUDA, ROCm, SYCL, Vulkan, MUSA and OpenVINO.

No official RAM figure; memory use depends on the model size and quantization you load.

Source: github.com/ggml-org/llama.cpp#description

How much does it cost to host llama.cpp per month?

How to deploy llama.cpp

Method: Docker (or install script / prebuilt binaries)

docker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server \
  -m /models/your-model.gguf -c 512 --host 0.0.0.0 --port 8080
# NVIDIA: use ghcr.io/ggml-org/llama.cpp:server-cuda with --gpus all --n-gpu-layers 99

From the official instructions: github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#docker. Always check the source for the current version before you run it.

Known gotchas when self-hosting llama.cpp

• llama-server has no API key by default. For anything beyond one machine, set --api-key and put it behind a reverse proxy; by default it reflects any Origin header back with credentials allowed. source
• The GPU Docker images are built by CI but not tested beyond building. If you need a different CUDA, ROCm or MUSA version you must build the image yourself. source
• The experimental --tools / --agent flags expose file read/write and shell tools over the API; do not enable them in untrusted environments. source

Alternatives in Local model runtimes

• Ollama The simplest way to download and run open-weight LLMs locally behind an API. Compare llama.cpp vs Ollama
• vLLM High-throughput LLM inference server for GPUs with an OpenAI-compatible API. Compare llama.cpp vs vLLM
• LocalAI OpenAI-compatible API for LLMs, vision, voice and images on any hardware. Compare llama.cpp vs LocalAI

Sources

• GitHub repository: github.com/ggml-org/llama.cpp
• Official documentation, Gotcha 2: github.com/ggml-org/llama.cpp/blob/master/docs/docker.md
• License text: github.com/ggml-org/llama.cpp/blob/master/LICENSE
• Hardware requirements: github.com/ggml-org/llama.cpp#description
• Deploy instructions: github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#docker
• Gotcha 1, Gotcha 3: github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• Hetzner Cloud pricing (cost estimate): www.hetzner.com/cloud/regular-performance
• GitHub stars, commits and releases: GitHub REST API, fetched Oct 22, 2026