Self-Hosted LLM Inference Endpoint

Overview

Four idempotent Ansible playbooks that take a bare Ubuntu GPU server to a hardened, TLS-terminated, authenticated LLM inference endpoint: vLLM in Docker behind an Nginx reverse proxy with per-user Bearer tokens.

Self-hosting an open-weights model is straightforward until it has to stay up. The parts that actually cost time aren’t docker run — they’re exposing the model safely, rotating access without downtime, surviving NVIDIA driver upgrades, and noticing when any of it breaks. This repository is the accumulated answer to those four problems.

All values in the repo are sanitized placeholders — example.com, RFC1918 addresses, a dummy model revision — but the playbook logic, task ordering, conditionals, and handler wiring are intact.

Architecture

Nginx is the only authentication boundary. vLLM runs without --api-key, bound to 127.0.0.1, so the model server is never directly reachable — UFW denies inbound by default, and only 80, 443, and a non-standard SSH port are open. A Lua block in Nginx checks the Bearer token against a key store at /etc/vllm/api_key (mode 0640, root:www-data), reading the file on every request.

Provisioning splits across four plays: baseline hardening (admin user, UFW, SSH on a non-standard port validated with sshd -t before restart, Fail2Ban, unattended-upgrades, Docker, Nginx); GPU enablement (driver, container toolkit, CDI spec plus a boot-time refresh unit, compose file, systemd unit, health gate); the public surface (Certbot, Nginx Lua module, key store, reverse proxy with TLS and Bearer auth, nginx -t before applying); and a one-shot teardown of a previous Ollama stack.

Key decisions & tradeoffs

Auth at the proxy, not the model server. vLLM’s --api-key is a process argument — changing it means restarting the container and reloading ~23 GB of weights into VRAM. Moving the boundary to Nginx makes key rotation an nginx -s reload: instant, with the model untouched. That single decision is what makes add/revoke viable as an operational tool rather than a maintenance window.

systemd owns the container, Docker does not restart it. The compose file sets restart: "no"; the unit carries Restart=on-failure with StartLimitBurst=5. A Docker-level restart policy would resurrect a crash-looping container underneath a unit still reporting active. With the limit, repeated failure lands in failed state where monitoring can see it, rather than looping silently for a day.

/healthz is deliberately unauthenticated. Every authenticated path returns 401 before reaching vLLM, so an uptime monitor pointed at /v1/models gets the same 401 whether the model is healthy or dead — the outage is invisible. /healthz proxies only vLLM’s own /health, which returns an empty 200 and exposes no model data, no inference capability, and no key material.

The driver install refuses to run over a broken driver. A failing nvidia-smi usually means a version mismatch, not a missing driver, and installing a pinned package in that state can downgrade the host. The play gates on package state rather than nvidia-smi, and fails loudly with the recovery commands instead of guessing.

The CDI spec is regenerated at boot. It pins driver libraries by exact versioned filename and lives on tmpfs, so a dedicated systemd unit rebuilds it every boot — and re-running the GPU playbook repairs a stale spec mid-incident, which makes the automation double as the remediation.

Offline by default. The model cache is mounted read-only with HF_HUB_OFFLINE=1. No Hugging Face token ever enters the container, and a deploy can’t be broken by an upstream outage. The play asserts the cached snapshot exists rather than silently downloading.

Results

• Key rotation with zero downtime — add or revoke a user in one command, with no model reload and no dropped requests.
• The model server is unreachable from outside the host even if a firewall rule were removed, because it publishes to loopback rather than 0.0.0.0.
• No secrets in the repo: keys are generated on-host with openssl rand -hex 32, the task that writes them is no_log: true, and only the example inventory is tracked.
• A class of GPU failure that presents as an application fault is handled automatically — the deployment playbook is also the incident fix.
• Every playbook is safe to re-run, with sshd -t, nginx -t, and visudo -cf validating each risky change before it is applied.