vLLM Inference

High-Performance, Local AI with Dynamic Adapter Switching

vLLM is what makes fast, local AI possible in the / foxlight ecosystem — especially inside Obscura. We train small, specialized models in the cloud, then run them on your own device with vLLM, so you get speed and full control without sending your data anywhere.

How a Model Gets to Your Device

• We train lightweight LoRA adapters (small, focused add-ons to a base model) on remote GPU clusters using our Thinking Machines Tinker training pipeline.
• These adapters stay small — typically 20MB to 150MB, even for a 4-billion-parameter model.
• Once training's done, the adapter downloads to your device and converts into a standard PEFT format vLLM can load directly.
• vLLM then runs the base model (a few gigabytes) alongside the adapter, all on your own hardware.

Swap Models Without Restarting

• vLLM can hot-swap adapters while it's running, through its /v1/load_lora_adapter endpoint — no restart, no downtime.
• That means your AI can keep getting better in the background. New training gets downloaded and swapped in live, and you'd never notice a hiccup.

Fast and Fully Offline

• Running locally with vLLM is quicker than round-tripping to a cloud API for most everyday tasks.
• Once the model and adapter are downloaded, you don't need an internet connection to use it — full control, zero cloud dependency.

The Technical Details

• vLLM runs as a background process on your device.
• It talks over a standard, OpenAI-compatible API (typically http://localhost:8000/v1/completions), so it plugs into familiar tooling.
• Our Tauri (Rust) backend starts it, stops it, and checks on its health automatically — you never touch it directly.
• Storage: Base models (a few GB each) and adapters (100–200MB each) live locally in the app's own folder.
• Hardware: Running a 4B model with LoRA typically needs about 6–8GB of VRAM.

How It All Fits Together

• The Rust backend manages the vLLM process, handles caching adapters on disk, and talks to the local vLLM server over HTTP.
• A small Python runtime handles the Thinking Machines Tinker side — mainly downloading and converting adapters after they finish training in the cloud.
• Put simply: train in the cloud, download the adapter, start vLLM with --enable-lora and --lora-modules, and you're running your own AI, locally.