Run LLMs offline on Windows: llama.cpp and Qwen3.6-27B on your GPU

You can run a genuinely capable language model on your own Windows PC, fully offline, with no account, no subscription and no data leaving your machine. This guide walks through the whole thing using llama.cpp and Qwen3.6-27B, a strong open model from April 2026, on a normal gaming GPU. No prior knowledge assumed.

Why run a model offline at all

Three practical reasons:

• Privacy. Your prompts and documents never leave the machine. You can literally pull the network cable and keep chatting.
• Cost. After the download, every answer is free. No tokens billed, no rate limits.
• Control. The model stays exactly as it is. No silent updates changing its behavior overnight.

The tradeoff: you need a decent PC, and the very largest cloud models are still ahead. But for writing, summarizing, coding help and translation, a local 27B model is more than good enough.

What a language model actually is

Strip away the hype and a model is a huge pile of numbers called weights. Qwen3.6-27B has 27 billion of them (that is what "27B" means). When you prompt the model, software multiplies your text through those numbers and produces an answer, token by token.

Two consequences follow directly:

• Bigger pile of numbers, bigger file. Stored at full 16-bit precision, 27 billion weights take up about 54 GB.
• The file has to fit in fast memory. For good speed, the weights need to sit in your graphics card's memory (VRAM), not on your disk.

54 GB does not fit on any consumer graphics card. That is where the next two concepts come in.

GGUF: the whole model in one file

GGUF is the file format of llama.cpp, the most widely used open source engine for running models locally. A GGUF file bundles everything into a single download: the weights, plus metadata that tells the engine how to run them (architecture, tokenizer, chat template).

For you this means: no installation of model code, no Python environment, no config hunting. You download one file, point llama.cpp at it, done.

GGUF files for popular models live on Hugging Face, the public model registry. For Qwen3.6-27B, the community repositories from unsloth and bartowski provide them within days of release.

Quantization, or why 54 GB becomes 16

Quantization is lossy compression for weights. Instead of storing every number at 16-bit precision, you store it at 8, 6, 4 or even 3 bits. The file shrinks dramatically and the model gets slightly, measurably dumber, but far less than you would expect.

GGUF filenames encode the compression level. You will see names like:

• Q8_0 (8 bit): about 28 GB for this model. Nearly indistinguishable from the original, but needs a very large card.
• Q4_K_M (roughly 4.8 bit): about 17 GB. The community sweet spot: small enough for common cards, quality loss barely noticeable in daily use.
• Q3_K_M (roughly 3.9 bit): about 13 GB. For smaller cards, with a visible but acceptable quality drop.

Rule of thumb for beginners: take Q4_K_M. If it does not fit, go one step lower. If you have VRAM to spare, go one step higher. The difference between "runs at all" and "runs perfectly" is smaller than the difference between Q4 and Q8.

What you need

• Windows 10 or 11, 64-bit.
• An NVIDIA graphics card with current drivers. Check your VRAM in Task Manager under Performance, GPU, "Dedicated GPU memory".
  • 24 GB (RTX 3090, 4090, 5090): Q4_K_M fits comfortably, even with a long conversation history.
  • 16 GB (RTX 4070 Ti Super, 4080 Laptop): Q4_K_M works with a moderate context size.
  • 12 GB (RTX 4070): use Q3_K_M, or expect partial offload.
• About 20 GB of free disk space.
• AMD Radeon instead of NVIDIA? Supported as well: llama.cpp ships prebuilt Vulkan and HIP builds for Windows, see step 1. No dedicated GPU at all also works via the CPU build, just much slower.

Not sure what your PC has? Let your browser check:

Step 1: Install llama.cpp

llama.cpp needs no installer. Download the prebuilt Windows release:

1. Open the llama.cpp releases page on GitHub.
2. Download the newest asset matching your graphics card:
   • NVIDIA: llama-<version>-bin-win-cuda-x64.zip
   • AMD Radeon: llama-<version>-bin-win-vulkan-x64.zip (runs on any GPU with Vulkan support, the easiest path) or llama-<version>-bin-win-hip-radeon-x64.zip (AMD's ROCm stack, usually the fastest option on recent cards like the RX 7000 and 9000 series; older cards may need a build from source)
   • No dedicated GPU: llama-<version>-bin-win-cpu-x64.zip
3. Extract it to a simple path, for example C:\llama.cpp.

That is the whole installation. Open a terminal in that folder (right-click in Explorer, "Open in Terminal") and check that it runs:

llama-server.exe --version

The start command in step 3 is identical for all three builds: --n-gpu-layers 99 offloads the model to whatever GPU the build supports. With the CPU build, simply leave that flag out.

Step 2: Download the model

1. Go to huggingface.co/unsloth/Qwen3.6-27B-GGUF and open the "Files" tab.
2. Download the file ending in Q4_K_M.gguf (about 17 GB, so this takes a while).
3. Put it somewhere memorable, for example C:\models\.

No Hugging Face account is needed for public repositories.

Step 3: Start the server

One command:

llama-server.exe -m "C:\models\Qwen3.6-27B-Q4_K_M.gguf" --n-gpu-layers 99 -c 32768

What the flags mean:

• -m is the path to your GGUF file.
• --n-gpu-layers 99 pushes all of the model onto the GPU. This is the single most important switch for speed.
• -c 32768 sets the context size: how much conversation and pasted text the model can hold at once. 32K is a good start; larger values cost more VRAM.

Wait a few seconds for the weights to load, then open http://localhost:8080 in your browser. You get a chat interface, running entirely on your machine. Ask it something. The first answer of a session is slower; after that it should stream at reading speed.

When it does not fit

If loading fails or crawls, the model does not fit your VRAM. Three knobs, in order:

1. Smaller quantization. Q4_K_M to Q3_K_M saves about 4 GB.
2. Shorter context. -c 8192 instead of 32768 frees noticeable memory.
3. Partial offload. Reduce --n-gpu-layers (for example to 40) so the overflow runs on the CPU. Slower, but it always works.

Where to go from here

The server you just started also speaks the OpenAI API at http://localhost:8080/v1. Most tools that let you configure a custom endpoint (code assistants, note-taking plugins, your own scripts) can use your local model as a drop-in replacement for a paid API.

And if you want to compare: the same three steps work for every other GGUF model on Hugging Face. Download a different file, change the -m path, and you are running a different brain. The model is a file. You own it now.