Alibaba Ships Qwen3-8B as Ready-to-Run Local AI With Dual Reasoning Modes
Qwen3-8B released as GGUF quantizations (Q4_K_M through Q8_0) for local inference on Hugging Face . Hybrid thinking: single checkpoint switches between reasoning and fast chat via /think and /no_think. 8.2B dense params, 36 layers, GQA 32/8, 32K native context extendable to 131K with YaRN.
Takeaways
- Qwen3-8B released as GGUF quantizations (Q4_K_M through Q8_0) for local inference on Hugging Face .
- Hybrid thinking: single checkpoint switches between reasoning and fast chat via /think and /no_think.
- 8.2B dense params, 36 layers, GQA 32/8, 32K native context extendable to 131K with YaRN.
- Outperforms QwQ and Qwen2.5-Instruct on math, code, and commonsense reasoning per the technical report .
- Runs out of the box on llama.cpp, Ollama, LM Studio, vLLM; Apache 2.0 license allows commercial use.
- Avoid greedy decoding; use presence_penalty 1.5 on quantized builds to stop repetition.
Alibaba’s Qwen team has published official quantized builds of Qwen3-8B on the model page . The repository provides Q4_K_M and Q8_0 GGUF files for llama.cpp, recent Ollama releases, LM Studio, and compatible runtimes. Developers can run the same checkpoint in an extended reasoning mode or a faster direct-answer mode without converting the original weights.
GGUF removes a conversion step
GGUF packages model weights and runtime metadata in a format designed for local inference. Quantization stores weights at reduced precision, lowering storage and memory requirements while introducing a potential quality tradeoff.
| Build | Approximate size | Practical tradeoff |
|---|---|---|
| Q4_K_M | 5.03 GB | Lower memory use and faster loading; suitable as a local default |
| Q8_0 | 8.71 GB | Higher weight precision with greater RAM and VRAM requirements |
Total runtime memory exceeds the file size because inference also allocates a key-value cache, compute buffers, and prompt state. With a default 16-bit KV cache, a 131,072-token window for this architecture can consume roughly 18 GiB for the cache alone, before weights and other buffers. KV-cache quantization and partial CPU offload can reduce VRAM pressure, with corresponding speed or quality costs.
Reasoning becomes a request-level switch
Qwen3-8B supports thinking and non-thinking behavior within one checkpoint. Thinking mode emits intermediate reasoning inside <think> blocks before the final response, while non-thinking mode answers directly with lower token use and latency.
| Mode | Behavior | Control |
|---|---|---|
| Thinking | Generates an extended reasoning trace before the answer | enable_thinking=true or /think |
| Non-thinking | Returns a direct answer | enable_thinking=false |
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
