Qwen3-8B-FP8
Qwen3-8B-FP8 is an 8.2B-parameter open-source language model from Alibaba's Qwen team. It offers dual modes: a 'thinking' mode for complex reasoning (math, code, logic) and a 'non-thinking' mode for fast general dialogue. The model supports 100+ languages and can handle 32K tokens natively, extending to 131K with YaRN. This FP8-quantized version trades some precision for reduced memory footprint and faster inference while maintaining compatibility with standard inference frameworks.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | Qwen |
| Parameters | 8.2B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 626.5k |
| Likes | 62 |
| Last updated | 2025-07-26 |
| Source | Qwen/Qwen3-8B-FP8 |
What Qwen3-8B-FP8 is
Qwen3-8B is a causal language model with 36 layers, 32 query heads and 8 key-value heads (GQA), trained with both pretraining and post-training stages. The FP8 variant uses fine-grained block-wise quantization (block size 128) and is distributed as safetensors. Native context is 32,768 tokens; YaRN extends to 131,072. The model includes a soft-switch mechanism (/think, /no_think) for dynamic mode toggling within conversations. Requires transformers ≥4.51.0.
Run Qwen3-8B-FP8 locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="Qwen/Qwen3-8B-FP8")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
ESTIMATE: FP8 quantization reduces memory vs. bfloat16. Typical GPU VRAM for inference: 16–24 GB (single A100/L40S) for batch inference, 8–12 GB for single-turn chat. Exact requirements depend on context length, batch size, and inference framework. See Qwen blog/docs for benchmarks. Multi-GPU setups require vLLM or SGLang to avoid fine-grained FP8 transformers bug.
Not explicitly stated in card. No LoRA, QLoRA, or instruction-tuning guidance provided. Infer: base model is post-trained; further fine-tuning feasibility unknown. Recommend checking Qwen GitHub or documentation for adapter strategies.
When to avoid it — and what to weigh
- Extreme latency constraints without thinking enabled — Thinking mode generates intermediate reasoning tokens before responses, increasing total latency. Non-thinking mode is faster but sacrifices reasoning depth. Check latency requirements first.
- Greedy decoding workflows — Model card explicitly warns against greedy decoding in thinking mode, which causes performance degradation and infinite loops. Requires sampling (Temperature 0.6, TopP 0.95, etc.).
- Multi-GPU distributed inference with fine-grained FP8 on transformers — Known issue: fine-grained FP8 in transformers has problems under distributed inference. Workaround is CUDA_LAUNCH_BLOCKING=1, but vLLM or SGLang are recommended.
- Applications requiring guaranteed output length or streaming truncation safety — No explicit documentation on output length guarantees, streaming truncation behavior, or edge cases in thinking-mode token counting. Requires testing.
License & commercial use
Apache-2.0 license. Permissive OSI-approved open-source license allowing broad use, modification, and redistribution.
Apache-2.0 is a permissive license compatible with commercial use. No gating restrictions (gated=false). However, consult your legal team and Alibaba Qwen's terms for any proprietary limitations not encoded in the license file, especially regarding model output attribution and derivatives. No commercial support vendor noted in card.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Strong |
| Assessment confidence | High |
No explicit security audit, adversarial robustness, or prompt-injection mitigations stated. As with all LLMs, model outputs can be manipulated by prompt engineering and may reflect training data biases. Thinking mode adds complexity (token parsing, special tokens like <think></think>); validate thinking-content extraction logic. No mention of content filtering or jailbreak defenses. Requires your own safety evaluation.
Alternatives to consider
Qwen2.5-Instruct
Smaller/faster option if reasoning depth not needed; Qwen3 cited as superior on math/code/reasoning. If full multi-mode flexibility unnecessary, older version may reduce resource overhead.
Meta Llama 3.1 or 3.2 (8B)
Comparable 8B scale, Apache-2.0 license, strong open-source ecosystem. No built-in thinking mode; trade reasoning transparency for simplicity and broader tool support.
Deepseek-R1 or QwQ-32B (larger sibling)
If advanced reasoning is critical, larger models offer deeper chain-of-thought. Deepseek-R1 and QwQ are reasoning-focused; trade size and cost for better complex task performance.
Ship Qwen3-8B-FP8 with senior software developers
Explore self-hosted LLM deployment, custom LLM applications, and RAG integrations. Start with vLLM or SGLang for production, or test locally with Ollama. Review Qwen GitHub and documentation for benchmarks and fine-tuning.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Qwen3-8B-FP8 FAQ
Can I use Qwen3-8B-FP8 commercially without additional licensing?
What GPU VRAM do I need to run this model?
How does thinking mode affect response latency?
Is there a quantized version smaller than FP8?
Software development & web development with DEV.co
DEV.co helps companies turn open-source tools like Qwen3-8B-FP8 into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your open-source llms stack.
Ready to Deploy Qwen3-8B-FP8?
Explore self-hosted LLM deployment, custom LLM applications, and RAG integrations. Start with vLLM or SGLang for production, or test locally with Ollama. Review Qwen GitHub and documentation for benchmarks and fine-tuning.