Qwen3-235B-A22B-FP8
Qwen3-235B-A22B-FP8 is a 235-billion-parameter mixture-of-experts (MoE) language model from Alibaba's Qwen team, with only 22B parameters active per inference pass. It is quantized to FP8 for reduced memory footprint while maintaining quality. The model uniquely supports switchable thinking/reasoning mode (for complex tasks) and non-thinking mode (for fast dialogue). It supports 100+ languages and is optimized for instruction-following, agent-based tasks, and multilingual translation. Distributed under Apache 2.0 license with no access gating.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | Qwen |
| Parameters | 235.1B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 131.8k |
| Likes | 94 |
| Last updated | 2025-07-26 |
| Source | Qwen/Qwen3-235B-A22B-FP8 |
What Qwen3-235B-A22B-FP8 is
MoE architecture with 128 total experts, 8 activated per token. 94 transformer layers, GQA attention (64 Q-heads, 4 KV-heads). Native context window 32,768 tokens; expandable to 131,072 with YaRN. FP8 quantization uses fine-grained block-size-128 scheme. Requires transformers>=4.51.0. Inference supported via transformers, SGLang (>=0.4.6.post1), vLLM (>=0.8.5). Token-level thinking/non-thinking switching via chat template flag or soft-switch prompts (/think, /no_think).
Run Qwen3-235B-A22B-FP8 locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="Qwen/Qwen3-235B-A22B-FP8")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
ESTIMATE: FP8 quantization (~115–130GB VRAM for single GPU; verify on your hardware). MoE activation reduces per-token compute but does not reduce peak memory footprint. Multi-GPU inference recommended for production. Known distributed inference issues with transformers fine-grained FP8; vLLM/SGLang preferable. Requires CUDA 11.8+, modern GPU (A100, H100, or equivalent).
Card does not explicitly document LoRA, QLoRA, or full fine-tuning feasibility. MoE models typically require careful expert routing updates. Recommend consulting official Qwen GitHub/documentation before fine-tuning. FP8 quantization may complicate gradient-based tuning; full precision (BF16) checkpoint may be required for training.
When to avoid it — and what to weigh
- Latency-Critical, Single-Token Applications — Despite MoE efficiency, 235B base model incurs significant computational overhead. Non-thinking mode is faster but still requires substantial resources for sub-100ms response budgets.
- Limited GPU Memory (<80GB VRAM) — FP8 quantization reduces footprint, but full model still requires substantial VRAM. Distributed inference on transformers has known fine-grained FP8 issues; consider vLLM/SGLang instead.
- Proprietary, Closed-Ecosystem Deployment — Model is open-source, but orchestration via SGLang/vLLM adds operational complexity. Requires careful version pinning and CUDA_LAUNCH_BLOCKING workarounds for multi-device setups.
- Real-Time Streaming with Thinking Mode — Thinking mode generates variable-length internal reasoning before output, making token-streaming unpredictable. Non-thinking mode better suited for streaming APIs.
License & commercial use
Apache 2.0 license. Permissive OSI license; permits commercial use, modification, and distribution with attribution and liability notice included.
Apache 2.0 is a permissive, OSI-approved license permitting commercial deployment. No usage restrictions, licensing fees, or commercial use caveats stated. However, Alibaba/Qwen retains no indemnification for model output. Verify model output compliance with your jurisdiction's AI/LLM regulations (e.g., EU AI Act, local data residency).
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Strong |
| Assessment confidence | High |
No security audit, vulnerability disclosure, or adversarial robustness claims stated. Model is open-source; source code and weights publicly available for review. Typical LLM risks apply: prompt injection, jailbreaking, hallucinations, bias. Thinking mode generates visible internal reasoning, which may aid interpretability but also expose model reasoning to manipulation. FP8 quantization may alter inference behavior; test on target hardware. No claims of differential privacy, watermarking, or safety alignment beyond general instruction-following.
Alternatives to consider
QwQ-32B or Qwen2.5-Instruct
Smaller alternatives from Qwen series. QwQ-32B pure reasoning; Qwen2.5-Instruct pure instruction-following. Choose if memory-constrained or if thinking/non-thinking switching unnecessary.
Llama 3.1 70B / 405B
Dense transformer, no MoE. Broader ecosystem support (vLLM, TGI, llama.cpp). Smaller models (70B) lower memory; larger (405B) higher performance. No native thinking mode.
Mistral Large or Mixtral 8x22B
Smaller MoE alternatives. Mistral Large is dense and well-supported; Mixtral is MoE with 22B active. Simpler architecture, mature tooling; lower reasoning capability claim than Qwen3.
Ship Qwen3-235B-A22B-FP8 with senior software developers
Review hardware requirements, set up vLLM or SGLang for production inference, and test thinking/non-thinking modes on your workload. Contact our team for architecture review and cost estimation.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Qwen3-235B-A22B-FP8 FAQ
Can I use this model commercially?
What is the minimum GPU memory required?
What does 'thinking mode' do, and when should I use it?
Does FP8 quantization affect accuracy?
Software development & web development with DEV.co
DEV.co helps companies turn open-source tools like Qwen3-235B-A22B-FP8 into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your open-source llms stack.
Ready to Deploy Qwen3?
Review hardware requirements, set up vLLM or SGLang for production inference, and test thinking/non-thinking modes on your workload. Contact our team for architecture review and cost estimation.