Qwen3-235B-A22B-Instruct-2507-AWQ
Qwen3-235B-A22B-Instruct-2507-AWQ is a 235-billion-parameter mixture-of-experts (MoE) language model quantized to 4-bit AWQ format by QuantTrio. It claims improvements in reasoning, knowledge coverage, and 256K context length. The 116GB quantized model runs on 8 GPUs with vLLM and supports non-thinking mode only. Licensed under Apache 2.0, ungated, with 35K downloads and modest engagement (12 likes).
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | QuantTrio |
| Parameters | 235.1B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 35.2k |
| Likes | 12 |
| Last updated | 2025-08-19 |
| Source | QuantTrio/Qwen3-235B-A22B-Instruct-2507-AWQ |
What Qwen3-235B-A22B-Instruct-2507-AWQ is
AWQ-quantized derivative of Qwen/Qwen3-235B-A22B-Instruct-2507 base model. 235B total parameters with 22B activated (128 experts, 8 active per token). 94 layers, 64 Q-heads + 4 KV-heads (GQA). Native 262K context; card notes vLLM deployments at 32K context to avoid OOM. Last update 2025-08-19 (bug fix for vLLM 0.10.1 compatibility). Requires transformers>=4.51.0 and vLLM>=0.9.2.
Run Qwen3-235B-A22B-Instruct-2507-AWQ locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="QuantTrio/Qwen3-235B-A22B-Instruct-2507-AWQ")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
Quantized model weight: ~116 GB. Estimated total VRAM for 8-GPU deployment (tf32): 150–200 GB aggregate (18.75–25 GB per GPU). Context length 32K uses less than 262K native; vLLM card advises reducing context if OOM. Single-node 8× A100 40GB or H100 80GB recommended; smaller GPUs or multi-node scaling not documented.
No explicit mention of LoRA, QLoRA, or fine-tuning support in the card. Base model (Qwen/Qwen3-235B-A22B-Instruct-2507) may support fine-tuning, but quantized AWQ variant stability for parameter-efficient methods is Unknown. Recommend testing on base model before quantized variant.
When to avoid it — and what to weigh
- Latency-critical or single-GPU deployments — Model requires minimum 4 GPUs (8 recommended for expert parallelism). No evidence of single-GPU or CPU inference support. Unsuitable for edge deployment or real-time, low-latency scenarios.
- Thinking/reasoning-intensive tasks requiring explicit chain-of-thought — This variant explicitly does not generate <think></think> blocks. If step-by-step reasoning output is required, the non-quantized base or alternative thinking-enabled models are necessary.
- Commercial closed-source systems requiring proprietary model licensing — Apache 2.0 allows commercial use, but model is quantized by a third party (QuantTrio). Verify indemnification and support terms with QuantTrio; no clarity on Alibaba/Qwen official support for quantized derivatives.
- Production environments without vLLM/SGLang infrastructure — Model requires actively maintained serving frameworks. llama.cpp, Ollama, and local tools may have incomplete support for MoE routing. Deployment outside vLLM/SGLang/TGI increases operational risk.
License & commercial use
Licensed under Apache License 2.0 (OSI-approved permissive license). Grants commercial use rights, modification, and redistribution with attribution and warranty disclaimer. No restrictions on commercial applications stated in license itself.
Apache 2.0 permits commercial use. However, this is a third-party quantized derivative published by QuantTrio, not the official Qwen/Alibaba release. Before production deployment: (1) verify QuantTrio's license compliance and derivative rights; (2) confirm Alibaba has no restrictions on quantized variants; (3) clarify indemnification and support SLAs with QuantTrio. Requires legal review for high-value or liability-sensitive applications.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Adequate |
| License clarity | Needs review |
| Deployment complexity | High |
| DEV.co fit | Good |
| Assessment confidence | High |
No explicit security audit, adversarial robustness testing, or hardening details provided. Standard LLM considerations apply: model may generate hallucinations or toxic content; input validation and output filtering required in production. Third-party quantization introduces supply-chain consideration—verify QuantTrio repository integrity. No vulnerability disclosure process documented.
Alternatives to consider
Qwen/Qwen3-235B-A22B-Instruct-2507 (base, non-quantized)
Official Alibaba release; native thinking mode support; higher baseline performance and flexibility for fine-tuning. Trade-off: higher VRAM (270+ GB estimated).
Deepseek-V3 (671B or quantized variants)
Closed-weight but higher reasoning/knowledge benchmarks (MMLU-Pro 81.2 vs 83.0, AIME25 46.6 vs 70.3 in Qwen3's favor, mixed performance). Larger model; may offer different deployment/licensing terms.
Mistral Large (200B+) or other open MoE models
Alternative MoE architecture with simpler deployment (4-GPU viable). Smaller parameter count may reduce VRAM but typically lower reasoning/knowledge performance than Qwen3-235B.
Ship Qwen3-235B-A22B-Instruct-2507-AWQ with senior software developers
Qwen3-235B-A22B-Instruct-2507-AWQ offers strong reasoning and knowledge capabilities at quantized efficiency. Verify QuantTrio licensing and support terms, then contact our team to architect a production vLLM cluster tailored to your inference and fine-tuning needs.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Qwen3-235B-A22B-Instruct-2507-AWQ FAQ
Can I use this model in a commercial product?
What GPU configuration do I need?
Why does the card mention --enable-expert-parallel for 8 GPUs but not 4?
Does this support thinking/reasoning output like other Qwen3 variants?
Software developers & web developers for hire
Need help beyond evaluating Qwen3-235B-A22B-Instruct-2507-AWQ? DEV.co is a software development agency offering software development services and web development for teams of every size. Our software developers and web developers build custom software, web applications, APIs, and open-source llms integrations — and maintain them long-term.
Ready to Deploy Large-Scale Reasoning Models?
Qwen3-235B-A22B-Instruct-2507-AWQ offers strong reasoning and knowledge capabilities at quantized efficiency. Verify QuantTrio licensing and support terms, then contact our team to architect a production vLLM cluster tailored to your inference and fine-tuning needs.