Qwen3-Coder-480B-A35B-Instruct-FP8
Qwen3-Coder-480B-A35B-Instruct-FP8 is a 480-billion-parameter mixture-of-experts (MoE) code generation model from Alibaba's Qwen team, with 35B parameters active per inference. It targets agentic coding tasks, repository-scale understanding via 256K native context (extendable to 1M), and tool-calling workflows. The FP8 quantization variant reduces memory footprint while maintaining inference compatibility with transformers, vLLM, and sglang. Licensed under Apache 2.0, ungated, and actively maintained as of August 2025.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | Qwen |
| Parameters | 480.2B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 529.6k |
| Likes | 159 |
| Last updated | 2025-08-21 |
| Source | Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 |
What Qwen3-Coder-480B-A35B-Instruct-FP8 is
Qwen3-Coder-480B-A35B uses a MoE architecture (160 experts, 8 activated per token) with 62 transformer layers, GQA attention (96 Q-heads, 8 KV-heads), and 262,144-token native context. The FP8-quantized checkpoint employs fine-grained block-wise quantization (block size 128). Best practices include temperature=0.7, top_p=0.8, repetition_penalty=1.05, and max_new_tokens≤65,536. Known limitation: distributed inference in transformers requires CUDA_LAUNCH_BLOCKING=1. No thinking-mode output. Compatible with OpenAI-API-compliant endpoints.
Run Qwen3-Coder-480B-A35B-Instruct-FP8 locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
ESTIMATE: FP8 quantization (~240 GB weights + KV cache) typically requires 2–4× H100 80GB or A100 80GB GPUs for batch=1 at 256K context. Bfloat16 equivalent (480 GB weights) requires 6–8 such GPUs. Context reduction to 32K can reduce VRAM by ~50%. Exact numbers depend on batch size, sequence length, and inference framework optimizations. Verify with your target infrastructure.
Not explicitly stated in model card. LoRA/QLoRA feasibility on FP8 weights is Unknown; recommend checking community issues and Qwen documentation. Fine-tuning infrastructure typically requires same multi-GPU setup as inference. No instruction-tuning data or adapter configs provided.
When to avoid it — and what to weigh
- Real-time, Ultra-low-latency Endpoints — 480B parameters (even with 35B active) incur significant first-token and per-token latency. Avoid latency-critical SLAs under ~500ms unless aggressively quantized or batched with large hardware.
- Single-GPU or Consumer Hardware Deployments — Model card explicitly warns of OOM; FP8 variant still requires multi-GPU or high-VRAM systems (A100 80GB, H100, or equivalent). Not suitable for edge devices or laptop inference without severe context truncation.
- Non-Code-Centric Use Cases — This is a specialized code model; performance on general-purpose NLP, reasoning, or non-English tasks is not documented. Use a general-purpose LLM (Qwen3-72B, Llama 3.1) for diverse domains.
- Distributed Inference at Scale (transformers library only) — Known bug in transformers' fine-grained FP8 handling for multi-device inference; requires manual environment-variable workaround. Consider vLLM or sglang for production distributed setups.
License & commercial use
Apache License 2.0 (OSI-approved permissive license). Permits commercial use, modification, and distribution with attribution and liability disclaimer.
Apache 2.0 is a standard OSI-permissive license that explicitly permits commercial use. No gating or restrictions stated. You may deploy in production, behind APIs, or in commercial products provided you include a copy of the license and retain copyright notices. Verify with legal counsel for your jurisdiction, but license terms are clear and business-friendly.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | High |
| DEV.co fit | Strong |
| Assessment confidence | High |
Standard LLM considerations apply: model generates code programmatically—review outputs for unintended side effects (e.g., shell commands, file operations). No explicit security audit or threat model documented. FP8 quantization introduces quantization-specific inference paths; verify with framework maintainers (vLLM/transformers) for any known numerical stability issues. Use in isolated environments if processing untrusted prompts.
Alternatives to consider
Claude 3.5 Sonnet (proprietary API)
Claimed comparable performance on agentic coding but requires API dependency and per-token cost. Use if you prioritize simplicity over IP control and self-hosting.
Qwen3-72B-Coder or smaller Qwen3-Coder variants
Significantly lower VRAM and latency if repository-scale context and 480B performance are not critical. Trade-off between capability and resource efficiency.
Llama 3.1 405B (Meta, gated)
Larger general-purpose model with broader domain coverage, but gated model access and no explicit agentic-coding optimization. Consider for multi-domain setups; avoid for code-only deployments.
Ship Qwen3-Coder-480B-A35B-Instruct-FP8 with senior software developers
Start with a small context (32K) on 2× H100 GPUs to validate performance and integration. Use FP8 quantization to reduce VRAM. Monitor GitHub for updates and check vLLM/sglang compatibility before scaling to production.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Qwen3-Coder-480B-A35B-Instruct-FP8 FAQ
Can I use this model commercially in a SaaS product?
What GPU hardware do I need to run this model?
What is the difference between the FP8 and non-FP8 variants?
Does this model support fine-tuning or LoRA?
Custom software development services
Need help beyond evaluating Qwen3-Coder-480B-A35B-Instruct-FP8? DEV.co is a software development agency offering software development services and web development for teams of every size. Our software developers and web developers build custom software, web applications, APIs, and open-source llms integrations — and maintain them long-term.
Ready to Deploy Qwen3-Coder in Your Infrastructure?
Start with a small context (32K) on 2× H100 GPUs to validate performance and integration. Use FP8 quantization to reduce VRAM. Monitor GitHub for updates and check vLLM/sglang compatibility before scaling to production.