Qwen3.6-27B-Text-NVFP4-MTP
Qwen3.6-27B-Text-NVFP4-MTP is a quantized text-only variant of Alibaba's Qwen3.6-27B base model. It uses NVIDIA's NVFP4 quantization format (modelopt native) and restores a Multi-Token Prediction (MTP) head in bf16 to enable speculative decoding. Designed for vLLM on NVIDIA Blackwell (SM120) hardware, it achieves ~1.7× throughput improvement over the baseline compressed-tensors variant on concurrent long-form decodes. The model is 27.78B parameters, supports 256K context length (trained max), and handles text generation in 12+ languages.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | sakamakismile |
| Parameters | 16.7B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 339.8k |
| Likes | 80 |
| Last updated | 2026-04-29 |
| Source | sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP |
What Qwen3.6-27B-Text-NVFP4-MTP is
NVFP4-quantized 27.78B-parameter causal language model based on Qwen3.6-27B. Quantization uses nvidia-modelopt 0.43.0 with NVFP4_DEFAULT_CFG (calibrated on 20 samples from neuralmagic/calibration, max_seq_len 8192). Vision tower stripped; MTP head (1 layer, ~850 MB bf16) and linear-attention convolutions (48/64 layers) kept in bf16; lm_head kept in bf16. Model footprint ~14 GB after quantization + bf16 components. Supports speculative decoding with num_speculative_tokens=3 via single MTP layer applied recursively (mean acceptance length ~3.0–4.0 at n=3). Verified on RTX PRO 6000 Blackwell with vLLM 0.19.1rc1.
Run Qwen3.6-27B-Text-NVFP4-MTP locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
ESTIMATE: ~14–15 GB VRAM (NVFP4-quantized model ~13 GB + bf16 MTP/conv1d/lm_head ~850 MB + vLLM overhead). Tested on NVIDIA RTX PRO 6000 Blackwell (SM120). Also expected to work on RTX 5090 and other Blackwell cards with ≥16 GB VRAM. Requires `--gpu-memory-utilization 0.85–0.9` to fit 256K context with KV cache in FP8. Older architectures (A100, H100, RTX 4090) are not verified; vLLM's modelopt native path is Blackwell-specific.
No guidance provided in the model card. Quantized models (NVFP4) are typically not suitable for full fine-tuning due to frozen weights; QLoRA or prompt-tuning on the base bf16 Qwen3.6-27B is an alternative. Contact the maintainer (sakamakismile) or defer to the base Qwen3.6-27B documentation for tuning strategies. MTP head is restored in bf16, but its fine-tuning feasibility is Unknown.
When to avoid it — and what to weigh
- Vision/multimodal input required — Vision tower is physically deleted in this variant to save memory and reduce model size. If you need image-to-text or VLM capabilities, use the full Qwen3.6-27B or the VLM-capable Qwen3.6-27B-NVFP4 (non-text-only) instead.
- Non-Blackwell GPU or low-VRAM hardware — Model is specifically optimized for Blackwell SM120 (vLLM native modelopt path). Performance on older architectures (A100, H100, 4090) unverified. Requires ~15 GB VRAM for inference; not suitable for sub-16GB cards.
- Guaranteed exact-match output reproducibility or determinism requirements — Quantization (NVFP4) and speculative decoding introduce numerical divergence vs the fp32/bf16 baseline. For safety-critical or audit-heavy use cases, benchmarking against your tolerance is mandatory.
- Very small batch sizes or single-token inference latency critical — Speculative decoding shine on batch/long-form decodes; single short-request latency (50-token prompt) is ~14 ms (1/71 tok/s) without speculation, ~7.5 ms (1/132 tok/s) with n=3 speculation. Prefill is not heavily optimized in the card.
License & commercial use
Apache 2.0 (inherited from base Qwen3.6-27B). Permissive OSI-approved license permitting commercial use, modification, and redistribution under attribution.
Apache 2.0 is a permissive OSI license that explicitly permits commercial use, derivative works, and private deployment. No proprietary restrictions, gating, or commercial licensing fees apply. You may use this model in commercial products (SaaS, closed-source, for-profit) without seeking permission from Alibaba or sakamakismile, subject only to retaining the Apache 2.0 notice. However, verify integration with any proprietary vLLM or NVIDIA software licenses if used in a commercial service.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Strong |
| Assessment confidence | High |
Quantized model artifacts are safetensors format (no arbitrary code execution on load if using trusted transformers library). No security audit or adversarial robustness claims are made. MTP head is restored in bf16, but integrity depends on the open-source quantization pipeline (lna-lab/GGUF-to-NVFP4-SM120) being properly reviewed. Recommend standard LLM safety practices: input sanitization, output filtering, rate limiting, and monitoring for misuse in production. Quantization may reduce model's ability to recognize certain adversarial or harmful patterns; no impact assessment provided.
Alternatives to consider
Qwen3.6-27B-NVFP4 (compressed-tensors variant)
Same base model but uses slower compressed-tensors quantization format and MTP head is dropped (0% draft acceptance). ~1.7× slower on concurrent long-form decodes than this -Text-NVFP4-MTP variant. Choose if vision/VLM capability is needed (this one is text-only).
Osoleve/Qwen3.5-27B-Text-NVFP4-MTP
Qwen3.5 predecessor (not 3.6); same quantization and MTP-restoration approach. Smaller context window (4K trained max vs 256K). Use if you need the Qwen3.5 architecture or if 3.6 is not available in your region/tool.
Official Qwen/Qwen3.6-27B (bf16 base model)
Full-precision baseline (~55 GB VRAM required). No quantization loss; enables fine-tuning and full capability access. Choose if you have ample GPU memory, need exact output reproducibility, or plan to customize the model extensively.
Ship Qwen3.6-27B-Text-NVFP4-MTP with senior software developers
Qwen3.6-27B-Text-NVFP4-MTP delivers 1.7× faster inference on Blackwell hardware via optimized quantization and speculative decoding. Perfect for high-volume, long-context inference with Apache 2.0 commercial freedom. Start a proof-of-concept with vLLM or contact our AI infrastructure team to integrate into your self-hosted or cloud deployment.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Qwen3.6-27B-Text-NVFP4-MTP FAQ
Can I use this model commercially in a SaaS product?
What GPU do I need to run this?
Why is speculative decoding (MTP) better than vanilla decoding?
Can I fine-tune this quantized model?
Work with a software development agency
DEV.co is a software development agency delivering custom software development services to companies building on open source. Our software developers and web developers design, integrate, and ship production systems — spanning web development, APIs, AI, data, and cloud. If Qwen3.6-27B-Text-NVFP4-MTP is part of your open-source llms roadmap, our team can implement, customize, migrate, and maintain it.
Ready to Deploy a High-Throughput Quantized LLM?
Qwen3.6-27B-Text-NVFP4-MTP delivers 1.7× faster inference on Blackwell hardware via optimized quantization and speculative decoding. Perfect for high-volume, long-context inference with Apache 2.0 commercial freedom. Start a proof-of-concept with vLLM or contact our AI infrastructure team to integrate into your self-hosted or cloud deployment.