Qwen3.5-122B-A10B-heretic-MTP-NVFP4
Qwen3.5-122B-A10B-heretic-MTP-NVFP4 is a quantized (W4A4) variant of Qwen's 122B mixture-of-experts model with abliteration modifications. It uses NVFP4 quantization for weights and activations while preserving critical components (vision encoder, MTP heads for speculative decoding) in higher precision. Designed for vLLM deployment with multi-token prediction (MTP) speculative decoding, achieving ~190 tok/s on dual RTX 6000 Pro Blackwell with TP=2.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | OptimizeLLM |
| Parameters | 73.7B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 47.8k |
| Likes | 8 |
| Last updated | 2026-04-11 |
| Source | OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4 |
What Qwen3.5-122B-A10B-heretic-MTP-NVFP4 is
122B parameter MoE (10B active) model quantized to W4A4 (FP4 E2M1, group size 16) using llm-compressor. MTP heads and vision components remain BF16 (4.8GB separate shard). Total size ~76GB across 16 shards. Calibrated on 512 samples (UltraChat + Nemotron-CC). MTP speculative decoding tested on vLLM 0.19+. Tokenizer from base Qwen3.5. Model is "abliterated" (safety filter reduction, KL ~0.09).
Run Qwen3.5-122B-A10B-heretic-MTP-NVFP4 locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
**Minimum:** 2× RTX 6000 Pro Blackwell (46GB VRAM each) with TP=2. **Estimated VRAM per GPU:** ~38–40GB (model ~76GB total divided by TP + KV cache overhead). **Quantization:** W4A4 (NVFP4) reduces footprint ~4–8× vs BF16. **Calibration (development):** ~400–445GB peak RAM+swap on CPU-only; GPU calibration requires MoE unfusing and is prone to OOM on single H100/A100 (80GB). **Production:** vLLM v0.19+ on Blackwell-generation or NVIDIA L40S/L40 (vendor support for NVFP4 required).
Fine-tuning guidance not stated in model card. Quantized W4A4 weights and activations complicate gradient-based updates. LoRA feasibility: Unknown (would require adapter insertion in non-quantized layers: vision encoder, LM head, embed tokens, MoE gates, MTP heads). QLoRA: Unlikely due to W4A4 precision constraints. Recommend: (1) use for inference only, or (2) fine-tune base [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) and re-quantize with llm-compressor. Reproduction notes suggest CPU-only calibration is feasible but slow (~3 days, 64-core CPU, 400GB swap).
When to avoid it — and what to weigh
- Single-GPU or consumer hardware — Model requires at least 2× high-end GPU (RTX 6000 Pro or similar) for acceptable performance. TP=2 minimum for MTP benefits. Not viable on A100 SXM (40GB) without higher quantization or offloading.
- Need for reproducible, certified safety behavior — Abliteration (KL ~0.09) intentionally reduces safety-filter steering. If regulatory compliance, content moderation, or guardrail consistency is required, this is not a fit.
- Limited infrastructure expertise — Deployment requires vLLM 0.19+, tensor parallelism configuration, MoE calibration knowledge, and memory tuning (malloc arenas, swap, offload folders). Model card notes reproduction challenges (OOM during calibration, transformer 5.x compatibility issues).
- Latency-sensitive single-user applications — Speculative decoding throughput gains apply to batch/concurrent load. For single-request latency, MTP adds scheduling overhead. Prefer smaller, finely-tuned models for real-time chat.
License & commercial use
Licensed under Apache 2.0. Permissive OSI-approved license permitting use, modification, and redistribution with original license and copyright notice. No special restrictions stated.
Apache 2.0 permits commercial use, modification, and derivative works. No license barriers to commercial deployment. However, **two caveats apply:** (1) The underlying base model [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) has separate licensing terms (Unknown from provided data — **Requires review** of Qwen's base model license, which may be non-permissive); (2) Abliteration by trohrbaugh introduces intentional safety-filter degradation, which may introduce liability or compliance risk depending on jurisdiction and use case (e.g., GDPR, AI Act, sector-specific guardrail mandates). **Recommendation:** legal review of Qwen base model terms and abliteration implications before production deployment.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Unknown |
| Documentation | Adequate |
| License clarity | Needs review |
| Deployment complexity | High |
| DEV.co fit | Good |
| Assessment confidence | Medium |
No explicit security audit, adversarial robustness, or dependency scanning mentioned. Abliteration reduces safety-filter effectiveness (KL ~0.09), potentially increasing risk of harmful outputs. Quantization introduces precision reduction; unknown impact on adversarial robustness. Safetensors format mitigates arbitrary code execution in pickle. Deployment on private infrastructure (no API exposure) reduces external attack surface. **Recommendation:** threat model review for use case; consider prompt injection testing and output filtering if exposed to untrusted input.
Alternatives to consider
Qwen/Qwen3.5-122B-A10B (base model, BF16)
Full precision, no quantization artifacts, simpler reproduction. Trade-off: ~4–8× larger VRAM, slower throughput on same hardware. Recommended if hardware headroom and cost allow.
RedHatAI/Qwen3.5-122B-A10B-NVFP4
Alternative NVFP4 quantization of the same base (non-abliterated). Maintains safety filters; model card references this as reference recipe. Choose if safety compliance is critical.
Meta/Llama-3.1-405B (or smaller variants)
Permissive license (Llama 2 Community), broader adoption, established tooling. Larger (405B), similar inference cost; unknown abliteration. Evaluate for your safety/compute trade-off.
Ship Qwen3.5-122B-A10B-heretic-MTP-NVFP4 with senior software developers
This model requires careful infrastructure planning (2× high-end GPUs, vLLM expertise, MoE calibration knowledge). Start with a cost/benefit analysis: does MTP throughput gain justify TP=2 overhead? Verify base Qwen license compliance and abliteration safety implications for your use case. Contact Devco for deployment architecture review.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Qwen3.5-122B-A10B-heretic-MTP-NVFP4 FAQ
Can I use this commercially without restrictions?
What GPU should I buy to run this?
How much faster is MTP speculative decoding, and is it worth the complexity?
Can I fine-tune this model?
Custom software development services
From first prototype to production, DEV.co delivers software development services around tools like Qwen3.5-122B-A10B-heretic-MTP-NVFP4. Our software development agency staffs experienced software developers and web developers for custom software development, web development, integrations, and ongoing support across open-source llms and beyond.
Ready to Deploy High-Throughput Quantized LLMs?
This model requires careful infrastructure planning (2× high-end GPUs, vLLM expertise, MoE calibration knowledge). Start with a cost/benefit analysis: does MTP throughput gain justify TP=2 overhead? Verify base Qwen license compliance and abliteration safety implications for your use case. Contact Devco for deployment architecture review.