Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP
Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP is a 27-billion-parameter text-generation model optimized for deployment on 4× NVIDIA Blackwell GPUs. It combines hybrid attention (linear + periodic full), built-in multi-token prediction (MTP) for speculative decoding, and NVFP4 W4A4 quantization to fit ~7.2 GiB per GPU. The model supports reasoning tags, XML tool-calling, and long contexts (up to 256K tokens). Apache 2.0 licensed, non-gated, and designed for self-hosted inference via vLLM.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | sakamakismile |
| Parameters | 17.1B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 38.6k |
| Likes | 62 |
| Last updated | 2026-05-31 |
| Source | sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |
What Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP is
A Qwen3.5-family hybrid-attention model (17.1B parameters) with an integrated MTP draft head for speculative decoding. Quantized via NVIDIA's modelopt (NVFP4, W4A4, group size 16) on weights and core linear layers; embedding, routers, and lm_head remain higher precision. Supports reasoning via `<think>` tags, XML-formatted tool calls, and multimodal token inputs (though served as text-generation). Benchmark: ~81–83 tok/s single-stream (TP=4), scaling to ~880 tok/s at 24 concurrent requests (64K context). KV-cache uses FP8; context window up to 262K tokens. Requires vLLM ≥0.22 with qwen3_5 architecture and MTP support.
Run Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
4× NVIDIA RTX PRO 2000 Blackwell (16 GB GDDR6 each), connected via PCIe (NVLink not required). Model weights: ~7.2 GiB per GPU (NVFP4 W4A4). KV-cache: ~2–6 GiB per GPU depending on batch size and context length (FP8). Total GPU memory: 64 GB (4× 16 GB). Host: 32 GB shared memory (shmmax) for vLLM V1. CUDA 13.x + NVIDIA Container Toolkit for containerized deployment. Bare-metal setup also supported with matching toolkit.
Not documented. The checkpoint is NVFP4-quantized (W4A4); unquantized base weights and fine-tuning procedures are not provided. QLoRA on quantized weights is not explicitly supported. Domain adaptation would require either: (1) access to the unquantized Qwen3.5-27B, apply quantization post-training; or (2) evaluate inference-time prompt engineering / in-context learning. Requires review with the developer (sakamakismile) for training-time details.
When to avoid it — and what to weigh
- Single-GPU or dual-GPU inference — Model requires TP=4 sharding (4× 16 GB GPUs). Cannot run on 2× RTX 6000 or a single H100 without significant refactoring or post-hoc LoRA/pruning.
- Sub-millisecond latency requirements — Per-token latency at max throughput is 37–40 ms per request; single-stream latency is ~12 ms per token (~83 tok/s). Not suitable for real-time interactive chat where sub-50 ms end-to-end response time is critical.
- Model fine-tuning or domain adaptation at scale — Quantized W4A4 model is not explicitly documented for QLoRA or LoRA training. Fine-tuning feasibility and convergence are unknown; requires custom integration or access to the unquantized checkpoint.
- Environments without NVIDIA CUDA 13.x + Blackwell GPU support — NVFP4 kernels and Triton compilation require Blackwell (SM120) or newer. No CPU or AMD GPU support documented. Older NVIDIA architectures (A100, H100) may lack required Triton kernels for NVFP4.
License & commercial use
Apache License 2.0. Permissive, OSI-approved open-source license. Allows commercial use, modification, and distribution under the terms of Apache 2.0.
Apache 2.0 is a permissive OSI license that permits commercial use and derivative works, provided that the Apache 2.0 license text and any NOTICE file are included. No explicit restrictions on commercial deployment, SaaS offerings, or proprietary modifications are stated in the license itself. However, verify upstream dependencies (vLLM, NVIDIA modelopt, PyTorch) for any incompatible licenses if bundling a closed-source service.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Moderate |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Strong |
| Assessment confidence | High |
Quantization (NVFP4 W4A4) reduces model transparency but does not eliminate inference-time risks. No explicit security audit, adversarial robustness analysis, or input sanitization recommendations are stated. Deployment on self-hosted infrastructure mitigates data exfiltration risks vs. managed cloud APIs. Reasoning tags (`<think>`) and tool-call XML parsing introduce attack surface if untrusted input is fed without validation. Consider input length limits, prompt injection safeguards, and monitoring of tool-call payloads if exposed to end-users. Standard LLM safety practices (content moderation, rate limiting) apply.
Alternatives to consider
Meta Llama 3.1-70B (quantized, e.g. Ollama/llama.cpp)
Larger model (70B vs. 27B), may offer stronger reasoning and multi-lingual support. Llama 2/3 have broader community and open-source tooling (llama.cpp, Ollama). Requires more GPU memory (2–4× H100/A100); quantization reduces model size but less aggressive than NVFP4.
Qwen2.5-72B-Instruct (unquantized or bfloat16)
Newer Qwen family (2.5 vs. 3.5), larger model size, simpler (no MTP/hybrid attention). Standard bfloat16 inference via vLLM/TGI; broader framework support. Requires ~144 GiB VRAM unquantized or ~36 GiB quantized (1-bit/W4A8).
Deepseek-V3 (MoE, long-context)
Mixture-of-experts architecture allows sparse activation, lower per-token compute. Native long-context (128K+). If latency and throughput at fixed hardware are critical, MoE may offer better scaling. Licensing and quantization status require review.
Ship Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP with senior software developers
Huihui-Qwen3.6-27B-abliterated offers production-grade inference on modest hardware (4× RTX PRO 2000). Start with the provided Docker Compose setup, benchmark on your data, and evaluate reasoning/tool-calling for your agentic or RAG workload. Contact the developer or Devco for deployment support and fine-tuning feasibility reviews.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP FAQ
Can I use this model commercially?
What GPU hardware do I need?
How fast is the inference?
Can I fine-tune or LoRA this model?
Software developers & web developers for hire
Adopting Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP is usually one piece of a larger software development effort. As a software development agency, DEV.co provides software development services and web development expertise — pairing senior software developers and web developers with your team to design, build, and operate open-source llms software in production.
Ready to Deploy a Quantized Reasoning LLM?
Huihui-Qwen3.6-27B-abliterated offers production-grade inference on modest hardware (4× RTX PRO 2000). Start with the provided Docker Compose setup, benchmark on your data, and evaluate reasoning/tool-calling for your agentic or RAG workload. Contact the developer or Devco for deployment support and fine-tuning feasibility reviews.