Gemma-4-26B-A4B-it-NVFP4
Gemma-4-26B-A4B-it-NVFP4 is a community-quantized variant of Google's Gemma 4 Mixture-of-Experts model. It compresses the original 49 GB model to 16.5 GB via W4A4 quantization (4-bit weights and activations), roughly doubling inference throughput while retaining 97.6% average quality. The model supports text, image, and video inputs with a 256K token context window. Deployment requires vLLM with specific dependencies and a custom patch for correct MoE scaling.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | bg-digitalservices |
| Parameters | 15.1B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 87k |
| Likes | 33 |
| Last updated | 2026-04-05 |
| Source | bg-digitalservices/Gemma-4-26B-A4B-it-NVFP4 |
What Gemma-4-26B-A4B-it-NVFP4 is
This is a post-training W4A4 (FP4 weights, FP16 activations) quantization of Gemma 4's 25.2B-parameter MoE architecture (128 experts, 8 active per token). The quantizer addressed a core technical challenge: Gemma 4 fuses expert weights into 3D tensors unsupported by standard NVIDIA Model Optimizer, so the quantizer wrote a custom unfusing plugin. Calibration used 4,096 CNN/DailyMail samples to ensure each expert received sufficient tokens (~250 per expert). Benchmarks show 95.0% retained performance on math reasoning (GSM8K strict) and 99.1% on instruction-following (IFEval), with 2.07x speedup and 2.97x size reduction on DGX Spark (GB10 Blackwell, SM 12.1).
Run Gemma-4-26B-A4B-it-NVFP4 locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="bg-digitalservices/Gemma-4-26B-A4B-it-NVFP4")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
**ESTIMATE (verify before deployment):** ~16 GB VRAM for inference (FP8 KV cache at 256K tokens on DGX Spark with `gpu_memory_utilization=0.85`). Requires NVIDIA GPU with SM 12.1+ (Blackwell confirmed; Hopper/Ada untested). vLLM build with `transformers >= 5.4` mandatory. Marlin GEMM backend for W4A4 compute. CPU fallback behavior unknown.
Not supported. W4A4 is irreversible post-training quantization. Fine-tuning or LoRA adaptation would require either (a) dequantizing to BF16 (negating compression gains) or (b) training-aware quantization (requires retraining from scratch with quantization-aware loss, not documented). If adaptation is required, use the original unquantized Gemma 4 instead.
When to avoid it — and what to weigh
- High-Precision Math and Reasoning — GSM8K (strict) shows 4pp quality drop (86.96% → 82.64%), indicating rounding error accumulation in chained arithmetic. Avoid for symbolic math, formal reasoning, or applications requiring near-perfect numerical accuracy.
- Production Use Without Operational Maturity — This is a community quantization with a required vLLM patch not yet upstream. Deployment demands custom container builds, manual patching, and GPU-specific tuning (SM 12.1 Blackwell tested; generalization to other architectures is untested).
- Offline/Edge Deployment Without Validation — No benchmarks on CPU inference, mobile, or non-Blackwell GPUs. Marlin GEMM backend is Blackwell-specific; fallback behavior on older hardware is unknown.
- Fine-Tuning or Continued Training — W4A4 quantization is post-training only. Retraining or LoRA fine-tuning would require dequantization or training-aware quantization (not documented); feasibility is unclear.
License & commercial use
Apache 2.0. Inherited from base model (google/gemma-4-26B-A4B-it). This is a permissive OSI-approved license with clear commercial rights.
Commercial use is permitted under Apache 2.0. Self-hosting, SaaS deployment, and private LLM services are all allowed without restrictions. Note: Google's underlying Gemma 4 model carries no additional restrictions beyond Apache 2.0. The quantization itself introduces no new license barriers. Verify compliance with your jurisdiction's data residency and AI transparency laws independently.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | High |
| DEV.co fit | Strong |
| Assessment confidence | High |
No formal security audit documented. Quantization itself does not alter model outputs substantively (97.6% quality retention), so inherited model safety properties likely persist (e.g., Gemma 4's post-training alignment). Custom vLLM patch (`gemma4_patched.py`) adds attack surface — review before deploying to untrusted data. W4A4 format is proprietary (NVIDIA Model Optimizer); supply-chain risk depends on trust in NVIDIA's toolchain. No mention of input sanitization or adversarial robustness testing.
Alternatives to consider
google/gemma-4-26B-A4B-it (original BF16)
Higher quality (90.31% vs. 88.15% avg benchmark), simpler deployment (no patching), but 3x larger (49 GB) and 2x slower. Choose if quality/reliability outweighs resource constraints.
meta-llama/Llama-3.1-70B-Instruct (unquantized or AWQ/GPTQ-quantized)
Larger, higher capacity (70B vs. 25.2B active), but denser (no MoE), potentially requiring more VRAM. Consider if broader reasoning or code tasks matter more than memory efficiency.
NVIDIA/NeMo-Llama-2-70B-Chat (if NeMo inference available)
NVIDIA-native quantization and serving via NeMo Framework. More operational support than community patches, but smaller model and less recent than Gemma 4.
Ship Gemma-4-26B-A4B-it-NVFP4 with senior software developers
Evaluate Gemma-4-26B-A4B-it-NVFP4 on your infrastructure. Benchmark quality and latency against your current LLM stack. Start with the vLLM quick-start Docker example and test on Blackwell hardware (or validate fallback paths on your architecture). Contact [email protected] for reproducibility questions.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Gemma-4-26B-A4B-it-NVFP4 FAQ
Can I use this commercially?
What GPU hardware do I need?
Why does math reasoning drop 4 points but instruction-following barely changes?
Can I fine-tune or use LoRA?
Software development & web development with DEV.co
DEV.co is a software development agency delivering custom software development services to companies building on open source. Our software developers and web developers design, integrate, and ship production systems — spanning web development, APIs, AI, data, and cloud. If Gemma-4-26B-A4B-it-NVFP4 is part of your open-source llms roadmap, our team can implement, customize, migrate, and maintain it.
Deploy Efficient MoE Inference Today
Evaluate Gemma-4-26B-A4B-it-NVFP4 on your infrastructure. Benchmark quality and latency against your current LLM stack. Start with the vLLM quick-start Docker example and test on Blackwell hardware (or validate fallback paths on your architecture). Contact [email protected] for reproducibility questions.