GLM-5-NVFP4
GLM-5-NVFP4 is a 435B-parameter (40B activated) mixture-of-experts language model quantized to NVIDIA's FP4 format. It is a production-ready quantization of ZAI's GLM-5 base model, optimized for inference on NVIDIA GPUs via vLLM or SGLang. The model supports 200K context length and is released under the MIT license, permitting commercial and non-commercial use. It trades minimal accuracy loss (~0.3-1% on benchmarks) for significant inference speedup and reduced memory footprint compared to FP8 baseline.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | nvidia |
| Parameters | 435.2B |
| Context window | Unknown |
| License | mit — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 34.2k |
| Likes | 30 |
| Last updated | 2026-04-10 |
| Source | nvidia/GLM-5-NVFP4 |
What GLM-5-NVFP4 is
GLM-5-NVFP4 applies post-training quantization (PTQ) using NVIDIA ModelOpt v0.42.0, quantizing weights and activations of linear layers in MoE transformer blocks to NVFP4 precision. Calibration used cnn_dailymail and Nemotron-Post-Training-Dataset-v2. Evaluated on MMLU Pro, GPQA Diamond, SciCode, IFBench, and HLE benchmarks; results show NVFP4 within 0.3–1% of FP8 baseline. Architecture: auto-regressive transformer with GLM-5 topology and MoE routing. Input: 1D text strings; Output: 1D text sequences. Supported runtimes: vLLM, SGLang. Test hardware: B300. Requires Linux + NVIDIA GPU (Blackwell or compatible).
Run GLM-5-NVFP4 locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="nvidia/GLM-5-NVFP4")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
Minimum: NVIDIA GPU with ~100–150 GB VRAM for full model (435B params, 40B activated; NVFP4 ≈ 220–260 GB raw weights). Recommended: Multi-GPU setup (e.g., 8×H100/L40S or 4×B300) for tensor parallelism. Example vLLM config uses --tensor-parallel-size 8. Inference tested on B300. Linux OS required. Estimate: 40B activated params at FP4 ≈ 20 GB active memory per GPU with typical batch sizes; full 435B ≈ 217 GB quantized weights. Requires CUDA 12.1+ and compatible vLLM/SGLang builds.
Not supported or documented. Post-training quantization (PTQ) freezes weights in NVFP4 format; further fine-tuning would require de-quantization or continued training in quantized domain (not standard practice). If fine-tuning is needed, consider downloading the unquantized GLM-5 base model (zai-org/GLM-5) and applying LoRA/QLoRA. NVIDIA ModelOpt may support quantization-aware training (QAT), but no evidence in card. Recommend contacting NVIDIA for quantized adaptation workflows.
When to avoid it — and what to weigh
- Requires Maximum Accuracy on Specialized Benchmarks — While NVFP4 matches FP8 closely, it shows ~1% drops on SciCode and IFBench. If benchmark-specific 0.5% accuracy improvements are critical, FP8 or full-precision GLM-5 is preferable.
- Non-NVIDIA Hardware or CPU Inference — Model is optimized and tested only on NVIDIA Blackwell GPUs with vLLM/SGLang. CPU inference or non-NVIDIA accelerators (AMD, Intel) are not officially supported and will be significantly slower.
- Minimal Fine-tuning or Domain Adaptation Expected — The model card provides no fine-tuning guidance, and post-training quantization complicates weight updates. If your use case requires frequent domain-specific LoRA or QLoRA adaptation, a non-quantized base model may be simpler.
- Production Deployment Without Toxicity/Bias Mitigation — Model card explicitly states the base model inherits toxic language and societal biases from internet crawl data. Requires guardrails and content moderation testing before production use in customer-facing applications.
License & commercial use
MIT License (permissive OSI-approved open-source license). Permits commercial and non-commercial use, modification, and distribution with attribution. No copyleft obligations. Full license text: https://opensource.org/licenses/MIT.
MIT license explicitly permits commercial use. Model card states: 'This model is ready for commercial/non-commercial use.' No gating, no subscription required. However, note: (1) Model is a derivative of ZAI's GLM-5 base model (separate license review recommended); (2) Base model inherits internet-crawled biases and toxicity—vendors must implement content guardrails and test for use-case compliance; (3) NVIDIA disclaims ownership and responsibility; ZAI's original GLM-5 license should be reviewed for any upstream restrictions. Summary: Commercial use is permissible under MIT, but conduct due diligence on base model and deploy safeguards.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Moderate |
| Documentation | Adequate |
| License clarity | Clear |
| Deployment complexity | High |
| DEV.co fit | Strong |
| Assessment confidence | High |
No explicit security audit or vulnerability disclosures mentioned. Considerations: (1) Model relies on vLLM/SGLang runtimes; assess those projects' security posture; (2) FP4 quantization reduces data density but does not provide cryptographic guarantees; (3) Tokenizer and tool parsers (glm47, glm45) allow remote code execution if --trust-remote-code is used—verify source before deployment; (4) Base model trained on uncurated internet data may embed prompt-injection or jailbreak vectors; (5) No mention of differential privacy or adversarial robustness; (6) NVIDIA VDP program (https://app.intigriti.com/programs/nvidia/nvidiavdp/detail) available for reporting security issues. Recommendation: Conduct security review of inference framework, implement input validation, and test for prompt injection before production.
Alternatives to consider
zai-org/GLM-5-FP8
Same base model, FP8 precision. ~0.3–1% higher accuracy on benchmarks but ~2× higher memory and slower inference than NVFP4. Choose if accuracy is paramount and GPU memory is abundant.
meta-llama/Llama-3.1-405B or similar (non-quantized)
Larger alternatives with broader community support and fine-tuning examples. Require significantly more VRAM (~500+ GB) and lack MoE efficiency. Consider if NVIDIA-specific ecosystem is not required.
Qwen/Qwen2.5-32B-Instruct or Mistral-Large (quantized)
Smaller, widely-deployed alternatives with more established quantization tooling (llama.cpp, GGUF). Simpler to self-host on limited hardware but lower reasoning/tool-use capability than GLM-5.
Ship GLM-5-NVFP4 with senior software developers
Evaluate GLM-5-NVFP4 for your self-hosted LLM, RAG, or AI agent use case. Start with our vLLM Docker example, verify GPU requirements, and test toxicity/bias guardrails before production. Contact our team to architect multi-GPU tensor parallelism or discuss fine-tuning workflows.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
GLM-5-NVFP4 FAQ
Can I use GLM-5-NVFP4 commercially?
What GPU do I need to run this model?
Can I fine-tune GLM-5-NVFP4?
How much faster is NVFP4 vs. FP8?
Software developers & web developers for hire
Need help beyond evaluating GLM-5-NVFP4? DEV.co is a software development agency offering software development services and web development for teams of every size. Our software developers and web developers build custom software, web applications, APIs, and open-source llms integrations — and maintain them long-term.
Ready to Deploy GLM-5-NVFP4?
Evaluate GLM-5-NVFP4 for your self-hosted LLM, RAG, or AI agent use case. Start with our vLLM Docker example, verify GPU requirements, and test toxicity/bias guardrails before production. Contact our team to architect multi-GPU tensor parallelism or discuss fine-tuning workflows.