sarvam-30b-FP8-dynamic
sarvam-30b-FP8-dynamic is a quantized 30B parameter multilingual LLM optimized for inference. RedHatAI applied FP8 quantization to the original sarvamai/sarvam-30b model using LLM Compressor, reducing model size while maintaining ~99–101% accuracy recovery on standard benchmarks. It is validated on vLLM 0.18.0 and Red Hat AI platforms. The model supports 24 languages (primarily Indian and South Asian languages) and is released under Apache 2.0.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | RedHatAI |
| Parameters | 32.2B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 90.9k |
| Likes | 1 |
| Last updated | 2026-04-28 |
| Source | RedHatAI/sarvam-30b-FP8-dynamic |
What sarvam-30b-FP8-dynamic is
A FP8-quantized variant of sarvamai/sarvam-30b (32.2B parameters). Weights and activations of linear operators in transformer blocks are quantized to FP8 using dynamic per-token activation quantization; lm_head is excluded. Serialized in safetensors format. Evaluated against unquantized baseline on BBH, GSM8K, IFEval, MMLU-Pro, ARC-Challenge, HellaSwag, MMLU, TruthfulQA, and Winogrande, showing recovery rates 92.6–101.6%. Requires vLLM >= 0.18.0 for inference.
Run sarvam-30b-FP8-dynamic locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="RedHatAI/sarvam-30b-FP8-dynamic")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
ESTIMATE: ~16–20 GB VRAM for single-GPU inference (FP8 quantization reduces baseline ~64 GB FP16 footprint by ~75%). Tensor parallelism to 2 GPUs validated. Requires CUDA-capable GPU; CPU inference feasibility Unknown. vLLM deployment assumes modern NVIDIA GPU (H100, A100, L40S, RTX 4000+ generation).
Not addressed in card. Original model (sarvamai/sarvam-30b) fine-tuning approach Unknown. Quantized weights and activations may constrain LoRA/QLoRA feasibility; recovery from quantized base typically requires careful hyperparameter tuning or re-quantization. Recommend consulting vLLM and LLM Compressor documentation or contacting RedHatAI for official guidance.
When to avoid it — and what to weigh
- Long-context Applications — Context length is not documented. Without this specification, long-context use cases (e.g., document retrieval augmentation, book-length summarization) cannot be reliably assessed.
- Latency-Critical Real-time Systems — No inference latency benchmarks are provided. FP8 quantization typically reduces latency, but exact throughput and end-to-end response times remain Unknown; validate independently for SLA-sensitive applications.
- High-Precision Numerical Reasoning — IFEval accuracy drops to 92.63% recovery, suggesting some loss in instruction-following fidelity. Tasks requiring strict logical consistency or complex multi-step reasoning may benefit from unquantized baseline.
- Unsupported Languages — Model explicitly supports 24 Asian languages; use cases requiring other language families or low-resource languages are out of scope.
License & commercial use
Apache 2.0 license. Permissive OSI-approved open-source license permitting commercial use, modification, and distribution under the terms of the Apache 2.0 agreement.
Apache 2.0 is a permissive OSI license that explicitly allows commercial use, provided that: (1) a copy of the license is included, (2) significant changes are documented, and (3) a NOTICE file is maintained. No attribution to RedHatAI is legally required, but best practice is recommended. Verify compliance with internal IP policy and any downstream license obligations of deployed applications.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Adequate |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Strong |
| Assessment confidence | High |
Quantization does not eliminate LLM security risks: prompt injection, jailbreaking, and hallucination remain possible. No security audit, adversarial robustness testing, or model card sections on harmful output mitigation or safety layers are documented. Deployment should include: (1) input validation and prompt templating, (2) output monitoring for unsafe content, (3) access controls, (4) usage logging. Use in safety-critical or regulated domains (healthcare, finance, legal) requires independent security assessment. Out-of-scope note explicitly prohibits use violating laws or trade compliance—validate against your jurisdiction and export control rules.
Alternatives to consider
sarvamai/sarvam-30b (unquantized)
Original base model offers marginally higher accuracy (BBH +0.37%, IFEval +2.52%) at the cost of ~3× VRAM and slower inference; choose if accuracy is critical and hardware budget permits.
meta-llama/Llama-2-70b or Llama-3-70b (quantized variants)
Larger, English-dominant alternatives with stronger performance on English benchmarks and broader community ecosystem; multilingual performance and Indian language support lower priority.
mistralai/Mistral-7B or Mixtral-8x7B (quantized)
Smaller, faster alternatives with lower VRAM footprint if 30B capacity is unnecessary; trade-off is reduced multilingual coverage and benchmark performance.
Ship sarvam-30b-FP8-dynamic with senior software developers
sarvam-30b-FP8-dynamic offers production-grade quantized inference with Red Hat validation. Evaluate on your hardware, test multilingual capabilities, and deploy via vLLM or RHOAI. Start with a 1–2 GPU pilot to validate latency and accuracy for your use case.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
sarvam-30b-FP8-dynamic FAQ
Can I use this model commercially?
What GPU do I need, and how much VRAM?
Does this model support fine-tuning?
What is the maximum context length?
Software developers & web developers for hire
Adopting sarvam-30b-FP8-dynamic is usually one piece of a larger software development effort. As a software development agency, DEV.co provides software development services and web development expertise — pairing senior software developers and web developers with your team to design, build, and operate open-source llms software in production.
Ready to Deploy Efficient Multilingual AI?
sarvam-30b-FP8-dynamic offers production-grade quantized inference with Red Hat validation. Evaluate on your hardware, test multilingual capabilities, and deploy via vLLM or RHOAI. Start with a 1–2 GPU pilot to validate latency and accuracy for your use case.