Mistral-Small-24B-Instruct-2501-AWQ
Mistral-Small-24B-Instruct-2501-AWQ is a 24 billion parameter instruction-tuned language model from Mistral AI, quantized to 4-bit INT4 by stelterlab using AutoAWQ. It supports 32k context, multilingual input (12+ languages), function calling, and runs on modest hardware (single RTX 4090 or 32GB RAM). Published under Apache 2.0, it is gated=false and freely downloadable with 221k+ downloads.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | stelterlab |
| Parameters | 23.6B |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 221.9k |
| Likes | 29 |
| Last updated | 2025-03-30 |
| Source | stelterlab/Mistral-Small-24B-Instruct-2501-AWQ |
What Mistral-Small-24B-Instruct-2501-AWQ is
24B parameter instruction-tuned transformer with Tekken tokenizer (131k vocab), INT4 AWQ quantization, 32k context window, native function calling, JSON output support, and system prompt adherence. Compatible with vLLM, transformers, and text-generation-inference frameworks. Multilingual (EN, FR, DE, ES, IT, PT, ZH, JA, RU, KO confirmed). Last modified 2025-03-30. Original base by Mistral AI; quantization by stelterlab.
Run Mistral-Small-24B-Instruct-2501-AWQ locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="stelterlab/Mistral-Small-24B-Instruct-2501-AWQ")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
ESTIMATE: ~55 GB GPU VRAM (bf16/fp16, full precision). Quantized (INT4 AWQ): ~14–16 GB GPU VRAM or 32 GB RAM on CPU. RTX 4090 (24 GB) or modern H100 (80 GB) suitable; MacBook 32 GB satisfies quantized inference. Transformer-based, standard throughput depends on batch size and sequence length (32k max context).
Card mentions fine-tuning capability for subject-matter experts but provides no LoRA/QLoRA guidance. Quantization (INT4 AWQ) typically precludes direct fine-tuning; QLoRA on quantized weights or re-quantization after LoRA on full weights (unquantized base) is standard. Requires access to mistralai/Mistral-Small-24B-Base-2501 or mistralai/Mistral-Small-24B-Instruct-2501 (unquantized) for typical fine-tuning workflows. Not clearly documented.
When to avoid it — and what to weigh
- Production systems requiring very long context — 32k context window is moderate. For document processing over 50k+ tokens, larger models (Llama 3.3-70B, Qwen 2.5-32B) or retrieval-augmented strategies are preferable.
- High-volume, low-latency cloud inference at scale — While quantized, running 24B parameters across thousands of concurrent requests may require cluster deployment. Cloud API models (GPT-4o-mini) offer simpler scaling and SLA guarantees.
- Specialized domains without fine-tuning capacity — General-purpose model; specialized medical, legal, or domain-specific tasks will require fine-tuning or RAG integration. Card acknowledges fine-tuning for subject-matter experts but no pre-tuned variants provided.
- Guaranteed output format or determinism — LLM outputs are probabilistic. JSON output is stated but not guaranteed. Critical systems requiring 100% structural compliance need post-processing validation or constrained generation.
License & commercial use
Apache 2.0. Permissive OSI-approved open-source license. Allows use, modification, and distribution for commercial and non-commercial purposes, provided Apache 2.0 attribution and notice are retained.
Apache 2.0 explicitly permits commercial use (card states 'allowing usage and modification for both commercial and non-commercial purposes'). Model weights and quantized version are freely available (gated=false). No commercial license restrictions detected. However, verify with legal team if Mistral AI's original model includes any undocumented restrictions or if your deployment region has export/compliance implications. Quantization is by stelterlab (third party); ensure stelterlab's quantization process respects Apache 2.0.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Low |
| DEV.co fit | Strong |
| Assessment confidence | High |
Standard LLM considerations apply: quantized model reduces memory attack surface vs. full-precision. No explicit security audit, threat model, or robustness testing documented. Card does not claim hardened output, jailbreak resistance, or prompt injection mitigations. Local deployment avoids API-based data exposure but requires secure host infrastructure. INT4 quantization may reduce model expressiveness slightly, potentially affecting adversarial robustness—not quantified. No data poisoning or training-time attack details provided. Use standard LLM safety practices: prompt validation, output filtering, and user input sanitization.
Alternatives to consider
Qwen2.5-32B-Instruct
32B parameters, 128k context (vs. 24B/32k here). Slightly better on math_instruct (0.819 vs. 0.706) and wildbench (52.73 vs. 52.27). Larger model, higher VRAM cost but potentially better accuracy for reasoning-heavy workloads.
Llama-3.3-70B-Instruct
70B parameters, state-of-the-art instruction following and reasoning. Stronger on gpqa_main (0.531 vs. 0.453) and humaneval (0.854 vs. 0.848). Requires ~140 GB GPU VRAM; suitable for enterprises with infrastructure. Non-Apache license (Llama 2/3 Community); commercial use requires Llama AI review.
Gemma-2-27B-IT
27B parameters, Google-backed, permissive Apache 2.0 license. Slightly lower performance (mmlu_pro 0.536 vs. 0.663) but excellent community support and integration. Good alternative if you prioritize Google ecosystem and lower VRAM (27B).
Ship Mistral-Small-24B-Instruct-2501-AWQ with senior software developers
Start with vLLM on GPU or transformers on CPU. Full Apache 2.0 commercial license, no gating, 221k+ downloads. Ideal for cost-sensitive, privacy-first applications. Review benchmark comparisons and hardware requirements above, then pull the model from Hugging Face.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Mistral-Small-24B-Instruct-2501-AWQ FAQ
Can I use this model commercially?
What GPU/hardware do I need?
How does this compare to GPT-4o-mini?
Can I fine-tune this quantized version?
Software developers & web developers for hire
DEV.co helps companies turn open-source tools like Mistral-Small-24B-Instruct-2501-AWQ into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your open-source llms stack.
Deploy Mistral-Small-24B-Instruct-2501-AWQ Locally
Start with vLLM on GPU or transformers on CPU. Full Apache 2.0 commercial license, no gating, 221k+ downloads. Ideal for cost-sensitive, privacy-first applications. Review benchmark comparisons and hardware requirements above, then pull the model from Hugging Face.