Jan-v3.5-4B-gguf
Jan-v3.5-4B is a 4-billion-parameter open-source LLM fine-tuned for math reasoning and conversational personality. It runs locally via vLLM or llama.cpp, uses Apache 2.0 licensing, and targets users who want a smaller model with distinct personality traits rather than neutral assistant behavior.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | janhq |
| Parameters | Unknown |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 305.5k |
| Likes | 26 |
| Last updated | 2026-03-24 |
| Source | janhq/Jan-v3.5-4B-gguf |
What Jan-v3.5-4B-gguf is
Based on Qwen3-4B architecture with 36 layers, 32 Q-heads and 8 KV-heads (GQA), 3.6B non-embedding parameters. Native context length 262,144 tokens. Fine-tuned on curated math and identity datasets by Menlo Research. Available in GGUF quantization format. No information on original pretraining data, exact parameter counts per variant, or benchmark metrics.
Run Jan-v3.5-4B-gguf locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="janhq/Jan-v3.5-4B-gguf")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
Estimated 8–12 GB VRAM for inference at FP16 or Q8 quantization; lower with Q4/Q5 variants (4–6 GB estimated). Running at FP32 not practical for consumer hardware. CPU-only inference feasible but slow; GPU (NVIDIA/AMD/Metal) strongly recommended. No official specs provided—verification with actual deployment required.
Model card does not discuss LoRA, QLoRA, or parameter-efficient fine-tuning feasibility. As a 4B fine-tuned variant, LoRA adapters are plausible but require separate testing. Full fine-tuning would demand significant VRAM (>16 GB). Contact janhq for official guidance or consult base model Jan-v3-4B-base-instruct documentation.
When to avoid it — and what to weigh
- Production-scale deployments requiring sub-100ms latency — 4B models incur inference latency unsuitable for high-throughput APIs. Consider larger-capacity serving infrastructure or smaller distilled models.
- Neutral, corporate-friendly tone required — Jan-v3.5 has deliberate personality (lowercase defaults, casual humor, self-aware style). Not suitable for formal customer service, compliance-heavy interactions, or risk-averse brand voice.
- Benchmark-driven model selection — No published benchmarks, comparative metrics, or evaluation against standard LLM leaderboards provided. Cannot verify math reasoning claims quantitatively.
- Extensive code generation or complex reasoning — Training data focuses on math and identity. No mention of code datasets, reasoning chains, or logic-heavy capabilities. Unsuitable for software engineering tasks without validation.
License & commercial use
Apache 2.0 license. Permissive OSI-approved license allowing commercial use, modification, and distribution with attribution and liability disclaimer.
Apache 2.0 is a permissive open-source license that explicitly permits commercial use, derivative works, and private modification. No gating, no model card restrictions beyond attribution. Commercial deployment is legally supported. However, downstream liability and warranty disclaimers apply per Apache 2.0 terms. Verify compliance with your legal counsel if model outputs or behavior carry material risk.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Adequate |
| License clarity | Clear |
| Deployment complexity | Low |
| DEV.co fit | Good |
| Assessment confidence | Medium |
No security audit, adversarial robustness data, or harmful output testing documented. Personality training may increase likelihood of sarcasm, humor, or off-brand outputs in sensitive contexts. Recommend sandboxing and output validation for production. Model runs locally (reduces supply-chain risk vs. API-only models). No mention of data poisoning detection or training data filtering.
Alternatives to consider
Llama 2 7B or Mistral 7B
Similar parameter count, more proven benchmarks and broader community adoption. Neutral tone. Better for general-purpose use without personality constraints.
Qwen2.5-4B (if available)
Same family as Jan's base architecture; likely better benchmarks and official Alibaba support. Better for math without personality overhead.
TinyLlama 1.1B or Phi-2
Smaller footprint for edge/mobile. Trade math reasoning for lower latency and VRAM. Only if you prioritize hardware constraints over capability.
Ship Jan-v3.5-4B-gguf with senior software developers
Test Jan-v3.5-4B on your math and conversational workloads. Start with vLLM or llama.cpp, or integrate directly into Jan Desktop. For production sizing, infrastructure planning, or fine-tuning guidance, consult our AI engineering team.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Jan-v3.5-4B-gguf FAQ
Can I use Jan-v3.5-4B in a commercial product?
What GPU do I need?
How does Jan-v3.5 compare to the base Jan-v3-4B model in math performance?
Will the personality affect my use case?
Work with a software development agency
Need help beyond evaluating Jan-v3.5-4B-gguf? DEV.co is a software development agency offering software development services and web development for teams of every size. Our software developers and web developers build custom software, web applications, APIs, and open-source llms integrations — and maintain them long-term.
Ready to Deploy Jan-v3.5-4B?
Test Jan-v3.5-4B on your math and conversational workloads. Start with vLLM or llama.cpp, or integrate directly into Jan Desktop. For production sizing, infrastructure planning, or fine-tuning guidance, consult our AI engineering team.