Ornith-1.0-9B-MTP-GGUF
Ornith-1.0-9B-MTP-GGUF is a 9-billion-parameter language model quantized in GGUF format with a built-in multi-token prediction (MTP) head for speculative decoding acceleration. It runs on consumer and data-center GPUs via llama.cpp with 1.38–1.73× throughput gains depending on quantization and hardware. The model itself is a Qwen3.5-9B hybrid fine-tune, available in 10 quantization variants ranging from 3.9 GB (very low VRAM) to 18.4 GB (reference precision). MIT license, no access gate.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | protoLabsAI |
| Parameters | Unknown |
| Context window | Unknown |
| License | mit — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 55.1k |
| Likes | 60 |
| Last updated | 2026-07-04 |
| Source | protoLabsAI/Ornith-1.0-9B-MTP-GGUF |
What Ornith-1.0-9B-MTP-GGUF is
Ornith-1.0-9B is a Qwen3.5-9B base with linear + full attention layers, fine-tuned by deepreinforce-ai. The MTP variant adds a KL-distilled draft head (15 tensors, 1 layer) grafted into the trunk before quantization. The head enables llama.cpp's lossless multi-token self-speculative decoding (no separate draft model required) via `--spec-type draft-mtp`. GGUF quantization profiles span importance-matrix (IQ) and k-quantized (K) rungs. On Blackwell (RTX 50xx), NVFP4 format (6.6 GB) achieves ~306 tok/s with MTP vs. ~201 tok/s base; on Ampere, Q4_K_M (5.8 GB) hits ~153 tok/s MTP vs. ~105 tok/s base. Acceptance rates hold ~0.65–0.84 across quants at n-max 3. Requires llama.cpp ≥ b9616 (Qwen3.5 arch support, NVFP4 type 40 for Blackwell variant). Last modified 2026-07-04.
Run Ornith-1.0-9B-MTP-GGUF locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="protoLabsAI/Ornith-1.0-9B-MTP-GGUF")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
**Minimum VRAM:** 3.9 GB (IQ2_M); **balanced:** 5.8 GB (Q4_K_M, Ampere) or 6.6 GB (NVFP4, Blackwell); **reference:** 9.8 GB (Q8_0); **full precision:** 18.4 GB (BF16). **GPU:** NVIDIA, CUDA 11.0+. **CPU:** Intel/AMD x86_64; MTP speculative decoding requires GPU acceleration (CPU inference untested/unsupported). **Host RAM:** ~4 GB additional for llama.cpp runtime + context buffer. **Precision note:** IQ variants trade ~1–3% accuracy for 15–20% VRAM savings; k-quants (Q4–Q8) are lossless-adjacent.
Card does not document LoRA, QLoRA, or full fine-tuning feasibility. Base model (deepreinforce-ai/Ornith-1.0-9B) is a Qwen3.5 hybrid; compatibility with peft/bitsandbytes unknown. The MTP head is KL-distilled post hoc and frozen into GGUF (no separate training documented). For continued adaptation: (1) fine-tune the base Qwen3.5-9B model separately, then re-graft/re-quantize; (2) or use non-GGUF format (HF checkpoint) and handle MTP head manually. Requires review with protoLabsAI or upstream Qwen3.5 documentation.
When to avoid it — and what to weigh
- Bitwise Reproducibility Required — MTP's batched verification computes logits in different floating-point reduction order than sequential decode. Output distribution is identical, but greedy/temp-0 sequences may diverge. Card states this is expected, not a defect—but if your application requires deterministic token-by-token replay, use base model without speculative decoding.
- Context Length >8192 or Real-Time Streaming at Sub-100ms Latency — Card does not publish tested context lengths beyond 8192. MTP adds per-step parallel verify cost; while net throughput improves, first-token latency is not optimized. If generation must start within 50–100ms cold, profile before committing.
- CUDA Compute Capability <6.1 or Non-GPU Deployment — GGUF quantization and MTP speculative decoding are GPU-accelerated paths. CPU inference possible but slow (no benchmarks provided). Older GPUs (pre-Ampere) lack tensor cores; NVFP4 requires Blackwell. Verify GPU compatibility before deployment.
- Production Quality Assurance Without Acceptance Rate Monitoring — Card reports acceptance at ~0.65–0.84 depending on quant/prompt. Lower acceptance (creative prose: 287 tok/s vs. code: 330 tok/s) means MTP gains vary per workload. Requires profiling on representative data to validate SLA.
License & commercial use
MIT license. Base (Ornith-1.0-9B by deepreinforce-ai) is MIT. MTP head (protoLabsAI/Ornith-1.0-9B-MTP) is MIT. Derived GGUF quantizations are MIT. License is OSI-approved permissive.
MIT license permits commercial use, modification, and distribution. No restrictions stated. However, verify: (1) base Qwen3.5-9B upstream license (Alibaba; typically permissive but requires confirmation); (2) whether your intended commercial use (e.g., embedded model in a paid SaaS) requires trademark/attribution compliance under MIT terms (attribution required, not warranty). Recommend legal review of downstream Qwen license and MIT attribution chain before large-scale production deployment.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Low |
| DEV.co fit | Strong |
| Assessment confidence | High |
Standard LLM model considerations: (1) GGUF format is a binary serialization; verify source integrity (HuggingFace CDN + git commit hash). (2) No adversarial robustness claims stated; model inherits Qwen3.5-9B base properties (unknown). (3) Speculative decoding (MTP) does not introduce new attack surface (verification is deterministic). (4) No data privacy/training data documentation provided. (5) Quantization to GGUF may compress adversarial gradients; security implications unknown. For production: audit base Qwen3.5-9B security posture and consider input sanitization/rate-limiting on public-facing inference endpoints.
Alternatives to consider
Llama-2-13B GGUF (Meta/Hugging Face)
Larger, well-established baseline. No built-in MTP; requires separate draft model for speculative decoding. Simpler setup but slower throughput (~70–100 tok/s on comparable hardware without MTP).
Mistral-7B GGUF (Mistral AI)
Smaller footprint (3.3–6 GB), similar latency profile. No MTP head; wider community support and integration (Ollama, LM Studio native). Trade-off: lower absolute throughput, but better VRAM scaling.
DeepSeek-Coder-6.7B-GGUF (DeepSeek)
Comparable size, code-optimized training. No speculative decoding. If code-generation quality > throughput, may offer better output despite slower inference.
Ship Ornith-1.0-9B-MTP-GGUF with senior software developers
Download Ornith-1.0-9B-MTP-GGUF and benchmark on your hardware. Start with Q4_K_M (Ampere) or NVFP4 (Blackwell). See build recipes and full benchmark data in the model card. Questions? Open a community discussion on HuggingFace—protoLabs responds within 48h.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Ornith-1.0-9B-MTP-GGUF FAQ
Can I use Ornith-1.0-9B-MTP commercially in a SaaS product?
Which GGUF file should I download for an RTX 4090?
What is the difference between 'bundled' and 'standalone head' GGUF files?
Does MTP speculative decoding guarantee the same output as greedy decode without MTP?
Software developers & web developers for hire
DEV.co is a software development agency delivering custom software development services to companies building on open source. Our software developers and web developers design, integrate, and ship production systems — spanning web development, APIs, AI, data, and cloud. If Ornith-1.0-9B-MTP-GGUF is part of your open-source llms roadmap, our team can implement, customize, migrate, and maintain it.
Ready to Deploy High-Speed Local LLM Inference?
Download Ornith-1.0-9B-MTP-GGUF and benchmark on your hardware. Start with Q4_K_M (Ampere) or NVFP4 (Blackwell). See build recipes and full benchmark data in the model card. Questions? Open a community discussion on HuggingFace—protoLabs responds within 48h.