gpt-oss-20b-speculator.eagle3
This is a speculator model (854M parameters) designed to accelerate inference of openai/gpt-oss-20b using EAGLE-3 speculative decoding. It predicts multiple future tokens to reduce latency. It's not a standalone model—it requires the 20B verifier model and compatible serving infrastructure (vLLM). Apache-2.0 licensed and ungated.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | RedHatAI |
| Parameters | 855M |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | text-generation |
| Gated on HuggingFace | No |
| Downloads | 40k |
| Likes | 11 |
| Last updated | 2026-04-08 |
| Source | RedHatAI/gpt-oss-20b-speculator.eagle3 |
What gpt-oss-20b-speculator.eagle3 is
A 854M-parameter Eagle3Speculator trained via distillation from openai/gpt-oss-120b hidden states using the speculators library. Accepts num_speculative_tokens (1–5) to trade off acceptance rate and speedup. Evaluated on seven use cases (coding, math, QA, RAG, summarization, translation) showing acceptance lengths of 1.5–2.75 tokens depending on k and task. Requires vLLM 0.17.1+ and compatible chat template integration with gpt-oss-20b.
Run gpt-oss-20b-speculator.eagle3 locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="RedHatAI/gpt-oss-20b-speculator.eagle3")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
Estimate: 20B verifier + 854M speculator on single A100 (40 GB) or equivalent. Benchmark run used 1×A100 with vLLM 0.17.1 at batch size 128. For multi-GPU: tp=1 (tensor parallelism) indicated in example. Peak VRAM unknown; requires empirical profiling per batch size and num_speculative_tokens. FP16 or BF16 assumed but not explicitly stated.
Not applicable. This is a distilled speculator model trained on hidden states from gpt-oss-120b. Retraining or LoRA adaptation is not documented and likely not intended. Use case is fixed: acceleration of gpt-oss-20b inference.
When to avoid it — and what to weigh
- Standalone inference required — This model is a speculator only; it cannot generate output independently. Must pair with openai/gpt-oss-20b verifier and compatible serving runtime.
- Latency-insensitive workloads — If absolute throughput (tokens/sec) and cost are primary concerns over wall-clock latency, speculative decoding overhead may not justify the added complexity.
- Single-GPU or resource-constrained deployment — Running both verifier (20B) and speculator (854M) requires sufficient VRAM; minimal hardware benefit on CPU-only or edge devices.
- Non-vLLM serving infrastructure — Integration outside vLLM (e.g., llama.cpp, ONNX Runtime) is not documented; custom implementation required.
License & commercial use
Apache-2.0 (OSI-approved permissive license). No restrictions on use, modification, or distribution provided license text is retained.
Apache-2.0 permits commercial use without restriction. However, verify that the upstream verifier model (openai/gpt-oss-20b) and training datasets (Magpie-Llama-3.1-Pro, HuggingFaceH4/ultrachat) comply with your intended use case. The speculator itself poses no commercial-use barrier.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Adequate |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Good |
| Assessment confidence | High |
Standard LLM considerations apply. Model is distilled from gpt-oss-120b; inherit any latent biases or vulnerabilities. Speculative decoding does not introduce new attack vectors but reduces visibility into token-level verification. Serve behind authenticated endpoints if handling sensitive data. No CVE or security audit data provided.
Alternatives to consider
Ollama with speculative decoding (if supported)
Simpler single-binary deployment; trade-off is less explicit control over speculative parameters and narrower model ecosystem.
TensorRT-LLM speculative plugin
Nvidia's inference optimization framework; may offer better throughput on H100/A100 but steeper integration overhead and proprietary CUDA dependency.
llama.cpp with custom speculator (not out-of-box)
If CPU inference is mandatory; requires manual speculative decoding implementation, not provided by this model.
Ship gpt-oss-20b-speculator.eagle3 with senior software developers
Integrate this speculator model with vLLM for real-time conversational AI and reasoning tasks. Contact our team to evaluate on your workload or request hosted endpoint configuration.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
gpt-oss-20b-speculator.eagle3 FAQ
Can I use this model without the openai/gpt-oss-20b verifier?
Is this model suitable for commercial AI applications?
What GPU do I need to run this?
Does speculative decoding always improve latency?
Custom software development services
From first prototype to production, DEV.co delivers software development services around tools like gpt-oss-20b-speculator.eagle3. Our software development agency staffs experienced software developers and web developers for custom software development, web development, integrations, and ongoing support across open-source llms and beyond.
Ready to deploy low-latency LLM inference?
Integrate this speculator model with vLLM for real-time conversational AI and reasoning tasks. Contact our team to evaluate on your workload or request hosted endpoint configuration.