AI Frameworks · skyzh

tiny-llm

tiny-llm is an open-source educational course that teaches LLM inference optimization and serving on Apple Silicon using Python and MLX. It covers building core components from scratch—attention, quantization, flash attention, continuous batching—using the Qwen model, with a companion book and structured weekly modules.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/skyzh/tiny-llm

4.3k

GitHub stars

340

Forks

Python

Primary language

Apache-2.0

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	skyzh/tiny-llm
Owner	skyzh
Primary language	Python
License	Apache-2.0 — OSI-approved
Stars	4.3k
Forks	340
Open issues	11
Latest release	Unknown
Last updated	2026-06-13
Source	https://github.com/skyzh/tiny-llm

What tiny-llm is

A hands-on systems engineering course implementing LLM serving primitives (KV cache, quantized matmul, flash attention 2, continuous batching, paged attention) using MLX array APIs on macOS ARM64. Covers weeks 1–2 completely (model loading and basic inference) with week 3 (MoE, speculative decoding, RAG) in progress.

Quickstart

Get the tiny-llm source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/skyzh/tiny-llm.git2cd tiny-llm3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Learning LLM Systems Internals

Ideal for engineers studying how production LLM serving systems (vLLM, llama.cpp) optimize inference. Step-by-step implementations of attention, KV cache, and batching strategies build deep understanding without abstraction layers.

Building Local Serving on Apple Silicon

Practical course for macOS developers targeting Qwen models on M-series chips. Covers quantization (CPU/GPU), flash attention variants, and continuous batching optimized for ARM64 without requiring NVIDIA GPU infrastructure.

Teaching Systems Engineering Teams

Well-structured curriculum (book + code + tests) for onboarding engineers into inference optimization. Modular progression from basic ops (week 1) to advanced techniques (week 3) supports self-paced and cohort-based learning.

Implementation considerations

macOS ARM64 (Apple Silicon M1/M2/M3+) is a hard requirement for optimal performance; Linux/Windows users will struggle with MLX support and GPU acceleration.
Course assumes Python-only implementation; no C++, Rust, or compiled kernel bindings are used, which impacts production performance expectations.
Week 1–2 are production-ready reference implementations; Week 3 modules (MoE, speculative decoding, RAG, agents) are in-progress—plan accordingly if relying on those.
Model loading depends on official MLX quantized Qwen3 files; custom model formats or fine-tuned variants require manual adaptation.
KV cache, batching, and quantization logic are implemented in pure MLX—no fallback to vendor-specific kernels (e.g., Metal, CUDA) for edge cases.

When to avoid it — and what to weigh

Production LLM Service Needed Today — This is a teaching tool, not a hardened production system. Use vLLM, llama.cpp, or similar mature projects for real deployments. Week 3 is incomplete; speculative decoding, RAG, and agent frameworks lack full implementation.
Non-Apple Silicon or GPU Environments — Course is tightly coupled to MLX and macOS ARM64. NVIDIA GPU or x86 Linux setups will require significant adaptation. Not designed for cloud datacenter or heterogeneous hardware deployments.
Seeking Pre-Built Model Integration — Codebase supports only Qwen models in specific MLX-quantized formats. No built-in adapters for HuggingFace, OpenAI API, or other model sources. Week 3 (RAG, tool calling) is work-in-progress and undocumented.
Minimal Dependencies or Offline Use — Requires MLX framework, Python environment, and access to official Qwen3 model files. No clear guidance on dependency pinning, offline setup, or reproducible environment isolation.

License & commercial use

Apache License 2.0 (Apache-2.0): permissive, OSI-approved open-source license. Allows commercial use, modification, and distribution with attribution and liability disclaimer. Source code can be used in proprietary products.

Apache-2.0 permits commercial use, but this is a course/educational project, not a production system. Any commercial LLM serving product derived from tiny-llm must retain Apache-2.0 license headers and acknowledge the original source. Verify compliance with your legal team before shipping; maturity and warranty disclaimers apply.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Strong
License clarity	Clear
Deployment complexity	High
DEV.co fit	Good
Assessment confidence	High

Security considerations

Course-oriented project; no security audit, threat model, or hardening for production workloads. Model loading uses official MLX-quantized Qwen files—verify source integrity. No input validation, rate limiting, or isolation guarantees. Designed for local/trusted environments. If deploying any derived system with untrusted user input, add your own defenses (input sanitization, sandboxing, resource limits).

Alternatives to consider

vLLM

Production-grade LLM serving framework with CUDA/ROCm/CPU backends, dynamic batching, and paged attention. More mature and scalable, but higher abstraction level—less suitable for systems learning.

llama.cpp

Lightweight C++ inference engine for CPU and Apple Metal, supporting diverse quantization. Easier to deploy locally on macOS, but fewer teaching materials on optimization internals.

MLC LLM

MLX-based project with similar ARM64 focus; production deployment and broader hardware support. Steeper learning curve; not designed as a course.

Software development agency

Build on tiny-llm with DEV.co software developers

Explore tiny-llm's structured course on GitHub, work through the step-by-step modules, and join the community Discord. Ideal for engineers learning inference optimization on Apple Silicon.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

self-llm nano-vllm TinyLLama-v0

Related on DEV.co

Explore the category and the services that help you build with it.

AI Frameworks AI Application Development Custom Software Development Talk to DEV.co

tiny-llm FAQ

Can I use tiny-llm for production inference serving?

Not recommended. It is a teaching course for learning LLM systems internals. vLLM, llama.cpp, or similar production systems are better choices. Week 3 features (RAG, agents) are incomplete.

Does it work on Linux, Windows, or NVIDIA GPUs?

Not as designed. MLX is Apple Silicon–only; significant porting effort would be required for other platforms. Consider llama.cpp or vLLM for cross-platform support.

What models does it support?

Qwen3 models in official MLX-quantized format. Custom or fine-tuned models require manual adaptation. Week 3 RAG/agent features are incomplete.

Is there commercial support or an SLA?

No formal support. Community help via Discord. Apache-2.0 license permits commercial derivative works, but you assume all risk and responsibility for production use.

Software development & web development with DEV.co

DEV.co helps companies turn open-source tools like tiny-llm into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your ai frameworks stack.

Ready to Master LLM Systems?

Explore tiny-llm's structured course on GitHub, work through the step-by-step modules, and join the community Discord. Ideal for engineers learning inference optimization on Apple Silicon.

Talk to DEV.co Schedule a Consultation