AI Frameworks · jundot

omlx

oMLX is a macOS-native LLM inference server optimized for Apple Silicon, featuring continuous batching and a two-tier KV cache (RAM + SSD) to manage context efficiently. It runs locally on M1/M2/M3/M4 chips and offers a menu-bar interface for easy model management and a web dashboard for monitoring and configuration.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/jundot/omlx

17.6k

GitHub stars

1.5k

Forks

Python

Primary language

Apache-2.0

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	jundot/omlx
Owner	jundot
Primary language	Python
License	Apache-2.0 — OSI-approved
Stars	17.6k
Forks	1.5k
Open issues	679
Latest release	v0.4.4 (2026-06-16)
Last updated	2026-07-08
Source	https://github.com/jundot/omlx

What omlx is

Built in Python on MLX, oMLX implements continuous batching via mlx-lm's BatchGenerator and block-based KV cache with prefix sharing and Copy-on-Write semantics. The hot tier (RAM) spills to cold tier (SSD in safetensors format) with automatic LRU eviction, supporting text LLMs, VLMs, embeddings, and rerankers through an OpenAI-compatible API endpoint.

Quickstart

Get the omlx source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/jundot/omlx.git2cd omlx3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Local AI development on Apple Silicon Macs

Run inference for coding assistants, IDE integrations, and development tools without cloud dependencies. Menu-bar control and continuous batching make it practical for daily development workflows.

Multi-model serving with memory constraints

Pin small models in RAM while auto-evicting larger ones to SSD, with per-model TTL and manual load/unload. Ideal for orchestrating multiple LLMs and VLMs on a single Mac.

Context persistence across conversations

Two-tier KV cache preserves context across server restarts and model switches. Useful for stateful AI applications requiring long context reuse without recomputation.

Implementation considerations

Requires macOS 15.0+ (Sequoia), Python 3.10+, and Apple Silicon M1/M2/M3/M4; verify host hardware and OS before deployment.
SSD tier requires adequate free disk space for cold KV cache; configure via environment variables or `~/.omlx/settings.json` for production use.
Models auto-discovered from subdirectories in `--model-dir`; ensure proper directory structure and naming conventions to avoid loading issues.
Memory limit defaults to system RAM minus 8GB; adjust via CLI or settings for constrained environments to prevent OOM.
Optional custom kernels (GLM-5.2, MiniMax M3) require HEAD build with `OMLX_WITH_CUSTOM_KERNEL=1`; verify support for your target models.

When to avoid it — and what to weigh

Requires cross-platform deployment (Linux/Windows) — oMLX is macOS and Apple Silicon only. No Windows or Linux support, and ARM-only within macOS (M-series chips).
Production multi-user inference at scale — Single-machine architecture without distributed inference, authentication, or multi-tenant isolation. Not designed for cloud or SaaS deployments.
Need for GPU acceleration beyond Apple Metal — Locked to MLX (Apple Silicon neural engine). Cannot leverage NVIDIA/AMD GPUs or broader CUDA ecosystem.
Mature, battle-tested production stacks — Project started Feb 2026 (4 months old). Early stage with 679 open issues; stability and long-term support unknown.

License & commercial use

Apache License 2.0 (Apache-2.0). Permissive OSI-approved license allowing commercial use, modification, and distribution with attribution and liability disclaimer. No copyleft requirements.

Apache-2.0 is a permissive license that permits commercial use, including bundling and resale. However, verify compliance with any bundled dependencies (MLX, mlx-lm, etc.) for commercial products. No warranty provided; test thoroughly before production deployment. No commercial support or SLA mentioned in repository.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Adequate
License clarity	Clear
Deployment complexity	Low
DEV.co fit	Good
Assessment confidence	High

Security considerations

Localhost-only by default (`http://localhost:8000`). No built-in authentication, TLS, or multi-user isolation. Suitable for single-user development machines. For multi-user or network exposure, add reverse proxy (nginx) with auth/TLS. No mention of input sanitization, prompt injection mitigations, or data handling policies. Model loading from filesystem requires trust in model sources.

Alternatives to consider

LM Studio

Cross-platform (macOS, Windows, Linux) with similar menu-bar UX and OpenAI API. Broader hardware support (CPU, GPU) but lacks tiered KV cache and Apple Silicon optimization.

Ollama

Lightweight, multi-platform CLI-first LLM server with broad model library. Simpler API and easier deployment, but no admin dashboard or advanced memory management.

vLLM (via MLX wrapper)

Production-grade inference stack with advanced scheduling and multi-GPU support. Over-engineered for local macOS, requires Linux/CUDA, but mature and battle-tested.

Software development agency

Build on omlx with DEV.co software developers

Download oMLX from Releases or install via Homebrew. Requires macOS 15.0+ and Apple Silicon (M1–M4).

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

OpenLLM mlx-vlm web-llm

Related on DEV.co

Explore the category and the services that help you build with it.

AI Frameworks AI Development API Development Talk to DEV.co

omlx FAQ

Does oMLX work on Intel Macs?

No. Requires Apple Silicon (M1/M2/M3/M4) and macOS 15.0+. Intel Macs are not supported.

Can I expose oMLX to the network?

By design, oMLX listens on localhost only. To expose over network, use a reverse proxy (nginx, Caddy) with authentication and TLS. Not recommended without security hardening.

How much disk space does the SSD tier need?

Depends on model size and context window. Cold tier stores offloaded KV blocks in safetensors format. Allocate based on your largest models' projected context usage; configurable per model.

Is there commercial support or SLA?

Unknown. No support tier or SLA mentioned in repository. Contact [email protected] for inquiries. Community support likely via GitHub Issues.

Work with a software development agency

Need help beyond evaluating omlx? DEV.co is a software development agency offering software development services and web development for teams of every size. Our software developers and web developers build custom software, web applications, APIs, and ai frameworks integrations — and maintain them long-term.

Ready to run LLMs locally on your Mac?

Download oMLX from Releases or install via Homebrew. Requires macOS 15.0+ and Apple Silicon (M1–M4).

Talk to DEV.co Schedule a Consultation