AI Frameworks · predibase

lorax

LoRAX is an Apache 2.0–licensed inference server that serves thousands of fine-tuned LoRA adapters on a single GPU by dynamically loading task-specific weights per request. It combines heterogeneous batching, memory scheduling, and optimized inference kernels to maintain low latency and high throughput across many concurrent adapters.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/predibase/lorax

3.8k

GitHub stars

323

Forks

Python

Primary language

Apache-2.0

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	predibase/lorax
Owner	predibase
Primary language	Python
License	Apache-2.0 — OSI-approved
Stars	3.8k
Forks	323
Open issues	183
Latest release	lorax-0.4.0 (2025-01-13)
Last updated	2026-05-28
Source	https://github.com/predibase/lorax

What lorax is

LoRAX is a Python-based LLM serving framework built on PyTorch and Hugging Face Transformers that uses dynamic LoRA adapter loading, heterogeneous continuous batching, and CPU↔GPU adapter scheduling to enable scalable multi-model inference. It supports tensor parallelism, flash-attention, paged attention, SGMV kernels, quantization (bitsandbytes, GPTQ, AWQ), and OpenAI-compatible APIs for both REST and chat interfaces.

Quickstart

Get the lorax source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/predibase/lorax.git2cd lorax3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Multi-tenant SaaS LLM platforms

Host hundreds of customer-specific fine-tuned models on shared GPU infrastructure with per-request tenant isolation and dynamic adapter loading, reducing serving costs while maintaining API compatibility.

Cost-optimized inference for many task-specific models

Serve numerous domain-specific LoRA adapters (customer service, code generation, summarization, translation) from a single base model without duplicating GPU memory, ideal for enterprises with diverse workloads.

Low-latency specialized model inference

Leverage heterogeneous batching and asynchronous adapter scheduling to minimize request latency when switching between adapters, supporting real-time applications requiring multiple fine-tuned variants.

Implementation considerations

Requires Nvidia Ampere-generation GPU minimum (A100, H100, RTX 4090, etc.) and CUDA 11.8 drivers; validate against your target hardware before deployment.
Docker image recommended to avoid custom CUDA kernel compilation; ensure nvidia-container-toolkit is installed and Docker daemon is correctly configured for GPU passthrough.
Adapter discovery: adapters must be available on HuggingFace Hub, Predibase, or mounted local filesystem; validate adapter format (PEFT/Ludwig LoRA) and base model compatibility.
Memory budgeting: monitor GPU memory usage during concurrent requests; use `bitsandbytes`, GPTQ, or AWQ quantization if serving very large base models or numerous adapters simultaneously.
OpenAI API compatibility: multi-turn chat and structured output (JSON mode) are supported; verify your client SDKs and prompt templates align with the expected API contract.

When to avoid it — and what to weigh

Not using LoRA-based fine-tuning — LoRAX is purpose-built for LoRA adapters; if your workflow relies on full model fine-tuning, LoRAX offers no cost advantage and adds unnecessary operational overhead.
Serving single monolithic models at scale — For single large models or few adapters, vLLM, TensorRT-LLM, or other general-purpose inference engines may offer simpler deployment and comparable performance without adapter scheduling complexity.
CPU-only or older GPU deployments — LoRAX requires Nvidia GPUs (Ampere or newer), CUDA 11.8+, and Linux; it is not portable to AMD, Intel GPUs, or legacy hardware, limiting deployment flexibility.
Heavy customization of inference kernels — LoRAX uses pre-compiled CUDA kernels (flash-attention, paged attention, SGMV); integrating custom attention or quantization kernels requires rebuilding and testing the full stack.

License & commercial use

Apache License 2.0 (Apache-2.0). Permissive OSI-approved license allowing free redistribution, modification, and commercial use under standard attribution and liability disclaimers.

Apache 2.0 is a permissive OSI license that explicitly permits commercial use without additional licensing fees or proprietary restrictions. No commercial use restrictions are evident from the license text. However, ensure any dependencies (CUDA, underlying models, quantization libraries) meet your compliance requirements independently.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Strong
License clarity	Clear
Deployment complexity	Moderate
DEV.co fit	Strong
Assessment confidence	High

Security considerations

Runs untrusted code in adapter weights; validate all adapter sources (HuggingFace, Predibase, local) for integrity before loading. Per-request tenant isolation is claimed for private adapters but requires verification of implementation details (not provided). CUDA kernel dependencies introduce supply-chain risk. Network exposure via REST/OpenAI API requires authentication, rate limiting, and input validation (verify LoRAX configuration). No explicit security audit or vulnerability disclosure policy mentioned.

Alternatives to consider

vLLM

General-purpose LLM inference engine with paged attention and token-level scheduling; excels at single-model throughput but lacks native multi-adapter support and requires manual workarounds for LoRA loading.

TensorRT-LLM (Nvidia)

Highly optimized inference for single large models with compiled execution plans; requires model-specific optimization and lacks dynamic adapter loading, best for production inference of fixed models.

Baseten / Runwayml / Predibase managed services

Hosted alternatives that abstract away infrastructure, CUDA, and deployment complexity; higher cost per inference but lower operational burden, suitable if DevOps capacity is constrained.

Software development agency

Build on lorax with DEV.co software developers

LoRAX reduces inference costs for diverse fine-tuned models without sacrificing latency. Evaluate GPU requirements, adapter sources, and memory budgeting for your workload. Contact our team for hands-on deployment support.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

mlc-llm self-llm omlx

Related on DEV.co

Explore the category and the services that help you build with it.

AI Frameworks AI Development DevOps & Infrastructure API Development Talk to DEV.co

lorax FAQ

Can I use LoRAX with non-LoRA fine-tuned models?

No. LoRAX is designed specifically for LoRA adapters (supported via PEFT and Ludwig). Full model fine-tuning requires alternative serving engines like vLLM or TensorRT-LLM.

What GPUs does LoRAX support?

Nvidia Ampere-generation and newer (A100, H100, RTX 4090, L40S, etc.). Older architectures (V100, P100) are not supported. AMD and Intel GPUs are not supported.

How many adapters can I serve on a single GPU?

Depends on base model size, adapter size, quantization, and GPU memory. LoRAX schedules adapters between GPU and CPU memory; typical deployments serve 10s to 100s on a single H100 or A100. Run benchmarks with your workload.

Is per-request adapter loading thread-safe for multi-tenant use?

LoRAX claims per-request tenant isolation for private adapters, but the implementation details are not fully described in the README. Validate isolation in your threat model and consider network-layer authentication.

Work with a software development agency

DEV.co is a software development agency delivering custom software development services to companies building on open source. Our software developers and web developers design, integrate, and ship production systems — spanning web development, APIs, AI, data, and cloud. If lorax is part of your ai frameworks roadmap, our team can implement, customize, migrate, and maintain it.

Ready to optimize multi-model LLM serving?

Talk to DEV.co Schedule a Consultation