AI Frameworks · mosaicml

llm-foundry

LLM Foundry is an open-source training and deployment toolkit for large language models, built and maintained by Databricks (formerly MosaicML). It provides end-to-end workflows for training, fine-tuning, evaluating, and deploying LLMs at scale, with support for models ranging from 125M to 70B parameters.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/mosaicml/llm-foundry

4.4k

GitHub stars

586

Forks

Python

Primary language

Apache-2.0

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	mosaicml/llm-foundry
Owner	mosaicml
Primary language	Python
License	Apache-2.0 — OSI-approved
Stars	4.4k
Forks	586
Open issues	65
Latest release	v0.22.0 (2025-07-29)
Last updated	2026-03-25
Source	https://github.com/mosaicml/llm-foundry

What llm-foundry is

Python-based codebase built on PyTorch and Composer framework, offering modular training pipelines, data preparation utilities, inference optimization scripts, and benchmarking tools. Supports HuggingFace model integration, distributed training on NVIDIA GPUs (A100, H100), and export to HuggingFace or ONNX formats.

Quickstart

Get the llm-foundry source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/mosaicml/llm-foundry.git2cd llm-foundry3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Large-scale LLM pre-training and fine-tuning

Production-grade training infrastructure for models 125M–70B parameters; includes data preparation, distributed training, and throughput benchmarking. Suitable for organizations with GPU clusters and in-house training needs.

Rapid model experimentation and evaluation

Quick iteration on model architectures, datasets, and training techniques with modular design. Built-in evaluation on academic in-context-learning tasks and inference benchmarking.

Model optimization and deployment

Tools to convert models to HuggingFace or ONNX format, profile inference latency/throughput, and deploy optimized versions. Useful for inference cost reduction and latency-critical applications.

Implementation considerations

Requires PyTorch 2.4+ and CUDA 12.8 environment; setup complexity increases with distributed training. Docker images provided but not fully documented in excerpt.
Data preparation step converts raw text to StreamingDataset format; plan for ETL pipeline and storage for large datasets.
Model selection critical: verify commercial use rights for chosen base model (MPT vs. DBRX vs. HuggingFace) before deployment.
Training hyperparameters, optimization techniques, and hardware configuration directly impact cost and convergence; benchmarking scripts available for profiling.
Inference optimization (ONNX export, quantization) is optional but recommended for production latency/cost targets.

When to avoid it — and what to weigh

No GPU infrastructure or limited compute — Codebase assumes multi-GPU setups (A100/H100). Tested primarily on enterprise NVIDIA hardware; consumer or AMD cards not actively supported, though community reports some success.
Preference for managed/serverless training — Requires managing infrastructure, environment setup, and distributed training orchestration. Users seeking fully managed platforms should evaluate MosaicML's commercial platform or alternatives like Hugging Face AutoTrain.
Uncertainty about commercial model licensing — MPT models have mixed licensing: some variants (e.g., MPT-7B-Chat) prohibit commercial use. DBRX allows commercial use under Databricks' open license. Requires careful model selection and legal review per use case.
Minimal documentation or limited community support needed — While README and tutorials exist, no comprehensive API reference, architecture deep-dives, or troubleshooting guides evident in excerpt. Community support via Slack, but responsiveness unknown.

License & commercial use

LLM Foundry codebase itself is Apache-2.0 (permissive, allows commercial use of the software). However, trained model weights follow separate licenses: MPT models are either Apache-2.0 or non-commercial; DBRX is available under Databricks Open Source License with Acceptable Use Policy. Always verify the specific model's license before deployment.

The LLM Foundry codebase (Apache-2.0) can be used commercially. However, the trained models it produces or distributes have mixed licensing: MPT-7B-Chat and MPT-30B-Chat explicitly prohibit commercial use; MPT-7B, MPT-30B (base/instruct), and DBRX models permit commercial use under stated terms. Requires legal review of the specific model's license and acceptable use policy before commercial deployment.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Adequate
License clarity	Needs review
Deployment complexity	High
DEV.co fit	Good
Assessment confidence	High

Security considerations

No security audit or hardening details provided in excerpt. Codebase runs arbitrary Python/PyTorch code on GPU infrastructure; standard DevSecOps practices apply (code review, dependency scanning, supply-chain verification). Data handling (StreamingDataset pipeline) should be reviewed for sensitive information leakage. MosaicML platform use adds third-party security dependency; review SOC 2 or similar compliance if required.

Alternatives to consider

Hugging Face Transformers + AutoTrain

Simpler UI-driven fine-tuning for smaller models; managed hosting option. Fewer low-level optimization tools but lower barrier to entry.

vLLM / Ray Serve

Inference-focused frameworks with superior deployment ergonomics and multi-framework support. Better for production serving; less focused on pre-training.

DeepSpeed / Megatron-LM

Lower-level, research-oriented training frameworks with finer control; steeper learning curve but more flexible for custom architectures.

Software development agency

Build on llm-foundry with DEV.co software developers

LLM Foundry provides battle-tested tools for pre-training and fine-tuning, but requires GPU infrastructure and operational expertise. Evaluate your team's capabilities, verify model licensing for your use case, and consider the MosaicML platform for managed training.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

mlc-llm llmfit self-llm

Related on DEV.co

Explore the category and the services that help you build with it.

AI Frameworks AI Application Development Custom Software Development Cloud Deployment & Infrastructure Talk to DEV.co

llm-foundry FAQ

Can I use LLM Foundry for commercial applications?

The codebase itself (Apache-2.0) is permissible for commercial use. However, the trained model weights have mixed licenses. MPT-Chat models prohibit commercial use; MPT base/instruct and DBRX permit it. Always verify the specific model's license before production deployment.

What hardware is required?

Actively tested on NVIDIA A100-40GB/80GB and H100-80GB with PyTorch 2.7.0 and CUDA 12.8. Consumer NVIDIA and AMD cards may work but are not officially supported. Self-hosted multi-GPU infrastructure or MosaicML platform required.

Can I fine-tune on smaller models or single GPUs?

Community tutorials demonstrate single-GPU fine-tuning (e.g., MPT-7B on Google Colab), but codebase is optimized for distributed training. Implementation for smaller setups requires adaptation; benchmark your hardware first.

What about inference optimization and latency?

Tools provided to export to HuggingFace or ONNX format and benchmark latency/throughput. Further optimization (quantization, distillation) requires external tools. ONNX export enables non-PyTorch runtimes (TensorRT, ONNX Runtime).

Software developers & web developers for hire

DEV.co helps companies turn open-source tools like llm-foundry into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your ai frameworks stack.

Ready to train custom LLMs at scale?

Talk to DEV.co Schedule a Consultation