Open-Source DevOps · iterative

cml

CML is an open-source CLI tool that automates ML model training, evaluation, and reporting within existing CI/CD pipelines (GitHub Actions, GitLab CI, Bitbucket Pipelines). It generates visual experiment reports automatically on pull requests, enabling MLOps workflows without requiring additional infrastructure or databases.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/iterative/cml

4.2k

GitHub stars

345

Forks

JavaScript

Primary language

Apache-2.0

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	iterative/cml
Owner	iterative
Primary language	JavaScript
License	Apache-2.0 — OSI-approved
Stars	4.2k
Forks	345
Open issues	86
Latest release	v0.20.6 (2024-10-24)
Last updated	2025-06-02
Source	https://github.com/iterative/cml

What cml is

CML provides CLI commands to orchestrate ML workflows, launch cloud/self-hosted runners, generate markdown reports with metrics and plots, and post results as PR comments or checks. It integrates natively with Git platforms and optionally works with DVC for data versioning and cloud storage providers for artifact management.

Quickstart

Get the cml source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/iterative/cml.git2cd cml3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

ML experiment tracking in pull requests

Automatically generate and post reports comparing model metrics, plots, and results across branches. Teams review model changes alongside code changes in the same PR interface.

GitOps-driven MLOps workflows

Codify data science workflows as CI/CD pipelines using GitHub Actions, GitLab CI, or Bitbucket. No separate MLOps platform required; git becomes the single source of truth for experiments.

Distributed ML training with cloud runners

Launch ephemeral compute on AWS EC2, Azure, or self-hosted infrastructure for resource-intensive training jobs. CML manages runner provisioning and cleanup within CI/CD workflows.

Implementation considerations

Requires Git platform account (GitHub, GitLab, or Bitbucket) and basic understanding of CI/CD workflows; no additional backend services to provision.
Python and Node.js environments must be available in CI runners; pre-built Docker images available to reduce setup friction.
Reports are markdown-based; team must define and commit metrics output format and plot generation logic to version control.
Cloud runner provisioning requires cloud provider credentials (AWS, Azure) and appropriate IAM/permissions; self-hosted runners avoid cloud costs but require infrastructure.
Tight coupling between experiment code, CI/CD configuration, and report generation; refactoring workflows requires coordinated changes across files.

When to avoid it — and what to weigh

Real-time model monitoring in production — CML is designed for CI/CD automation, not live model monitoring, alerting, or serving. Use dedicated MLOps platforms for production observability.
Complex experiment management with centralized UI — CML lacks a built-in web dashboard for browsing historical experiments or comparing runs across projects. Teams requiring a central experiment registry should consider dedicated tools.
Strict air-gapped or offline environments — CML relies on external Git platform APIs and optionally cloud runners. Integration with closed networks requires careful credential and API endpoint configuration.
Projects without existing CI/CD infrastructure — CML assumes GitHub, GitLab, or Bitbucket already in use. Adoption introduces CI/CD learning overhead if your team lacks familiarity with these platforms.

License & commercial use

Licensed under Apache License 2.0 (Apache-2.0), a permissive OSI-approved open-source license allowing commercial use, modification, and distribution with attribution and liability disclaimers.

Apache 2.0 is a permissive license explicitly allowing commercial use. No license-level restrictions on deploying CML in commercial MLOps workflows. However, commercial support, SLAs, and managed hosting are not mentioned in the repository; verify with Iterative (the maintainer) if enterprise support is required.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Adequate
License clarity	Clear
Deployment complexity	Moderate
DEV.co fit	Good
Assessment confidence	High

Security considerations

CML stores Git platform tokens (REPO_TOKEN, GITHUB_TOKEN) as CI/CD secrets; review secret rotation and scope. Cloud runner provisioning requires cloud provider credentials; ensure least-privilege IAM policies. Reports posted to public PRs may expose metrics/plots; consider data sensitivity. No explicit audit logging or encryption documentation provided; review infrastructure-level security controls in your CI/CD platform and cloud provider.

Alternatives to consider

MLflow

Centralized experiment tracking with a web UI, model registry, and production serving. Requires separate backend/database; more heavyweight than CML but better for teams needing a dedicated MLOps hub.

Weights & Biases (W&B)

Hosted experiment tracking, hyperparameter sweep, and model monitoring with rich dashboards. SaaS model with managed infrastructure; tighter integrations with popular ML frameworks but vendor lock-in and costs.

GitHub/GitLab native CI/CD + DVC

Use Git platform CI/CD directly without CML's abstraction layer; pair with DVC for data versioning. Lower-level control but requires more custom scripting and manual report generation.

Software development agency

Build on cml with DEV.co software developers

Start with CML's quick-start guide and fork the example project to see experiment reports in action. No additional infrastructure required—use your existing Git platform.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

DevOps-Python-tools mlops-v2 azure-devops-cli-extension

Related on DEV.co

Explore the category and the services that help you build with it.

Open-Source DevOps DevOps AI/ML Application Development Cloud Deployment Talk to DEV.co

cml FAQ

Do I need DVC to use CML?

No, DVC is optional. CML works with any Git-based workflow and artifact storage. DVC simplifies data versioning and reproducibility, but you can manage data/models separately.

Can CML run on self-hosted runners?

Yes. CML supports self-hosted GitHub Actions runners, GitLab runners, and Bitbucket runners. You provision and manage the compute; CML orchestrates workflows on those runners.

What happens to reports after a PR is merged?

Reports are comments/checks in the PR; they remain visible in PR history. There is no built-in archival or search across historical experiments. Teams often export or link to reports manually for record-keeping.

Does CML support private Git repos and cloud storage?

Yes. CML works with private repos using standard Git credentials. Cloud storage (S3, Azure Blob, GCP) is supported when configured with credentials as CI/CD secrets.

Work with a software development agency

Adopting cml is usually one piece of a larger software development effort. As a software development agency, DEV.co provides software development services and web development expertise — pairing senior software developers and web developers with your team to design, build, and operate open-source devops software in production.

Ready to automate your ML experiments?

Start with CML's quick-start guide and fork the example project to see experiment reports in action. No additional infrastructure required—use your existing Git platform.

Talk to DEV.co Schedule a Consultation