AI Frameworks · Andyyyy64

whichllm

whichllm is a CLI tool that automatically detects your hardware and recommends the best local LLM to run on it, ranked by real benchmark scores rather than just model size. It supports multiple hardware types (NVIDIA, AMD, Intel, Apple Silicon, CPU-only) and helps you find the optimal model-quantization pair for your specific setup.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/Andyyyy64/whichllm

5.6k

GitHub stars

298

Forks

Python

Primary language

MIT

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	Andyyyy64/whichllm
Owner	Andyyyy64
Primary language	Python
License	MIT — OSI-approved
Stars	5.6k
Forks	298
Open issues	15
Latest release	v0.5.15 (2026-07-03)
Last updated	2026-07-03
Source	https://github.com/Andyyyy64/whichllm

What whichllm is

whichllm queries HuggingFace's live model registry, applies hardware-aware VRAM and bandwidth calculations (accounting for KV cache, quantization, MoE active params, and unified memory vs PCIe offload), scores candidates against merged benchmarks (LiveBench, Artificial Analysis, Aider, Chatbot Arena ELO, Open LLM Leaderboard), and ranks by quality-speed-fit tradeoffs. It supports GGUF (via llama-cpp-python), AWQ/GPTQ (via transformers), and FP16/BF16 formats.

Quickstart

Get the whichllm source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/Andyyyy64/whichllm.git2cd whichllm3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Pre-purchase GPU hardware evaluation

Simulate target GPUs (RTX 4090, RTX 5090, etc.) before buying to verify which models will actually run at acceptable speed on your target hardware, avoiding costly over/under-provisioning.

Local inference setup on existing hardware

One-command discovery of the best available model for your current machine (with auto-detection of VRAM, GPU type, CPU cores, RAM). Includes instant chat and Python snippet generation for rapid prototyping.

Multi-GPU workstation and cluster planning

Compare scaling scenarios with multi-GPU simulation (e.g., `2x RTX 4090`), reverse-lookup what hardware is needed for a specific model, and generate reproducible recommendations for infrastructure decisions.

Implementation considerations

Python 3.11+ is required; install via `pip`, Homebrew, or `uv tool install` for isolated tool environments. One-off runs use `uvx whichllm@latest` without local installation.
Benchmark data is fetched from HuggingFace; cached locally for offline reuse. If recommendations need to stay current, periodic updates via `uv tool upgrade` or `pip install --upgrade` are advisable.
VRAM estimates include GQA KV cache, activation memory, and quantization overhead; actual runtime may vary by backend (llama-cpp-python, transformers, etc.) and context length used.
Speed estimates are architecture-aware (MoE active params, bandwidth-bound throughput) but are not measured on your target hardware. Validate with actual benchmark runs before committing to production inference.
GPU simulation (e.g., `--gpu "RTX 4090"`) assumes standard VRAM and bandwidth; mobile or low-power variants (RTX 4060 Mobile, etc.) may have different real-world behavior.

When to avoid it — and what to weigh

You need offline-first inference with no internet at all — whichllm fetches live HuggingFace model data. It has cached fallbacks, but initial use and updates require network access. For fully air-gapped setups, a local model registry is required.
You require proprietary closed-source model recommendations — whichllm ranks only models published on HuggingFace. If your workflow relies on proprietary APIs (OpenAI, Anthropic, etc.) or private fine-tunes, this tool is not applicable.
You need real-time inference latency SLAs or P99 guarantees — whichllm provides estimated speed (tokens/sec) based on theoretical bandwidth and published benchmarks, not measured on your actual hardware. Production latency SLAs require deployed performance testing.
Your hardware is non-standard or custom-accelerated — Auto-detection supports common NVIDIA, AMD, Intel, and Apple Silicon. Custom TPUs, Cerebras, or other accelerators may not be recognized; manual VRAM and bandwidth overrides are required.

License & commercial use

MIT License (Expat). Permissive; allows commercial use, modification, distribution, and private use with no warranty. Attribution appreciated but not legally required. Full license text at https://opensource.org/licenses/MIT.

MIT License explicitly permits commercial use (redistributing whichllm, using its output for paid services, selling recommendations, etc.) without restriction. However, whichllm recommends third-party models from HuggingFace; verify the license of each recommended model (many are permissive, some are not) before commercial deployment. No support SLA or liability provided by MIT.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Strong
License clarity	Clear
Deployment complexity	Low
DEV.co fit	Good
Assessment confidence	High

Security considerations

Fetches model metadata from HuggingFace public API; no authentication or credential handling in whichllm itself. The `whichllm run` subcommand downloads model files from HuggingFace (verify model source and integrity before use in sensitive environments). Standard Python package supply-chain risk applies (transitive dependencies on llama-cpp-python, transformers, etc.); use lock files and SBOMs in regulated environments. No persistent data collection or telemetry evident from README.

Alternatives to consider

LM Studio

GUI-based local inference with model discovery and simple VRAM/speed heuristics. No CLI scripting; more user-friendly for non-technical users but less data-driven for optimization decisions.

Ollama

Simplified model downloading and local inference service. Smaller model library and less benchmark-driven ranking; better for quick-start, worse for hardware planning.

vLLM + HuggingFace Hub

Production inference engine with batching and throughput optimization. Requires deployment infrastructure; no high-level recommendation or hardware simulation, but more suitable for serving workloads.

Software development agency

Build on whichllm with DEV.co software developers

Try whichllm in seconds: `pip install whichllm && whichllm`. Get hardware-aware model recommendations ranked by real benchmarks, not just size.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

vllm airllm nano-vllm

Related on DEV.co

Explore the category and the services that help you build with it.

AI Frameworks AI Application Development API Development Talk to DEV.co

whichllm FAQ

Does whichllm download and run models automatically?

Not by default. `whichllm` prints recommendations. `whichllm run <model>` downloads and starts an interactive chat. `whichllm snippet <model>` generates Python code for you to integrate. You control when models are fetched.

How do I simulate a different GPU before buying?

Use `whichllm --gpu "GPU_NAME"`, e.g., `whichllm --gpu "RTX 4090"` or `whichllm --gpu "2x RTX 4090"`. Estimates are based on published VRAM and bandwidth, not actual deployment.

Are the speed estimates accurate?

Estimates are architecture-aware (bandwidth, quantization, MoE) but theoretical. Actual tokens/sec depend on backend (llama-cpp-python vs transformers), context length, and hardware variability. Use estimates for comparative planning; validate with real benchmarks before production.

Can I use whichllm in a CI/CD pipeline?

Yes. Use `--json` output and pipe to jq or your orchestration tool. Example: `whichllm --top 1 --json | jq .recommendations[0].model_id` for scripted model selection.

Software development & web development with DEV.co

DEV.co helps companies turn open-source tools like whichllm into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your ai frameworks stack.

Ready to find your optimal local LLM?

Try whichllm in seconds: `pip install whichllm && whichllm`. Get hardware-aware model recommendations ranked by real benchmarks, not just size.

Talk to DEV.co Schedule a Consultation