AI Frameworks · microsoft

aici

AICI is a Microsoft Research framework that lets you write programs (as WebAssembly modules) to control and constrain LLM output in real time. Controllers can enforce structured outputs, implement multi-agent flows, or apply custom decoding logic while the GPU generates tokens, with minimal overhead.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/microsoft/aici

2.1k

GitHub stars

Forks

Rust

Primary language

MIT

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	microsoft/aici
Owner	microsoft
Primary language	Rust
License	MIT — OSI-approved
Stars	2.1k
Forks	84
Open issues	41
Latest release	v0.2.1 (2024-04-29)
Last updated	2025-01-22
Source	https://github.com/microsoft/aici

What aici is

AICI provides a sandboxed Wasm runtime for token-by-token controller logic that runs on CPU during GPU inference. Controllers written in Rust, C++, Python, or JavaScript can guide generation through token constraints, prompt editing, and state management. Currently integrates with llama.cpp, HuggingFace Transformers, and rLLM; vLLM planned.

Quickstart

Get the aici source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/microsoft/aici.git2cd aici3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Constrained decoding and structured output

Enforce JSON schemas, regex patterns, or domain-specific grammars on LLM output without re-prompting or post-processing, reducing latency and improving consistency.

Real-time guided generation and interactive control

Implement dynamic prompt editing, multi-turn orchestration, or interactive steering during token generation by maintaining state and making token-level decisions.

Latency-sensitive inference pipelines

Offload control logic to CPU while GPU generates, leveraging parallel execution to minimize overhead compared to post-processing or re-generation pipelines.

Implementation considerations

Dev environment requires Rust, rustup, wasm32-wasi target, and C/C++ build tools (cmake, ccache, clang). Windows users must use WSL2 or devcontainer. CUDA builds require fiddly libtorch setup.
Controllers are deployed as Wasm modules; pyctrl and jsctrl can execute Python/JavaScript code inline, but production controllers require compilation and versioning strategy.
Integration is engine-specific. For llama.cpp use rllm-llamacpp backend; CUDA-based inference (A100, RTX 30x0+) requires rllm-cuda. vLLM support is stated as 'in the works'.
Performance gains come from parallel CPU/GPU execution, but token-by-token synchronization and state overhead are not quantified. Requires profiling for your workload.
Sandboxing is claimed but not formally verified. Controllers cannot access filesystem/network per design, but security review is recommended before running untrusted controllers.

When to avoid it — and what to weigh

You need production-grade stability and support — AICI is explicitly described as a prototype. No SLA, no official support channel, and active issues (41 open) suggest ongoing development. Not recommended for mission-critical deployments without internal resources.
Your inference engine is not supported — Requires tight integration with the LLM engine (llama.cpp, HF Transformers, rLLM, or future vLLM). If you use proprietary or niche serving platforms, AICI cannot integrate.
You need immediate production integrations — Latest release is v0.2.1 (April 2024). While actively maintained (last push Jan 2025), the API and supported backends are still evolving. No guarantee of backward compatibility.
Your team cannot compile or debug Wasm — Controllers must compile to Wasm or run interpreted inside Wasm. Development requires Rust toolchain setup, Wasm target, and debugging skills. Significant setup and learning curve.

License & commercial use

MIT License. Permissive OSI-approved license. Permits commercial use, modification, and distribution with attribution; no warranty. Clear licensing from Microsoft Research.

MIT License explicitly permits commercial use. However, AICI is a prototype with no official support, SLA, or maintenance guarantees from Microsoft. Recommend embedding only if you can maintain and debug Wasm controllers and rLLM integration independently, and assume operational risk.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Adequate
License clarity	Clear
Deployment complexity	High
DEV.co fit	Possible
Assessment confidence	Medium

Security considerations

Controllers run in Wasm sandbox with no filesystem, network, or OS access by design. However: (1) security claims are stated but not independently verified; (2) formal threat model or audit is not documented; (3) Wasm runtime (Wasmtime) is external; (4) rLLM server exposes HTTP interface—network isolation and authentication are not discussed; (5) untrusted controllers should be reviewed before deployment. Recommendation: do not assume 'secure by default' without internal security review.

Alternatives to consider

LLGuidance

Actively maintained evolution of AICI, specifically optimized for constrained decoding. If you only need structured output (JSON, regex, grammars), it may reduce complexity and integration burden. Recommended by AICI maintainers for this use case.

Guidance or LMQL (via other backends)

High-level DSLs for controlling LLM generation. Run on top of various inference engines without tight coupling. Less low-level control than AICI but higher portability and simpler authoring model.

vLLM + response format constraints

vLLM (mainstream, production-grade) natively supports constrained decoding via logit processors. No Wasm requirement, tighter coupling with vLLM, but less flexibility for custom multi-agent or dynamic prompt editing logic.

Software development agency

Build on aici with DEV.co software developers

Explore AICI if you need fine-grained control over LLM output and can invest in Wasm controller development. For constrained decoding alone, consider LLGuidance. Contact us to evaluate integration feasibility for your inference stack.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

aiconfig aichat AGiXT

Related on DEV.co

Explore the category and the services that help you build with it.

AI Frameworks AI application development API development Custom software Talk to DEV.co

aici FAQ

Is AICI production-ready?

No. Explicitly described as a prototype from Microsoft Research. No SLA, support, or maintenance guarantees. Use only if you can own and maintain the Wasm runtime and rLLM integration internally.

Can I use my existing vLLM or llama-cpp-python setup?

Partially. AICI requires integration via rLLM server, not direct library calls. vLLM support is listed as 'in the works.' For llama.cpp, use the rllm-llamacpp backend. Expect setup and bridging work.

What languages can I write controllers in?

Rust, C, C++, or any language compiling to Wasm. Alternatively, pyctrl and jsctrl allow Python or JavaScript code to run interpreted inside Wasm, with less performance optimization.

How much overhead does AICI add to generation latency?

Documentation claims 'minimal overhead' because controllers run on CPU during GPU token generation (parallel). However, no benchmarks, profiling results, or latency numbers are published. Requires profiling on your hardware and workload.

Work with a software development agency

Adopting aici is usually one piece of a larger software development effort. As a software development agency, DEV.co provides software development services and web development expertise — pairing senior software developers and web developers with your team to design, build, and operate ai frameworks software in production.

Build Constrained LLM Pipelines with AICI

Talk to DEV.co Schedule a Consultation