Vector Databases · lakesoul-io

LakeSoul

LakeSoul is an open-source lakehouse platform that combines data ingestion, storage, and querying with built-in support for streaming updates, ACID transactions, and vector search. It provides a production-ready alternative to assembling separate components, with a Rust-native core that works consistently across Spark, Flink, Ray, Daft, and other compute engines.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/lakesoul-io/LakeSoul

3.2k

GitHub stars

419

Forks

Java

Primary language

Apache-2.0

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	lakesoul-io/LakeSoul
Owner	lakesoul-io
Primary language	Java
License	Apache-2.0 — OSI-approved
Stars	3.2k
Forks	419
Open issues	17
Latest release	v3.0.0 (2025-09-05)
Last updated	2026-07-08
Source	https://github.com/lakesoul-io/LakeSoul

What LakeSoul is

LakeSoul implements a lakehouse architecture with Rust-based metadata management and IO layers, PostgreSQL-backed ACID control, LSM-Tree upsert semantics for hash-partitioned tables with primary keys, and multi-level compaction. It supports Parquet and Vortex file formats, offers bindings for Java/Python/C++, and integrates with Spark 3.5, Flink 1.20, Presto (Velox), Ray 2.55, Daft 0.7+, DuckDB, PyArrow, and Pandas.

Quickstart

Get the LakeSoul source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/lakesoul-io/LakeSoul.git2cd LakeSoul3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Real-time Data Warehouse with CDC Ingestion

Flink CDC connectors enable end-to-end synchronization of relational databases into LakeSoul with exactly-once guarantees, auto-schema evolution, and concurrent streaming writes. Supports both row-level upserts and delete semantics without manual join logic.

AI/ML Data Pipelines with Unified Storage

Native PyTorch Dataset, Ray, and Daft integrations allow training and distributed processing directly on lakehouse data. Python-only readers eliminate PySpark dependency; Vortex format stores multimodal embeddings and vector data with built-in vector search.

High-Throughput Streaming Analytics with Incremental Queries

LSM-Tree upsert structure and merge-on-read optimizations support sustained write throughput while incremental query APIs enable low-latency analytics. Concurrent batch and streaming reads share the same ACID guarantees and snapshot isolation.

Implementation considerations

PostgreSQL metadata store is mandatory and must be provisioned, maintained, and backed up; no in-process or SQLite fallback option.
Java/Scala primary implementation requires JVM infrastructure; Python and Rust bindings are secondary and may lag in feature parity.
LSM-Tree compaction is automated but requires tuning of compaction strategies (leveled vs. size-tiered) for your write patterns and query performance targets.
S3 proxy-based RBAC requires additional proxy service deployment for fine-grained access control; Postgres RBAC alone applies only to metadata.
Vortex format adoption for multimodal data is optional but necessary for vector search and embedding storage; Parquet is the default fallback.

When to avoid it — and what to weigh

Simple Append-Only Data Lake — If your workload is pure append-only with no updates, upserts, or complex schema evolution, the complexity of LSM-Tree management and PostgreSQL metadata overhead may be unnecessary.
Minimal DevOps Capacity — LakeSoul requires operational management of PostgreSQL for metadata, object store (S3/HDFS), and compute engine clusters (Spark, Flink). Organizations without dedicated data infra teams will face higher operational burden.
Strict Multi-Cloud or Vendor Lock-in Concerns — LakeSoul is deeply integrated with S3 and PostgreSQL. If you require easy portability across cloud vendors or have strict anti-vendor-lock policies, the coupling may be problematic.
Mature Ecosystem Lock-in (Iceberg/Delta) — If your organization has standardized on Apache Iceberg or Delta Lake with existing tools and expertise, switching to LakeSoul's proprietary table format and ecosystem introduces re-training and migration costs.

License & commercial use

LakeSoul is licensed under Apache License 2.0 (Apache-2.0), a permissive OSI-approved open-source license. No proprietary or commercial license variants are mentioned in the provided data.

Apache 2.0 permits commercial use, including proprietary modifications and distribution, provided you retain license notices and copyright attributions. However, the provided data does not document commercial support, SLAs, or vendor indemnification. Requires independent review of Apache 2.0 obligations and any commercial support offerings from project maintainers.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Strong
License clarity	Clear
Deployment complexity	High
DEV.co fit	Good
Assessment confidence	High

Security considerations

PostgreSQL stores metadata; password/credential management and network isolation are required. S3 proxy layer provides table-level RBAC but adds operational attack surface. Postgres row-level security policies enforce metadata isolation. No security audit report or formal threat model provided in the data. Object store access control depends on S3 IAM and proxy verification; misconfigurations could allow unauthorized data access. TLS/encryption at rest/in transit are not explicitly mentioned.

Alternatives to consider

Apache Iceberg

De-facto open table format with broader ecosystem adoption (Spark, Flink, Presto, Trino, Duckdb). Upsert support is newer (v1.3+) and less mature than LakeSoul's LSM-Tree approach. Requires separate compaction and catalog services but offers more vendor flexibility.

Delta Lake

Proprietary Parquet-based format with strong Spark integration, ACID via Delta Protocol, and Z-order indexing. Simpler operational model for Spark-dominant workloads. Upsert performance may differ; ecosystem tools tighter to Databricks.

Apache Druid

Time-series OLAP database with native upserts and vector capabilities. Better suited for metric/analytics queries at scale; not a general-purpose lakehouse. Requires different operational model and compute resource allocation.

Software development agency

Build on LakeSoul with DEV.co software developers

LakeSoul is ideal for real-time data warehouses, AI/ML pipelines, and streaming analytics. However, it requires PostgreSQL, careful operational planning, and deep familiarity with distributed systems. Schedule a technical review to assess fit for your workload, team capacity, and compute engine stack.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

spark VectorDBBench sail

Related on DEV.co

Explore the category and the services that help you build with it.

Vector Databases AI Model Development & Data Pipelines Cloud Data Infrastructure & DevOps Enterprise Data Platform Engineering Talk to DEV.co

LakeSoul FAQ

Can I use LakeSoul without PostgreSQL?

No. PostgreSQL is mandatory for metadata management and ACID control. The data provided does not mention in-process or alternative metadata stores.

Does LakeSoul support Spark 4.0 or Flink 2.0?

Not yet. The roadmap lists Spark 4.0+ and Flink 2.0+ as planned (2026), but the current support matrix shows Spark 3.5 and Flink 1.20. Verify compatibility with your engine version.

Is LakeSoul production-ready?

Yes, it is described as production-ready with ACID guarantees, exactly-once streaming, and concurrent updates. However, operational complexity (PostgreSQL, compaction tuning, S3 proxy) and reliance on newer compute engine versions mean careful testing and capacity planning are required.

How does LakeSoul compare to Iceberg for upserts?

LakeSoul uses LSM-Tree structure for efficient row-level upserts and high write throughput on hash-partitioned tables. Iceberg uses copy-on-write or merge-on-read; upsert support is newer and less optimized. LakeSoul may offer better write performance but less ecosystem flexibility.

Work with a software development agency

DEV.co helps companies turn open-source tools like LakeSoul into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your vector databases stack.

Evaluate LakeSoul for Your Data Architecture

Talk to DEV.co Schedule a Consultation