Open-Source Databases · toluaina

pgsync

PGSync is a real-time change data capture tool that automatically syncs data from PostgreSQL, MySQL, or MariaDB into Elasticsearch or OpenSearch. You define your document structure in JSON, and the tool handles denormalization, incremental updates, and consistency without requiring custom ETL code.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/toluaina/pgsync

1.4k

GitHub stars

210

Forks

Python

Primary language

MIT

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	toluaina/pgsync
Owner	toluaina
Primary language	Python
License	MIT — OSI-approved
Stars	1.4k
Forks	210
Open issues	208
Latest release	7.1.0 (2026-06-25)
Last updated	2026-06-29
Source	https://github.com/toluaina/pgsync

What pgsync is

PGSync leverages PostgreSQL logical replication (WAL) or MySQL/MariaDB binary logging to detect changes, auto-generates optimized SQL JOINs from schema definitions, and bulk-indexes denormalized documents into Elasticsearch/OpenSearch with checkpoint-based fault tolerance. Python-based, requires Redis for state management (optional in WAL mode).

Quickstart

Get the pgsync source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/toluaina/pgsync.git2cd pgsync3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Real-time search on relational data

Keep Elasticsearch/OpenSearch in sync with relational source of truth, enabling fast full-text and faceted search on complex normalized schemas without manual ETL maintenance.

Denormalization at scale

Automatically flatten multi-table hierarchies (book → authors, products → inventory) into search-ready JSON documents with nested arrays, updated atomically on any relation change.

Operational analytics dashboards

Power Kibana or OpenSearch dashboards with live data from production databases, handling schema evolution and incremental updates with minimal database overhead.

Implementation considerations

PostgreSQL requires `wal_level=logical` and replication slot configuration; MySQL/MariaDB require binary logging and replication user setup—plan 2–4 hours for database enablement.
Schema definition in JSON is straightforward but requires careful mapping of parent–child relationships and column selection; test with production schema complexity first.
Redis is needed for state/checkpoint management (or file-based fallback); plan for Redis HA/sentinel in production to avoid losing sync position.
Initial full sync of large tables (millions of rows) may take hours; test with production data volumes and tune bulk batch sizes.
Transforms (replace, rename, concat) are applied at index time; complex business logic may require custom post-processing or Elasticsearch ingest pipelines.

When to avoid it — and what to weigh

Requires sub-second latency guarantees — PGSync is near real-time but not guaranteed microsecond delivery; network, Redis, and Elasticsearch commit latencies apply. Not suitable for high-frequency trading or hard real-time systems.
Need to sync multiple databases or complex multi-source workflows — PGSync is single-source CDC; if you need to aggregate data from disparate systems or handle complex ETL workflows, consider Kafka, Debezium, or purpose-built data orchestration platforms.
Elasticsearch/OpenSearch is not your search engine — PGSync is hardcoded for Elasticsearch/OpenSearch targets. If your search layer is Solr, Algolia, Meilisearch, or in-database full-text search, this tool adds no value.
Limited DevOps/Kubernetes operational readiness — Project has 208 open issues, recent activity, and MIT license clarity, but unknown production deployment best practices, SLA expectations, and Kubernetes Operator maturity.

License & commercial use

MIT License. Permissive OSI-approved license allowing commercial use, modification, and redistribution with attribution. No copyleft restrictions.

MIT license permits commercial use without restrictions. However, the project is community-maintained; no commercial support, SLA, or warranty is provided by the maintainer. Evaluate your risk tolerance for production use without paid support contracts.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Adequate
License clarity	Clear
Deployment complexity	Moderate
DEV.co fit	Good
Assessment confidence	High

Security considerations

No exploit details provided. General considerations: PGSync runs as a daemon with database credentials and Elasticsearch auth—store secrets (PG_URL, ELASTICSEARCH_URL) in vault/secrets manager, not environment files. Ensure network policies restrict database and search engine access. MySQL/MariaDB replication user should be scoped to minimal privileges. No third-party security audit data available; review dependencies (psycopg2, pymysql, redis client) for CVEs before production use.

Alternatives to consider

Debezium + Kafka Connect

Enterprise-grade CDC platform with pluggable sinks, higher fault tolerance, multi-database support, and active Confluent backing. Higher operational complexity and resource footprint.

Elasticsearch Connector for Postgres (wal2json-based)

Lower-level WAL-native syncing with fine-grained control; requires more manual pipeline orchestration but avoids Redis dependency. Steeper learning curve.

Custom application-level dual-write + queue

Software development agency

Build on pgsync with DEV.co software developers

Evaluate PGSync for your project or consult with our team on deployment architecture, database prerequisite setup, and scaling strategies for production use.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

sqlsync jasync-sql node-pg-migrate

Related on DEV.co

Explore the category and the services that help you build with it.

Open-Source Databases API Development DevOps Cloud Deployment Talk to DEV.co

pgsync FAQ

Does PGSync require downtime to set up?

No. Enable logical replication (PostgreSQL) or binary logging (MySQL/MariaDB) on the live database without stopping services. PGSync can bootstrap from a snapshot and then consume WAL/binlog incrementally. Plan a few hours for database config change and replication user setup.

What happens if PGSync crashes or network drops?

PGSync stores LSN/binlog position in Redis (or file-based state). On restart, it resumes from the last checkpoint and replays changes. If Redis is lost and no backup exists, a full re-sync is required; plan for persistent Redis in production.

Can I sync to multiple Elasticsearch clusters simultaneously?

Not natively. PGSync targets a single Elasticsearch/OpenSearch URL per instance. Run multiple PGSync instances (with separate configs) if you need multi-cluster replication, or use Elasticsearch cross-cluster replication as a post-layer.

Is PGSync suitable for GDPR/right-to-be-forgotten compliance?

PGSync syncs deletions from the source database to Elasticsearch, but you must ensure your Elasticsearch retention and deletion policies align with regulations. PGSync does not provide audit trails or soft-delete handling; implement deletion workflows in your application layer.

Work with a software development agency

DEV.co is a software development agency delivering custom software development services to companies building on open source. Our software developers and web developers design, integrate, and ship production systems — spanning web development, APIs, AI, data, and cloud. If pgsync is part of your open-source databases roadmap, our team can implement, customize, migrate, and maintain it.

Ready to sync your database to Elasticsearch?

Evaluate PGSync for your project or consult with our team on deployment architecture, database prerequisite setup, and scaling strategies for production use.

Talk to DEV.co Schedule a Consultation