Open-Source Databases · sfu-db

connector-x

ConnectorX is a Rust-based library that rapidly loads data from multiple SQL databases (Postgres, MySQL, Oracle, BigQuery, etc.) into Python DataFrames with minimal memory overhead. It achieves 3–21× faster load times and 3× lower memory usage than pandas by using zero-copy principles and parallel partition-based downloads.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/sfu-db/connector-x

2.6k

GitHub stars

217

Forks

Rust

Primary language

MIT

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	sfu-db/connector-x
Owner	sfu-db
Primary language	Rust
License	MIT — OSI-approved
Stars	2.6k
Forks	217
Open issues	225
Latest release	v0.4.5 (2026-01-18)
Last updated	2026-06-29
Source	https://github.com/sfu-db/connector-x

What connector-x is

Written in Rust with Python bindings, ConnectorX reads SQL query results via multi-threaded partition-level downloads (optional), allocates memory once based on schema introspection and row counts, and writes data directly to destination formats (Pandas, PyArrow, Polars, Dask, Modin) without intermediate copies. It supports federated queries across multiple databases and handles numerical column partitioning for SPJA queries.

Quickstart

Get the connector-x source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/sfu-db/connector-x.git2cd connector-x3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Large-scale data warehouse extraction

Loading multi-gigabyte tables from Postgres, Redshift, or BigQuery into analytical DataFrames where memory efficiency and speed are critical; parallel partition-based downloads scale well with CPU cores.

Data pipeline initialization in production services

Python-based ETL or data science workflows that require fast, predictable ingestion of structured SQL data without adding significant memory overhead or processing latency.

Cross-database federated queries

Joining and loading data from two or more databases in a single query, useful for data consolidation or ad-hoc analytics across heterogeneous database systems.

Implementation considerations

Partition column selection must be numerical and non-NULL to enable parallel reads; poorly chosen or skewed partition columns may degrade performance.
Federated query support is marked experimental; test cross-database joins thoroughly before production deployment to confirm correctness and performance.
Connection string format and credential handling vary by database type; review documentation for each source (Postgres, MySQL, Oracle, BigQuery, etc.).
Memory allocation is based on schema introspection and row counts via COUNT queries; very large result sets or dynamic queries may require tuning.
Output format selection (Pandas, PyArrow, Polars) affects downstream compatibility; PyArrow offers the broadest interoperability but may have different null/type semantics than Pandas.

When to avoid it — and what to weigh

Unstructured or semi-structured data — ConnectorX is designed for SQL databases with fixed schemas; it is not suitable for document stores, file systems, or JSON-based NoSQL sources.
Real-time streaming ingestion — This is a batch-oriented tool for extracting snapshots of SQL query results; it does not support continuous streaming or change-data-capture patterns.
Partitioning on non-numerical or nullable columns — Parallel partition-based optimization only works on numerical columns without NULL values; other data types must use non-partitioned single-threaded reads.
Minimal Rust/Python integration overhead required — The library relies on PyO3 Python bindings; if C/Rust interop or compilation is a concern, evaluate build-time and runtime dependency costs.

License & commercial use

MIT License. Permissive OSI-approved license allowing commercial use, modification, and distribution with minimal restrictions.

MIT license permits commercial use without formal approval or negotiation. No copyleft obligations or proprietary licensing fees. For production deployments, verify that Rust dependencies (transitive via PyPI wheel) meet your compliance requirements, but the library itself imposes no commercial restrictions.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Adequate
License clarity	Clear
Deployment complexity	Low
DEV.co fit	Strong
Assessment confidence	High

Security considerations

No security audit or hardening details provided in available data. Standard considerations: credentials are passed in connection strings (use secure credential management); SQL injection depends on parameterization in user queries; no encryption or TLS enforcement described. Rust's memory safety mitigates buffer overflows. Review transitive Rust dependencies for known vulnerabilities before production use.

Alternatives to consider

Pandas read_sql()

Built-in, zero dependencies, but significantly slower (13–21× slower) and higher memory overhead; suitable only for small datasets or when ConnectorX unavailable.

SQLAlchemy + Pandas

More flexible ORM abstraction, supports diverse databases, but layered abstraction adds overhead; slower than ConnectorX for bulk data loads.

Polars native SQL reader (Polars 0.20+)

Native SQL support in Polars reduces external dependency; however, ConnectorX offers direct Polars output and proven benchmarks; both are valid depending on ecosystem preference.

Software development agency

Build on connector-x with DEV.co software developers

ConnectorX integrates with Python workflows and reduces load time and memory overhead. Our team can help you evaluate database sources, optimize partition strategies, and integrate ConnectorX into your ETL or analytics infrastructure.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

dbx sqlx snowflake-connector-python

Related on DEV.co

Explore the category and the services that help you build with it.

Open-Source Databases Data Engineering & ETL Development Cloud Deployment & Infrastructure Custom Software Development Talk to DEV.co

connector-x FAQ

Does ConnectorX support my database?

Supported: Postgres, MySQL, MariaDB, SQLite, Redshift, ClickHouse, SQL Server, Azure SQL, Oracle, BigQuery, Trino. ODBC is in progress. Check the official database docs for exact protocol versions and connection string formats.

How much faster is it compared to Pandas?

In the provided benchmark (10× TPC-H lineitem, 8.6GB, 4-core Postgres): ConnectorX is 13–21× faster and uses 3× less memory than Pandas read_sql(). Actual speedups vary by database, query complexity, and hardware.

Do I need to partition my queries manually?

No. Partitioning is optional. Specify partition_on=<numerical_column> and partition_num=<count>; ConnectorX auto-splits the query. Without it, data loads in a single thread.

Is federated query support production-ready?

Marked experimental in the README. It works for joining tables across two or more databases, but thoroughly test cross-database joins before production use; correctness and performance are not yet guaranteed.

Software development & web development with DEV.co

DEV.co helps companies turn open-source tools like connector-x into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your open-source databases stack.

Ready to accelerate your data pipelines?

Talk to DEV.co Schedule a Consultation