{"id":29354266,"url":"https://github.com/erans/pgsqlite","last_synced_at":"2026-05-06T08:31:50.082Z","repository":{"id":301992954,"uuid":"1010845961","full_name":"erans/pgsqlite","owner":"erans","description":"A PostgreSQL protocol adapter for SQLite databases. This project allows PostgreSQL clients to connect to and query SQLite databases using the PostgreSQL wire protocol.","archived":false,"fork":false,"pushed_at":"2026-03-28T03:33:51.000Z","size":3057,"stargazers_count":101,"open_issues_count":0,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-28T07:40:45.531Z","etag":null,"topics":["postgres","postgresql","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/erans.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-29T23:09:50.000Z","updated_at":"2026-03-28T03:33:53.000Z","dependencies_parsed_at":"2025-06-30T02:25:20.115Z","dependency_job_id":"2232c51d-1c29-4c01-92e2-29e148ae41a2","html_url":"https://github.com/erans/pgsqlite","commit_stats":null,"previous_names":["erans/pgsqlite"],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/erans/pgsqlite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Fpgsqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Fpgsqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Fpgsqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Fpgsqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/erans","download_url":"https://codeload.github.com/erans/pgsqlite/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Fpgsqlite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32684605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-06T02:33:58.958Z","status":"ssl_error","status_checked_at":"2026-05-06T02:33:39.611Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["postgres","postgresql","sqlite"],"created_at":"2025-07-09T03:12:04.400Z","updated_at":"2026-05-06T08:31:50.075Z","avatar_url":"https://github.com/erans.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgsqlite\n## 🐘+🪶=\u003cspan style=\"color: red; font-size: 1.5em\"\u003e♥\u003c/span\u003e\n\n\u003cimg width=\"150\" src=\"./pgsqlite.png\"/\u003e\n\n**PostgreSQL protocol for SQLite databases.** Turn any SQLite database into a PostgreSQL server that your existing tools and applications can connect to.\n\n\u003e **⚠️ WARNING: Experimental Project**\n\u003e This is an experimental project and is not yet ready for production use. It is under active development and may contain bugs, incomplete features, or breaking changes.\n\n## Why pgsqlite?\n\n**pgsqlite** lets you use PostgreSQL tools and libraries with SQLite databases. This is perfect for:\n\n- **🚀 Rapid Testing \u0026 CI/CD**: Run integration tests without spinning up PostgreSQL. Just copy your SQLite file and go.\n- **🌿 Feature Branch Deployments**: Each branch gets its own database. Just copy the SQLite file - no complex database provisioning.\n- **🤖 AI Agent Development**: Multiple agents can work on isolated sandbox environments with zero setup. Clone the database instantly.\n- **💻 Local Development**: Use your favorite PostgreSQL tools (psql, pgAdmin, DataGrip) with lightweight SQLite storage.\n- **🔧 Migration Path**: Prototype with SQLite, seamlessly move to PostgreSQL later without changing application code.\n\n## Quick Start\n\n### Installation\n\n**Option 1: Download Pre-built Binaries (Recommended)**\n\nVisit the [GitHub Releases page](https://github.com/erans/pgsqlite/releases) to download the latest pre-built binary for your platform:\n\n```bash\n# Example for Linux x64:\nwget https://github.com/erans/pgsqlite/releases/latest/download/pgsqlite-linux-x64.tar.gz\ntar -xzf pgsqlite-linux-x64.tar.gz\nchmod +x pgsqlite\n./pgsqlite\n```\n\n**Option 2: Build from Source**\n\n```bash\n# Clone and build from source\ngit clone https://github.com/erans/pgsqlite\ncd pgsqlite\ncargo build --release\n./target/release/pgsqlite\n```\n\n### Basic Usage\n\n1. **Start pgsqlite with a SQLite database:**\n```bash\n# Use an existing SQLite database\npgsqlite --database ./my-database.db\n\n# Or start with an in-memory database for testing\npgsqlite --in-memory\n```\n\n2. **Connect with any PostgreSQL client:**\n```bash\n# Using psql\npsql -h localhost -p 5432 -d my-database\n\n# Using connection string\npsql \"postgresql://localhost:5432/my-database\"\n```\n\n3. **Use it just like PostgreSQL:**\n```sql\n-- Create tables with PostgreSQL syntax\nCREATE TABLE users (\n    id SERIAL PRIMARY KEY,\n    email VARCHAR(255) UNIQUE NOT NULL,\n    created_at TIMESTAMP DEFAULT NOW()\n);\n\n-- Insert data\nINSERT INTO users (email) VALUES ('user@example.com');\n\n-- Query with PostgreSQL functions\nSELECT * FROM users WHERE created_at \u003e NOW() - INTERVAL '7 days';\n```\n\n## Usage Examples\n\n### For Testing Environments\n\n```bash\n# Copy your template database for each test run\ncp template.db test-1.db\npgsqlite --database test-1.db --port 5433 \u0026\n\n# Run your tests against it\nnpm test -- --database-url postgresql://localhost:5433/test-1\n\n# Cleanup is just removing the file\nrm test-1.db\n```\n\n### For Feature Branch Deployments\n\n```bash\n# Each branch gets its own database copy\ncp main.db feature-branch-123.db\npgsqlite --database feature-branch-123.db --port 5433\n```\n\n### Connect from Your Application\n\n**Python (psycopg2):**\n```python\nimport psycopg2\nconn = psycopg2.connect(\n    host=\"localhost\",\n    port=5432,\n    database=\"myapp\"\n)\n```\n\n**Node.js (pg):**\n```javascript\nconst { Client } = require('pg')\nconst client = new Client({\n  host: 'localhost',\n  port: 5432,\n  database: 'myapp'\n})\n```\n\n**Any PostgreSQL-compatible ORM:** Works with SQLAlchemy, Django ORM, ActiveRecord, Prisma, etc.\n\n## Configuration\n\n### Essential Options\n\n```bash\n# Basic options\npgsqlite \\\n  --database \u003cpath\u003e     # SQLite database file (default: sqlite.db)\n  --port \u003cport\u003e         # PostgreSQL port (default: 5432)\n  --in-memory           # Use in-memory database\n\n# Security\npgsqlite \\\n  --ssl                 # Enable SSL/TLS encryption\n  --ssl-cert \u003cpath\u003e     # Custom SSL certificate\n  --ssl-key \u003cpath\u003e      # Custom SSL key\n\n# Performance\npgsqlite \\\n  --journal-mode WAL    # Enable WAL mode for better concurrency\n\n# Connection Pooling (for concurrent workloads)\nPGSQLITE_USE_POOLING=true pgsqlite \\\n  --database \u003cpath\u003e     # Enable read/write connection separation\n```\n\nFor all configuration options, see the [Configuration Reference](docs/configuration.md).\n\n## Features\n\n### PostgreSQL Compatibility\n\n- ✅ **Wire Protocol**: Full PostgreSQL v3 protocol implementation\n- ✅ **Clients**: Works with psql, pgAdmin, DBeaver, and all PostgreSQL drivers\n- ✅ **SQL Syntax**: Most PostgreSQL queries work without modification\n- ✅ **Data Types**: 40+ PostgreSQL types including SERIAL, JSON, UUID, arrays (stored as JSON)\n- ✅ **Transactions**: Full ACID compliance via SQLite\n\n### Notable Features\n\n- **Connection Pooling**: Optional read/write connection separation for improved concurrent performance (enabled via `PGSQLITE_USE_POOLING=true`)\n- **Query Optimization System**: Advanced optimization infrastructure with context merging, lazy schema loading, pattern recognition, and integrated optimization management\n- **PostgreSQL Functions**: Comprehensive function support including:\n  - **String Functions**: `split_part()`, `string_agg()`, `translate()`, `ascii()`, `chr()`, `repeat()`, `reverse()`, `left()`, `right()`, `lpad()`, `rpad()`\n  - **Math Functions**: `trunc()`, `round()`, `ceil()`, `floor()`, `sign()`, `abs()`, `mod()`, `power()`, `sqrt()`, `exp()`, `ln()`, `log()`, trigonometric functions, `random()`\n- **Array Types**: Full support for PostgreSQL arrays (e.g., `INTEGER[]`, `TEXT[][]`) with ARRAY literal syntax, ALL operator, and unnest() WITH ORDINALITY\n- **JSON Support**: Complete `JSON` and `JSONB` implementation with operators (`-\u003e`, `-\u003e\u003e`, `@\u003e`, `\u003c@`, `#\u003e`, `#\u003e\u003e`, `?`, `?|`, `?\u0026`) and functions (json_agg, json_object_agg, row_to_json, json_populate_record, json_to_record, jsonb_insert, jsonb_delete, jsonb_pretty, etc.)\n- **Full-Text Search**: Complete PostgreSQL FTS implementation with `tsvector`/`tsquery` types, `@@` operator, `to_tsvector()`, `to_tsquery()`, `plainto_tsquery()` functions using SQLite FTS5 backend\n- **ENUM Types**: `CREATE TYPE status AS ENUM ('active', 'pending', 'archived')`\n- **RETURNING Clauses**: `INSERT INTO users (email) VALUES ('test@example.com') RETURNING id`\n- **CTEs**: `WITH` and `WITH RECURSIVE` queries\n- **Generated Columns**: `SERIAL` and `BIGSERIAL` auto-increment columns\n- **VARCHAR/CHAR Constraints**: Length validation for `VARCHAR(n)` and `CHAR(n)` with proper padding\n- **NUMERIC/DECIMAL Constraints**: Precision and scale validation for `NUMERIC(p,s)` and `DECIMAL(p,s)`\n- **CREATE INDEX with Operator Classes**: Support for PostgreSQL operator classes like `varchar_pattern_ops`, `text_pattern_ops` (mapped to SQLite `COLLATE BINARY` for pattern matching optimization)\n- **psql Compatibility**: Enhanced psql support with `\\d`, `\\dt`, and `\\d tablename` commands fully working\n\n### Security Features\n\n- **Advanced SQL Injection Protection**: AST-based SQL parsing with fallback pattern detection\n  - Detects tautology attacks (e.g., `1=1`, `'a'='a'`)\n  - Prevents dangerous function execution (`exec`, `xp_cmdshell`)\n  - Blocks suspicious UNION operations with sensitive tables\n  - Limits multi-statement execution\n- **Security Audit Logging**: Comprehensive audit trail with configurable severity filtering\n- **Rate Limiting**: Built-in DoS protection with circuit breaker pattern\n- **Input Validation**: Protocol-level validation for all client inputs\n- **Memory Safety**: Rust's ownership system prevents buffer overflows and memory corruption\n\n### Limitations\n\n- ❌ Stored procedures and custom functions\n- ❌ PostgreSQL-specific system functions (`pg_*`)\n- ❌ Some advanced data types (ranges, geometric types)\n- ⚠️  Some advanced array features (array assignment operations, advanced indexing)\n- ❌ Multiple concurrent writers (SQLite allows only one writer at a time, mitigated by connection pooling for reads)\n\nFor detailed compatibility information, see [Type Mapping Documentation](docs/type-mapping-prd.md).\n\n## Performance Considerations\n\npgsqlite acts as a translation layer between PostgreSQL protocol and SQLite, providing full PostgreSQL compatibility with measurable overhead:\n\n### Real-World Performance (2025-09-20)\n\n**Driver Performance Comparison** (100 operations each):\n| Driver | SELECT | INSERT | UPDATE | DELETE | Best For |\n|--------|--------|--------|--------|--------|----------|\n| **psycopg3-binary** | 0.452ms | 0.976ms | 0.219ms | 0.176ms | **Read-heavy** workloads |\n| **psycopg3-text** | 0.925ms | 1.067ms | 0.304ms | 0.271ms | **Balanced** usage |\n| **psycopg2** | 2.939ms | 0.214ms | 0.089ms | 0.063ms | **Write-heavy** workloads |\n\n**Overhead vs Pure SQLite** (200 operations):\n- **Pure SQLite**: 44.4ms (0.22ms per operation) - Maximum speed\n- **pgsqlite**: ~16 seconds (~80ms per operation) - **~360x overhead**\n- **Trade-off**: Raw performance vs full PostgreSQL compatibility + ORM support\n\n### When pgsqlite is the Right Choice\n- **Web applications**: 80ms database operations feel instant to users\n- **ORM integration**: Django, SQLAlchemy, Rails, Ecto work seamlessly\n- **Development/testing**: Full PostgreSQL feature compatibility\n- **API endpoints**: Database time typically 10-20% of total request time\n\n### Performance Optimizations\n- **Protocol choice**: Binary mode (psycopg3-binary) is 3.1% faster than text mode\n- **Batch operations**: Multi-row INSERT provides dramatic improvements:\n  - 10-row batches: ~11x faster than single-row INSERTs\n  - 100-row batches: ~51x faster\n  - 1000-row batches: ~76x faster\n- **Connection architecture**: Connection-per-session provides excellent isolation\n- **Ultra-fast path**: Optimized execution for simple SELECT queries\n\nFor applications requiring microsecond-level performance, use pure SQLite. For PostgreSQL compatibility with acceptable overhead, pgsqlite is ideal.\n\n### Connection Pooling\n\nFor concurrent read-heavy workloads, enable connection pooling to improve performance:\n\n```bash\n# Enable connection pooling with default settings (5 connections)\nPGSQLITE_USE_POOLING=true pgsqlite --database mydb.db\n\n# Custom pool configuration\nPGSQLITE_USE_POOLING=true \\\nPGSQLITE_POOL_SIZE=10 \\\nPGSQLITE_POOL_TIMEOUT=60 \\\npgsqlite --database mydb.db\n```\n\n**When to use connection pooling:**\n- ✅ Multiple concurrent clients with read-heavy workloads\n- ✅ TCP connections with sustained connection patterns\n- ✅ Applications with frequent SELECT queries\n- ❌ Single-client applications or simple scripts\n- ❌ Memory-constrained environments\n- ❌ Unix socket connections with low concurrency\n\n**Configuration options:**\n- `PGSQLITE_POOL_SIZE` - Maximum connections in read pool (default: 5)\n- `PGSQLITE_POOL_IDLE_TIMEOUT` - Idle connection timeout in seconds (default: 300)\n- `PGSQLITE_POOL_HEALTH_INTERVAL` - Health check interval in seconds (default: 60)\n\nConnection pooling automatically routes SELECT queries to the read pool while directing write operations (INSERT/UPDATE/DELETE) to the primary connection for consistency.\n\n## Security Configuration\n\n### SQL Injection Protection\n\npgsqlite includes enterprise-grade SQL injection protection that's enabled by default:\n\n```bash\n# Protection is always active, but can be monitored via audit logging\nPGSQLITE_AUDIT_ENABLED=true \\\nPGSQLITE_AUDIT_LOG_QUERIES=true \\\npgsqlite --database mydb.db\n```\n\nThe protection system uses:\n- **AST-based analysis**: Parses SQL using PostgreSQL dialect for accurate threat detection\n- **Pattern matching**: Fallback detection for malformed queries\n- **Configurable limits**: Control statement counts, nesting depth, and UNION operations\n\n### Security Audit Logging\n\nTrack security events and potential threats:\n\n```bash\n# Enable comprehensive security auditing\nPGSQLITE_AUDIT_ENABLED=true \\\nPGSQLITE_AUDIT_SEVERITY=info \\\nPGSQLITE_AUDIT_LOG_AUTH=true \\\nPGSQLITE_AUDIT_LOG_QUERIES=true \\\nPGSQLITE_AUDIT_LOG_ERRORS=true \\\npgsqlite --database mydb.db\n```\n\nAudit events include:\n- Authentication attempts (success/failure)\n- SQL injection attempts with detailed analysis\n- Permission violations\n- Rate limit violations\n- System errors and anomalies\n\n### Rate Limiting \u0026 DoS Protection\n\nBuilt-in rate limiting prevents abuse:\n\n```bash\n# Configure rate limiting (default: 1000 req/sec per client)\nPGSQLITE_RATE_LIMIT_REQUESTS=1000 \\\nPGSQLITE_RATE_LIMIT_WINDOW=1 \\\nPGSQLITE_CIRCUIT_BREAKER_THRESHOLD=0.5 \\\npgsqlite --database mydb.db\n```\n\nFeatures:\n- Per-client IP rate limiting\n- Circuit breaker pattern for failing clients\n- Automatic recovery after cooldown periods\n- Memory-efficient sliding window algorithm\n\n## Advanced Topics\n\n- **[Security Guide](docs/security.md)**: Comprehensive security architecture and configuration\n- **[Schema Migrations](docs/migrations.md)**: Automatic migration system for pgsqlite metadata\n- **[SSL/TLS Setup](docs/ssl-setup.md)**: Secure connections configuration\n- **[Unix Sockets](docs/unix-sockets.md)**: Lower latency local connections\n- **[Performance Tuning](docs/performance-tuning.md)**: Cache configuration and optimization\n- **[Architecture Overview](docs/architecture.md)**: How pgsqlite works internally\n- **[Array Support](docs/array-support.md)**: Comprehensive guide to PostgreSQL arrays\n- **[JSON/JSONB Support](docs/json-support.md)**: Comprehensive guide to JSON functionality\n- **[Full-Text Search](docs/fts_implementation_plan.md)**: PostgreSQL FTS implementation details\n- **[Future Features](docs/future-features.md)**: Roadmap for enhanced SQLite capabilities\n\n## Development\n\n### Building from Source\n\n```bash\n# Clone the repository\ngit clone https://github.com/erans/pgsqlite\ncd pgsqlite\n\n# Build\ncargo build --release\n\n# Run tests\ncargo test\n\n# Run with debug logging\nRUST_LOG=debug ./target/release/pgsqlite\n```\n\n### Running Integration Tests\n\n```bash\n# Run all test suites (includes comprehensive JSON/array function testing)\n./tests/runner/run_ssl_tests.sh\n\n# Run specific test mode\n./tests/runner/run_ssl_tests.sh --mode tcp-ssl --verbose\n\n# Run unit tests\ncargo test\n```\n\nThe test suite includes comprehensive validation of all JSON and array functions across multiple connection modes (TCP with/without SSL, Unix sockets, file-based databases).\n\n### Contributing\n\nWe welcome contributions! When reporting issues, please include:\n\n1. The SQL query that caused the problem\n2. Expected behavior\n3. Actual behavior\n4. Any error messages\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for development guidelines.\n\n## Documentation\n\n- [Type Mapping Reference](docs/type-mapping-prd.md) - Detailed PostgreSQL to SQLite type mappings\n- [Configuration Reference](docs/configuration.md) - All configuration options\n- [Architecture Overview](docs/architecture.md) - Technical deep dive\n- [Performance Analysis](docs/performance.md) - Detailed benchmarks and optimization strategies\n\n## License\n\nThis project is licensed under the Apache License, Version 2.0 - see the [LICENSE](LICENSE) file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferans%2Fpgsqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ferans%2Fpgsqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferans%2Fpgsqlite/lists"}