{"id":30687637,"url":"https://github.com/vahid110/sqlxport","last_synced_at":"2025-09-02T00:04:41.280Z","repository":{"id":295629920,"uuid":"990590471","full_name":"vahid110/sqlxport","owner":"vahid110","description":"sql2parquet: A modern CLI tool to export SQL query results from PostgreSQL or Amazon Redshift directly to Parquet files, with optional upload to S3 or MinIO.","archived":false,"fork":false,"pushed_at":"2025-06-29T19:46:06.000Z","size":923,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-31T09:30:22.885Z","etag":null,"topics":["athe","aws","csv","data-export","delta-lake","duckdb","minio","parquet","redshift","s3","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vahid110.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-26T10:45:43.000Z","updated_at":"2025-07-14T21:32:44.000Z","dependencies_parsed_at":"2025-06-04T01:50:22.634Z","dependency_job_id":"56c27f7b-d8c3-4e41-9d5a-48b866496ef8","html_url":"https://github.com/vahid110/sqlxport","commit_stats":null,"previous_names":["vahid110/sql2parquet","vahid110/sql2data","vahid110/sqlxport"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/vahid110/sqlxport","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vahid110%2Fsqlxport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vahid110%2Fsqlxport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vahid110%2Fsqlxport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vahid110%2Fsqlxport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vahid110","download_url":"https://codeload.github.com/vahid110/sqlxport/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vahid110%2Fsqlxport/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273208777,"owners_count":25064204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["athe","aws","csv","data-export","delta-lake","duckdb","minio","parquet","redshift","s3","sql"],"created_at":"2025-09-02T00:03:42.024Z","updated_at":"2025-09-02T00:04:41.241Z","avatar_url":"https://github.com/vahid110.png","language":"Python","readme":"[![PyPI version](https://badge.fury.io/py/sqlxport.svg)](https://pypi.org/project/sqlxport/)\n![CI](https://github.com/vahid110/sqlxport/actions/workflows/ci.yml/badge.svg)\n[![Code Style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\".github/logo.png\" alt=\"SQLxport Logo\" width=\"200\"/\u003e\n\u003c/p\u003e\n\n# sqlxport\n\n**Modular CLI + API tool to extract data from PostgreSQL, Redshift, SQLite (and more), exporting to formats like Parquet/CSV, with optional S3 upload and Athena integration.**\n\n---\n\n## ✅ Features\n\n* 🔄 Run custom SQL queries against PostgreSQL, Redshift, SQLite\n* 📦 Export to Parquet or CSV (`--format`)\n* 🩣 Upload results to S3 or MinIO\n* 🔄 Redshift `UNLOAD` support (`--export-mode redshift-unload`)\n* 🧹 Partition output by column\n* 📜 Generate Athena `CREATE TABLE` DDL\n* 🔍 Preview local or remote Parquet/CSV files\n* ⚙️ `.env` support for convenient config\n* 🐍 Reusable Python API\n\n---\n\n## ❓ Why SQLxport?\n\nSQLxport simplifies data export workflows and is designed for automation:\n\n* ✅ One command gives you SQL → Parquet/CSV → S3\n* 🧱 Works locally, in CI, or inside Docker\n* 🪢 Connects to Athena, MinIO, Redshift easily\n* 🔌 Clean format and database plugin model\n* 🧪 Fully tested, scriptable, production-ready\n\n---\n\n## 📦 Installation\n\n```bash\npip install .\n# or for development\npip install -e .\n```\n\n---\n\n## 🚀 Usage\n\n### Choose Export Mode\n\n| `--export-mode`        | Compatible DB URLs               | Description              |\n|------------------------|----------------------------------|--------------------------|\n| `postgres-query`       | `postgresql://`, `postgres://`   | SELECT + local export   |\n| `redshift-unload`      | `redshift://`                    | UNLOAD to S3             |\n| `sqlite-query`         | `sqlite:///path.db`              | For local/lightweight testing |\n\n---\n\n### CLI Examples\n\n#### Basic Export\n\n```bash\nsqlxport run \\\n  --export-mode postgres-query \\\n  --db-url postgresql://user:pass@localhost:5432/mydb \\\n  --query \"SELECT * FROM users\" \\\n  --output-file users.parquet \\\n  --format parquet\n```\n\n#### S3 Upload\n\n```bash\nsqlxport run \\\n  --export-mode postgres-query \\\n  --db-url postgresql://... \\\n  --query \"...\" \\\n  --output-file users.parquet \\\n  --s3-bucket my-bucket \\\n  --s3-key users.parquet \\\n  --s3-access-key AKIA... \\\n  --s3-secret-key ... \\\n  --s3-endpoint https://s3.amazonaws.com\n```\n\n#### Partitioned Export\n\n```bash\nsqlxport run \\\n  --export-mode postgres-query \\\n  --db-url postgresql://... \\\n  --query \"...\" \\\n  --output-dir output/ \\\n  --partition-by group_column \\\n  --format csv\n```\n\n#### Redshift UNLOAD Mode\n\n```bash\nsqlxport run \\\n  --export-mode redshift-unload \\\n  --db-url redshift://... \\\n  --query \"SELECT * FROM large_table\" \\\n  --s3-output-prefix s3://bucket/unload/ \\\n  --iam-role arn:aws:iam::123456789012:role/MyUnloadRole\n```\n\n---\n\n## 🐍 Python API\n\n```python\nfrom sqlxport.api.export import run_export, ExportJobConfig\n\nconfig = ExportJobConfig(\n    db_url=\"sqlite:///test.db\",\n    query=\"SELECT * FROM users\",\n    format=\"csv\",\n    output_file=\"out.csv\",\n    export_mode=\"sqlite-query\"\n)\n\nrun_export(config)\n```\n\n---\n\n## 🧪 Running Tests\n\n```bash\npytest tests/unit/\npytest tests/integration/\npytest tests/e2e/\n```\n\n---\n\n## 🔧 Environment Variables\n\nSupports `.env` or exported shell variables:\n\n```env\nDB_URL=postgresql://username:password@localhost:5432/mydb\nS3_BUCKET=my-bucket\nS3_KEY=data/users.parquet\nS3_ACCESS_KEY=...\nS3_SECRET_KEY=...\nS3_ENDPOINT=https://s3.amazonaws.com\nIAM_ROLE=arn:aws:iam::123456789012:role/MyUnloadRole\n```\n\nGenerate a template with:\n\n```bash\nsqlxport run --generate-env-template\n```\n\n---\n\n## 🛠 Roadmap\n\n* ✅ Modular export modes\n* ✅ CSV and partitioned output\n* ⏳ Add `jsonl`, `xlsx` formats\n* ⏳ Plugin system for writers/loaders\n* ⏳ SaaS mode / UI platform\n* ⏳ Kafka/Kinesis streaming support\n\n---\n\n## 🔐 Security\n\n* Don’t commit `.env` files\n* Use credential vaults when possible\n\n---\n\n## 👨‍💼 Author\n\nVahid Saber  \nBuilt with ❤️ for data engineers and developers.\n\n---\n\n## 📄 License\n\nMIT License\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvahid110%2Fsqlxport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvahid110%2Fsqlxport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvahid110%2Fsqlxport/lists"}