{"id":42703936,"url":"https://github.com/wareflowx/excel-to-sql","last_synced_at":"2026-01-29T14:25:28.535Z","repository":{"id":333548171,"uuid":"1135721850","full_name":"wareflowx/excel-to-sql","owner":"wareflowx","description":"A powerful CLI tool and Python SDK for importing Excel files into SQLite databases with advanced data transformation, validation, and quality profiling.","archived":false,"fork":false,"pushed_at":"2026-01-20T10:40:47.000Z","size":183,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-20T19:31:33.982Z","etag":null,"topics":["cli","excel","python","sql","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wareflowx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-16T13:54:54.000Z","updated_at":"2026-01-20T14:14:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/wareflowx/excel-to-sql","commit_stats":null,"previous_names":["aliiibenn/excel-to-sql"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/wareflowx/excel-to-sql","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wareflowx%2Fexcel-to-sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wareflowx%2Fexcel-to-sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wareflowx%2Fexcel-to-sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wareflowx%2Fexcel-to-sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wareflowx","download_url":"https://codeload.github.com/wareflowx/excel-to-sql/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wareflowx%2Fexcel-to-sql/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28879458,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T10:31:27.438Z","status":"ssl_error","status_checked_at":"2026-01-29T10:31:01.017Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","excel","python","sql","sqlite"],"created_at":"2026-01-29T14:25:28.476Z","updated_at":"2026-01-29T14:25:28.521Z","avatar_url":"https://github.com/wareflowx.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Excel to SQLite\n\nA powerful CLI tool and Python SDK for importing Excel files into SQLite databases with advanced data transformation, validation, and quality profiling features.\n\n[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)\n[![PyPI](https://img.shields.io/badge/pypi/v-excel--to--sql-blue)](https://pypi.org/project/excel-to-sql/)\n[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)\n\n## Overview\n\nExcel to SQLite simplifies the process of importing Excel data into SQLite databases. It provides automatic schema detection, data transformations, validation rules, and includes an intelligent Auto-Pilot mode for zero-configuration setup.\n\n**Key Features:**\n- Smart Import with automatic schema detection\n- Flexible data transformations (value mappings, calculated columns)\n- Comprehensive validation system\n- Data quality profiling and scoring\n- Auto-Pilot mode with pattern detection\n- Python SDK for programmatic access\n- Rich terminal output with detailed progress reporting\n\n## Table of Contents\n\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [CLI Reference](#cli-reference)\n- [Python SDK](#python-sdk)\n- [Auto-Pilot Mode](#auto-pilot-mode)\n- [Configuration](#configuration)\n- [Examples](#examples)\n- [Development](#development)\n- [Contributing](#contributing)\n- [Changelog](#changelog)\n- [License](#license)\n\n## Installation\n\n### From PyPI\n\n```bash\npip install excel-to-sql\n```\n\n### With uv\n\n```bash\nuv pip install excel-to-sql\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/wareflowx/excel-to-sql.git\ncd excel-to-sql\nuv sync\n```\n\n## Quick Start\n\nExcel to SQLite provides two ways to get started: choose **Auto-Pilot** for automatic setup or **Manual Configuration** for complete control.\n\n### Option 1: Auto-Pilot Mode (Recommended)\n\nThe fastest way to import Excel files. Auto-Pilot automatically detects patterns, suggests transformations, and guides you through the setup.\n\n```bash\n# Analyze Excel files and generate configuration automatically\nexcel-to-sql magic --data ./path/to/excels\n\n# Interactive mode with step-by-step guidance\nexcel-to-sql magic --data ./path/to/excels --interactive\n\n# Dry run to analyze without generating configuration\nexcel-to-sql magic --data ./path/to/excels --dry-run\n```\n\nAuto-Pilot detects:\n- Primary and foreign keys automatically\n- Value mappings (e.g., \"1\"/\"0\" to \"Active\"/\"Inactive\")\n- Data quality issues with prioritized recommendations\n- Optimal data types for each column\n\n### Option 2: Manual Configuration\n\nFor complete control over the import process.\n\n```bash\n# 1. Initialize project\nexcel-to-sql init\n\n# 2. Add mapping configuration\nexcel-to-sql config add --type products --table products --pk id\n\n# 3. Import Excel file\nexcel-to-sql import --file products.xlsx --type products\n\n# 4. Export data back to Excel\nexcel-to-sql export --table products --output report.xlsx\n\n# 5. Profile data quality\nexcel-to-sql profile --table products --output quality-report.html\n```\n\n## CLI Reference\n\n### Project Commands\n\n#### Initialize Project\n\nCreates a new excel-to-sql project with the required directory structure.\n\n```bash\nexcel-to-sql init\n```\n\nCreates:\n- `data/` - SQLite database location\n- `config/` - Configuration files\n- `imports/` - Imported Excel files\n- `exports/` - Exported Excel files\n\n#### Import Command\n\nImport an Excel file into the database.\n\n```bash\nexcel-to-sql import --file data.xlsx --type products\n```\n\n**Options:**\n- `--file, -f` - Path to Excel file (required)\n- `--type, -t` - Type configuration name (required)\n- `--force` - Re-import even if content unchanged\n\n#### Export Command\n\nExport database data back to Excel.\n\n```bash\nexcel-to-sql export --table products --output report.xlsx\n```\n\n**Options:**\n- `--table` - Table name to export\n- `--output, -o` - Output Excel file path\n\n#### Profile Command\n\nGenerate data quality reports.\n\n```bash\nexcel-to-sql profile --table products --output quality-report.html\n```\n\n### Configuration Commands\n\n#### Add Type Configuration\n\nCreate a new mapping type interactively.\n\n```bash\nexcel-to-sql config add --type customers --table customers --pk id\n```\n\n**Options:**\n- `--add-type` - Name for the new type\n- `--table` - Target table name\n- `--pk` - Primary key column(s)\n- `--file` - Excel file for auto-detection (optional)\n\n#### List Types\n\nShow all configured mapping types.\n\n```bash\nexcel-to-sql config --list\n```\n\n#### Show Type Details\n\nDisplay configuration for a specific type.\n\n```bash\nexcel-to-sql config --show products\n```\n\n#### Remove Type\n\nDelete a mapping type.\n\n```bash\nexcel-to-sql config --remove old_type\n```\n\n### Magic Command (Auto-Pilot)\n\nAutomatic configuration and import with pattern detection.\n\n```bash\n# Automatic mode\nexcel-to-sql magic --data ./excels --output .excel-to-sql\n\n# Interactive mode\nexcel-to-sql magic --data ./excels --interactive\n\n# Dry run\nexcel-to-sql magic --data ./excels --dry-run\n```\n\n**Options:**\n- `--data, -d` - Directory containing Excel files (default: current directory)\n- `--output, -o` - Output directory for configuration (default: .excel-to-sql)\n- `--dry-run` - Analyze without generating configuration\n- `--interactive, -i` - Interactive guided setup\n\n## Python SDK\n\nThe Python SDK provides programmatic access to all excel-to-sql features.\n\n### Basic Usage\n\n```python\nfrom excel_to_sql import ExcelToSqlite\n\n# Initialize SDK\nsdk = ExcelToSqlite()\n\n# Import Excel file with transformations\nresult = sdk.import_excel(\n    file_path=\"data.xlsx\",\n    type_name=\"products\",\n    tags=[\"q1-2024\", \"verified\"]\n)\n\nprint(f\"Imported {result['rows_imported']} rows\")\n\n# Query data\ndf = sdk.query(\"SELECT * FROM products WHERE price \u003e 100\")\nprint(df.head())\n\n# Profile data quality\nprofile = sdk.profile_table(\"products\")\nprint(f\"Quality score: {profile['summary']['quality_score']}\")\nprint(f\"Issues found: {len(profile['issues'])}\")\n\n# Export to Excel with multi-sheet support\nsdk.export_to_excel(\n    output=\"report.xlsx\",\n    sheet_mapping={\n        \"Products\": \"products\",\n        \"Categories\": \"SELECT * FROM categories\"\n    }\n)\n```\n\n### Advanced Transformations\n\n```python\nfrom excel_to_sql import ExcelToSqlite\nfrom excel_to_sql.transformations import ValueMapping, CalculatedColumn\nfrom excel_to_sql.validators import ValidationRule\n\nsdk = ExcelToSqlite()\n\n# Configure value mappings\nvalue_mappings = {\n    \"status\": {\"1\": \"Active\", \"0\": \"Inactive\"},\n    \"state\": {\"NY\": \"New York\", \"CA\": \"California\"}\n}\n\n# Configure calculated columns\ncalculated_columns = [\n    CalculatedColumn(\"total\", \"quantity * price\"),\n    CalculatedColumn(\"tax\", \"total * 0.1\"),\n    CalculatedColumn(\"grand_total\", \"total + tax\")\n]\n\n# Configure validation rules\nvalidation_rules = [\n    ValidationRule(\"id\", \"unique\"),\n    ValidationRule(\"email\", \"regex\", pattern=r\"^[^@]+@[^@]+\\.[^@]+$\"),\n    ValidationRule(\"age\", \"range\", min_value=0, max_value=120)\n]\n```\n\n## Auto-Pilot Mode\n\nAuto-Pilot mode provides zero-configuration import with intelligent pattern detection, quality scoring, and automated recommendations.\n\n### What Auto-Pilot Detects\n\n**Pattern Detection:**\n- **Primary Keys** - Identifies unique columns automatically\n- **Foreign Keys** - Detects relationships between tables\n- **Value Mappings** - Finds code columns requiring translation\n- **Split Fields** - Identifies redundant status columns to combine\n- **Data Types** - Infers optimal SQL types from data\n\n**Quality Analysis:**\n- **Quality Score** (0-100) with letter grades (A-D)\n- **Issue Detection** - Null values, duplicates, type mismatches\n- **Statistical Analysis** - Value distributions, outliers\n- **Data Profiling** - Column types, null percentages\n\n**Smart Recommendations:**\n- **Prioritized Suggestions** (HIGH/MEDIUM/LOW)\n- **Auto-fixable Issues** with one-click corrections\n- **Default Value Suggestions**\n- **French Code Detection** (ENTRÉE→inbound, SORTIE→outbound, etc.)\n\n### Auto-Pilot Components\n\nThe Auto-Pilot system consists of five main components:\n\n**PatternDetector** - Analyzes Excel files and detects patterns with confidence scores\n\n```python\nfrom excel_to_sql.auto_pilot import PatternDetector\n\ndetector = PatternDetector()\npatterns = detector.detect_patterns(df, \"table_name\")\n\n# Returns: primary_key, foreign_keys, value_mappings, split_fields, confidence\n```\n\n**QualityScorer** - Generates comprehensive quality reports\n\n```python\nfrom excel_to_sql.auto_pilot import QualityScorer\n\nscorer = QualityScorer()\nreport = scorer.generate_quality_report(df, \"table_name\")\n\n# Returns: score (0-100), grade (A-D), issues, column_stats\n```\n\n**RecommendationEngine** - Provides prioritized, actionable recommendations\n\n```python\nfrom excel_to_sql.auto_pilot import RecommendationEngine\n\nengine = RecommendationEngine()\nrecommendations = engine.generate_recommendations(\n    df, \"table_name\", quality_report, patterns\n)\n\n# Returns prioritized recommendations (HIGH/MEDIUM/LOW)\n```\n\n**AutoFixer** - Automatically fixes data quality issues\n\n```python\nfrom excel_to_sql.auto_pilot import AutoFixer\n\nfixer = AutoFixer()\nresult = fixer.apply_auto_fixes(\n    df, file_path, \"Sheet1\", recommendations, dry_run=False\n)\n\n# Fixes: null values, French codes, split fields with backup system\n```\n\n**InteractiveWizard** - Guided configuration workflow\n\n```python\nfrom excel_to_sql.ui import InteractiveWizard\n\nwizard = InteractiveWizard()\nresult = wizard.run_interactive_mode(\n    excel_files, patterns_dict, quality_dict, output_path\n)\n```\n\n### When to Use Auto-Pilot\n\n**Perfect for:**\n- Quick prototyping and testing\n- Ad-hoc data imports\n- Exploring new datasets\n- Learning the tool\n- Small to medium datasets\n\n**Not ideal for:**\n- Production deployments (use generated config as template)\n- Complex custom transformations\n- Highly specialized business logic\n- Performance-critical operations\n\n## Configuration\n\nMapping configuration is stored in `config/mappings.json`:\n\n```json\n{\n  \"products\": {\n    \"target_table\": \"products\",\n    \"primary_key\": [\"id\"],\n    \"column_mappings\": {\n      \"ID\": {\"target\": \"id\", \"type\": \"integer\"},\n      \"Name\": {\"target\": \"name\", \"type\": \"string\"},\n      \"Price\": {\"target\": \"price\", \"type\": \"float\"}\n    },\n    \"value_mappings\": [\n      {\n        \"column\": \"status\",\n        \"mappings\": {\"1\": \"Active\", \"0\": \"Inactive\"}\n      }\n    ],\n    \"calculated_columns\": [\n      {\n        \"name\": \"total\",\n        \"expression\": \"quantity * price\"\n      }\n    ],\n    \"validation_rules\": [\n      {\n        \"column\": \"id\",\n        \"type\": \"unique\"\n      }\n    ]\n  }\n}\n```\n\n### Column Types\n\nSupported SQL types:\n- `string` - TEXT columns (default)\n- `integer` - INTEGER with Int64 (nullable)\n- `float` - REAL columns\n- `boolean` - BOOLEAN (0/1)\n- `date` - TIMESTAMP (ISO-8601)\n\n## Examples\n\n### Example 1: E-commerce Product Import\n\n```bash\n# Initialize project\nexcel-to-sql init\n\n# Use Auto-Pilot to analyze products\nexcel-to-sql magic --data ./products --interactive\n\n# Review generated configuration\ncat config/mappings.json\n\n# Import with auto-generated configuration\nexcel-to-sql import --file products.xlsx --type products\n```\n\n### Example 2: Data Migration with Validation\n\n```python\nfrom excel_to_sql import ExcelToSqlite\nfrom excel_to_sql.validators import ValidationRule\n\nsdk = ExcelToSqlite()\n\n# Add custom validation rules\nrules = [\n    ValidationRule(\"email\", \"regex\", pattern=r\"^[^@]+@[^@]+\\.[^@]+$\"),\n    ValidationRule(\"age\", \"range\", min_value=0, max_value=120),\n    ValidationRule(\"id\", \"unique\")\n]\n\n# Import with validation\nresult = sdk.import_excel(\n    \"customers.xlsx\",\n    \"customers\",\n    validation_rules=rules\n)\n\nif result['validation_errors']:\n    print(f\"Found {len(result['validation_errors'])} validation errors\")\n```\n\n### Example 3: Quality Analysis\n\n```python\nfrom excel_to_sql import ExcelToSqlite\n\nsdk = ExcelToSqlite()\n\n# Profile data quality\nprofile = sdk.profile_table(\"orders\")\n\n# Check quality score\nscore = profile['summary']['quality_score']\ngrade = profile['summary']['grade']\nprint(f\"Quality Score: {score}/100 ({grade})\")\n\n# Review issues\nfor issue in profile['issues']:\n    print(f\"{issue['severity']}: {issue['column']} - {issue['message']}\")\n\n# Generate HTML report\nsdk.generate_quality_report(\n    \"orders\",\n    output=\"quality-report.html\"\n)\n```\n\n## Development\n\n### Running Tests\n\n```bash\n# Run all tests\nuv run pytest\n\n# Run with coverage\nuv run pytest --cov=excel_to_sql --cov-report=html\n\n# Run specific test file\nuv run pytest tests/test_auto_pilot.py -v\n```\n\n### Project Structure\n\n```\nexcel-to-sqlite/\n├── excel_to_sql/          # Main package\n│   ├── cli.py            # CLI interface\n│   ├── sdk/              # Python SDK\n│   ├── entities/         # Domain entities\n│   ├── transformations/  # Data transformations\n│   ├── validators/       # Data validation\n│   ├── profiling/        # Quality analysis\n│   ├── auto_pilot/       # Auto-Pilot mode\n│   └── ui/               # Interactive wizard\n├── tests/                # Test suite\n├── docs/                 # Documentation\n└── config/               # Configuration files\n```\n\n### Test Coverage\n\n- 200+ tests with comprehensive coverage\n- \u003e85% coverage on core modules\n- Integration tests with real Excel fixtures\n- Unit tests for all components\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\nFor documentation improvements, see `docs/issues/001-documentation-website.md`.\n\n## Changelog\n\nSee [CHANGELOG.md](CHANGELOG.md) for version history and release notes.\n\n### Version 0.3.0\n\n**Auto-Pilot Mode - Zero-Configuration Import:**\n- Pattern Detection - Automatic detection of PKs, FKs, value mappings, split fields\n- Quality Scoring - Multi-dimensional data quality analysis with grades (A-D)\n- Smart Recommendations - Prioritized, actionable suggestions (HIGH/MEDIUM/LOW)\n- Auto-Fix Capabilities - One-click corrections for common data issues\n- Interactive Wizard - Step-by-step guided configuration workflow\n- French Code Support - Automatic translation (ENTRÉE→inbound, SORTIE→outbound, etc.)\n- Split Field Detection - Intelligent COALESCE for redundant columns\n- CLI Integration - `magic` command with --interactive flag\n\n**Testing:**\n- 143+ tests for Auto-Pilot components\n- Integration tests with real Excel fixtures\n- \u003e85% coverage for core Auto-Pilot modules\n\n**Total: 200+ tests** with comprehensive coverage across all features.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Links\n\n- [Documentation](docs/)\n- [Changelog](CHANGELOG.md)\n- [Issue Tracker](https://github.com/wareflowx/excel-to-sql/issues)\n- [PyPI Package](https://pypi.org/project/excel-to-sql/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwareflowx%2Fexcel-to-sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwareflowx%2Fexcel-to-sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwareflowx%2Fexcel-to-sql/lists"}