{"id":22989313,"url":"https://github.com/olbrichattila/gocsvimporter","last_synced_at":"2025-04-02T11:44:46.408Z","repository":{"id":243942369,"uuid":"813850012","full_name":"olbrichattila/gocsvimporter","owner":"olbrichattila","description":"Fast and large file optimised CSV importer","archived":false,"fork":false,"pushed_at":"2025-02-03T21:39:16.000Z","size":83,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-03T22:30:31.983Z","etag":null,"topics":["csv","csv-import","golang-application","golang-cli"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/olbrichattila.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-11T21:38:00.000Z","updated_at":"2025-02-03T21:39:20.000Z","dependencies_parsed_at":"2024-09-13T03:41:23.135Z","dependency_job_id":"919164a4-6f18-4686-a537-a754958db359","html_url":"https://github.com/olbrichattila/gocsvimporter","commit_stats":null,"previous_names":["olbrichattila/gocsvimporter"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olbrichattila%2Fgocsvimporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olbrichattila%2Fgocsvimporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olbrichattila%2Fgocsvimporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olbrichattila%2Fgocsvimporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/olbrichattila","download_url":"https://codeload.github.com/olbrichattila/gocsvimporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246811244,"owners_count":20837745,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","csv-import","golang-application","golang-cli"],"created_at":"2024-12-15T04:17:19.638Z","updated_at":"2025-04-02T11:44:46.383Z","avatar_url":"https://github.com/olbrichattila.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go CSV Importer: Multi-threaded Fast CSV to Database Tool\n\n## Overview\n`Go CSV Importer` is a powerful, multi-threaded command-line tool designed to import large CSV files into databases quickly and efficiently. It supports various database types, offers customizable import modes, and is optimized for performance.\n\n### Key Features\n- **Support for Large CSV Files**: Handles millions of rows with ease.\n- **Multi-Database Compatibility**: Works with SQLite, MySQL, PostgreSQL, and Firebird.\n- **Configurable Import Modes**:\n  - Transactional or non-transactional imports.\n  - Batch SQL inserts or row-by-row operations.\n  - Single or multiple database connections.\n\n---\n\n## Installation\n\nInstall the latest version directly from Go:\n\n```bash\ngo install github.com/olbrichattila/gocsvimporter/cmd/csvimporter@latest\n```\n\n---\n\n## How It Works\n\n1. **Analyze the CSV**: Determines data types and structures.\n2. **Prepare the Database**: Automatically creates the necessary table, dropping it if it already exists.\n3. **Import Data**: Optimizes the process based on database type and chosen parameters.\n\n### Usage\nRun the tool with:\n\n```bash\ncsvimporter \u003ccsv_file\u003e \u003ctable_name\u003e [delimiter]\n```\n\n#### Parameters:\n1. **CSV File**: Path to the CSV file.\n2. **Table Name**: Target database table name.\n3. **Delimiter** *(Optional)*: CSV delimiter (default: `,`).\n\n#### Example:\n```bash\ncsvimporter data.csv vehicles \";\"\n```\n\n## Field Header Management for Faster Imports\nThe CSV importer command line tool now includes a powerful feature for handling CSV headers efficiently.\n### Feature Overview\n\nWhen importing CSV files, the tool automatically:\n1. **Analyzes the CSV content:** Deduplicates field names, determines proper field types, and calculates field sizes based on the data.\n2. **Optimizes imports:** Speeds up processing for repeated file formats by allowing you to save and reuse header definitions.\n\nThis analysis can take time, especially with large files. If you're importing the same file format multiple times, you can now save the calculated headers for reuse, bypassing recalculation for subsequent imports.\n\n## How It Works\n\n### Save Field Definitions:\nUse the -sh=\u003cfileName\u003e parameter to save the field definitions in a JSON file. You can optionally modify this JSON file to fine-tune your header configurations.\n```\ncsvimporter \u003ccsv_file\u003e \u003ctable_name\u003e [delimiter] -sh=customerFieldDefinition\n```\nThis will generate a JSON file with the field definitions.\n\n### Load Field Definitions:\nUse the -lh=\u003cfileName\u003e parameter to load the pre-saved field definitions, skipping the recalculation step.\n```\ncsvimporter \u003ccsv_file\u003e \u003ctable_name\u003e [delimiter] -lh=customerFieldDefinition\n```\nWhen this parameter is provided, the import process starts immediately, but progress percentage updates will not be displayed.\n### Example JSON Structure\nHere’s an example of the auto-saved JSON file generated by the tool:\n```json\n[\n {\n  \"name\": \"index\",\n  \"Type\": \"INT\",\n  \"Length\": 7\n },\n {\n  \"name\": \"customer_id\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 15\n },\n {\n  \"name\": \"first_name\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 11\n },\n {\n  \"name\": \"last_name\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 11\n },\n {\n  \"name\": \"company\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 38\n },\n {\n  \"name\": \"city\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 24\n },\n {\n  \"name\": \"country\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 51\n },\n {\n  \"name\": \"phone_a1\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 22\n },\n {\n  \"name\": \"phone_a2\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 22\n },\n {\n  \"name\": \"email\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 44\n },\n {\n  \"name\": \"subscription_date\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 10\n },\n {\n  \"name\": \"website\",\n  \"Type\": \"VARCHAR\",\n  \"Length\": 40\n }\n]\n```\n### Benefits\n- ***Time-Saving:*** Skip repetitive calculations for consistent file formats.\n- ***Customizable:*** Modify the saved JSON to align with specific import requirements.\n- ***Improved Efficiency:*** Ideal for workflows requiring frequent CSV imports.\n\n### Notes\n- The -lh parameter disables progress percentage display.\n- Ensure the JSON file accurately reflects your data structure for seamless imports.\n\n\u003e By leveraging this feature, the CSV importer command line tool becomes even more efficient for managing recurring import tasks and large datasets.\n\n## Supported Databases\n\n### SQLite\n```env\nDB_CONNECTION=sqlite\nDB_DATABASE=./database/database.sqlite\n```\n\n### MySQL\n```env\nDB_CONNECTION=mysql\nDB_HOST=127.0.0.1\nDB_PORT=3306\nDB_DATABASE=mydb\nDB_USERNAME=myuser\nDB_PASSWORD=mypassword\n```\n\n### PostgreSQL\n```env\nDB_CONNECTION=pgsql\nDB_HOST=127.0.0.1\nDB_PORT=5432\nDB_DATABASE=postgres\nDB_USERNAME=postgres\nDB_PASSWORD=postgres\nDB_SSLMODE=disable\n```\n\n### Firebird\n```env\nDB_CONNECTION=firebird\nDB_HOST=127.0.0.1\nDB_PORT=3050\nDB_DATABASE=/path/to/database.fdb\nDB_USERNAME=SYSDBA\nDB_PASSWORD=masterkey\n```\n\n---\n\n## Configuration\n\nCreate a `.env.csvimporter` file in the application's directory or export environment variables:\n\n```env\nBATCH_SIZE=500               # Rows per batch (default: 100)\nMAX_CONNECTION_COUNT=25      # Maximum connections (default: 10)\nBATCH_INSERT=on              # Enable/disable batch insert\nMULTIPLE_CONNECTIONS=on      # Enable/disable multi-threading\nTRANSACTIONAL=off            # Enable/disable transactions\n```\n\n*Note: Unsupported options for certain databases (e.g., SQLite) are ignored.*\n\n---\n\n## Performance\n\n### Speed Test: 2 Million Rows\n**System Configuration**:\n- **OS**: Ubuntu Linux\n- **Processor**: Intel® Core™ i7-3770S @ 3.10GHz\n- **Storage**: SSD\n\n| Database    | Duration   | Mode                     | Threads |\n|-------------|------------|--------------------------|---------|\n| **SQLite**  | 52 seconds | Transactional, Batch SQL | 1       |\n| **MySQL**   | 65 seconds | Transactional, Multi-Threaded | 10      |\n| **PostgreSQL** | 43 seconds | Transactional, Multi-Threaded | 10      |\n| **Firebird**| 5m 42s     | Transactional, Multi-Threaded | 10      |\n\n---\n\n## Makefile Targets\n\n### Test Imports\n```bash\nmake vehicles\nmake customers\n```\n\n### Switch Environments\n```bash\nmake switch-sqlite\nmake switch-mysql\nmake switch-pgsql\nmake switch-firebird\n```\n\n---\n\n## Local Testing with Docker\nA `docker-compose` setup is provided for testing:\n\n```bash\ncd docker\ndocker-compose up -d\n```\n\n---\n\n## Roadmap\n\n### Planned Improvements\n- **Distributed Import**: Split CSV files across multiple instances (pods/servers) for faster parallel imports.\n- **Enhanced Configuration**: Support more advanced database-specific settings.\n\n---\n\nStart importing your CSV files faster and more efficiently today!\n\n\n## About me:\n- Learn more about me on my personal website. https://attilaolbrich.co.uk/menu/my-story\n- Check out my latest blog blog at my personal page. https://attilaolbrich.co.uk/blog/1/single\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folbrichattila%2Fgocsvimporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folbrichattila%2Fgocsvimporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folbrichattila%2Fgocsvimporter/lists"}