{"id":17912921,"url":"https://github.com/medcl/csv2sql","last_synced_at":"2025-06-25T07:03:59.898Z","repository":{"id":57609736,"uuid":"131955185","full_name":"medcl/csv2sql","owner":"medcl","description":"A data import tool written for the non-profit project: ifish(http://bit.ly/ifish-project)","archived":false,"fork":false,"pushed_at":"2018-06-14T07:02:35.000Z","size":32288,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-04T21:12:35.042Z","etag":null,"topics":["csv2db","csv2sql","golang","mysql","tool","util","xlsx"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/medcl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-03T07:03:35.000Z","updated_at":"2023-08-21T02:45:03.000Z","dependencies_parsed_at":"2022-08-27T11:33:08.145Z","dependency_job_id":null,"html_url":"https://github.com/medcl/csv2sql","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/medcl/csv2sql","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/medcl%2Fcsv2sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/medcl%2Fcsv2sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/medcl%2Fcsv2sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/medcl%2Fcsv2sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/medcl","download_url":"https://codeload.github.com/medcl/csv2sql/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/medcl%2Fcsv2sql/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261823686,"owners_count":23215140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv2db","csv2sql","golang","mysql","tool","util","xlsx"],"created_at":"2024-10-28T19:48:06.102Z","updated_at":"2025-06-25T07:03:59.866Z","avatar_url":"https://github.com/medcl.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\nCSV2SQL, An util to convert csv data to SQL scripts and load to MySQL.\n\n## How it works\n\nPipeline:\n\n[Read the xlsx file] -\u003e [Convert to SQL scripts] -\u003e [Execute SQL scripts]\n\n## How to use\n\n1. Prepare the xlsx file with your data\n\nAssume the file is dropped into `test/import.xlsx`, you can download it from [here](https://github.com/medcl/csv2sql/blob/master/test/import.xlsx)\n\n\u003cimg width=\"800\"  src=\"https://github.com/medcl/csv2sql/raw/master/doc/assets/img/Snip20180505_8.png\"\u003e\n\n2. Config the `csv2sql.yml`\n\n```\nmodules:\n- name: pipeline\n  enabled: true\n  runners:\n    - name: process_excel\n      enabled: true\n      input_queue: primary\n      max_go_routine: 1\n      threshold_in_ms: 0\n      timeout_in_ms: 5000\n      default_config:\n        start:\n          joint: read_csv\n          enabled: true\n          parameters:\n            file_name: \"test/import.xlsx\"\n        process:\n        - joint: convert_sql\n          enabled: true\n          parameters:\n            sheet_name: fish_information\n            data_start_from_index: 3\n            column_name:\n            - id\n            - outer_code\n            - common_name\n            - scientific_name\n            - english_name\n            - chinese_name\n            - region_name\n            - aquatic_category_id\n            - category_name\n            - is_homemade\n            - aquatic_region_id\n            - inner_code\n            - produce_pattern\n            - feed_pattern\n            - catch_pattern\n            row_format:\n            - 'INSERT INTO `aquatic_base_info` (`id`, `outer_code`, `common_name`, `scientific_name`, `english_name`, `chinese_name`, `region_name`, `aquatic_category_id`)'\n            - 'VALUES (\u003c{id: }\u003e, \u003c{outer_code: }\u003e, \u003c{common_name: }\u003e, \u003c{scientific_name: }\u003e, \u003c{english_name: }\u003e, \u003c{chinese_name: }\u003e, \u003c{region_name: }\u003e, \u003c{aquatic_category_id: }\u003e);'\n            - 'INSERT INTO `aquatic_source` (`inner_code`, `aquatic_base_info_id`, `is_homemade`, `aquatic_region_id`, `produce_pattern`, `feed_pattern`, `catch_pattern`) '\n            - 'VALUES (\u003c{inner_code: }\u003e, \u003c{id: }\u003e, \u003c{is_homemade: }\u003e, \u003c{aquatic_region_id: }\u003e, \u003c{produce_pattern: }\u003e, \u003c{feed_pattern: }\u003e, \u003c{catch_pattern: }\u003e);'\n\n        - joint: convert_sql\n          enabled: false\n          parameters:\n            sheet_name: aquatic_region\n            data_start_from_index: 1\n            column_name:\n            - id\n            - code\n            - name\n            - is_homemade\n            row_format:\n            - 'INSERT INTO `aquatic_region` (`id`, `code`, `name`, `is_homemade`) VALUES (\u003c{id: }\u003e, \u003c{code: }\u003e, \u003c{name: }\u003e, \u003c{is_homemade: }\u003e);'\n\n        - joint: import_sql\n          enabled: true\n          parameters:\n            mysql_conn: root:password@tcp(localhost:3306)/ifish?charset=utf8\n            rollback_enabled: true\n\n        - joint: logging\n          enabled: true\n\n        error:\n          joint: on_error\n          enabled: true\n```\n\nNote, there are more than one joint config with name: `convert_sql`, which means you can import multi data sheet with different config,\nin this example the second `convert_sql` joint has been disabled, you can enable it if you wish, and also you can add more `convert_sql` joints.\n\nAnd also as you can see, in the first `convert_sql` joint, with the data sheet `fish_information`, the config `row_format` is a array,\nand have more than one SQL template, separated with `;` , which means you can generate multi SQL from one single data sheet,\nmap one data row to multi mysql data records, and then we can inset the data into different mysql tables.\n\nThe config `row_format` is a SQL template, and the config `column_name` is how your data sheet will be used in your template,\nlike this template variable `\u003c{is_homemade: }\u003e`, it will looking for the data from column `is_homemade` which we have already configured in section `column_name`,\nyou can get the SQL template by using MySQLWorkBench quickly([select db]-\u003e[select table]-\u003e[Copy to Clipboard]-\u003e[Insert Statement])\n\u003cimg width=\"800\"  src=\"https://github.com/medcl/csv2sql/raw/master/doc/assets/img/Snip20180505_9.png\"\u003e\n\nNote, once you change the column in data sheet, you must keep the `column_name` updated.\n\nNote, please also change MySQL connection in joint `mysql_conn` config, ie: `your_mysql_user:you_password@tcp(your_mysql_host:3306)/your_mysql_db?charset=utf8`\n\nAnd, the example is jus a example, you can use it to map any data sheet to any mysql table, update the config and import the data.\n\n3. Start to import the data\n```\n➜  csv2sql git:(master) ✗ ./bin/csv2sql\n  _____  _______      _____   _____  ____  _\n / ____|/ ____\\ \\    / /__ \\ / ____|/ __ \\| |\n| |    | (___  \\ \\  / /   ) | (___ | |  | | |\n| |     \\___ \\  \\ \\/ /   / / \\___ \\| |  | | |\n| |____ ____) |  \\  /   / /_ ____) | |__| | |____\n \\_____|_____/    \\/   |____|_____/ \\___\\_\\______|\n[CSV2SQL] An util to convert csv data to SQL scripts.\n0.1.0_SNAPSHOT,  67026ee, Sat May 5 13:23:42 2018 +0800, medcl, support import multi datasheet\n\n[05-05 15:29:38] [INF] [instance.go:23] workspace: data/APP/nodes/0\n[05-05 15:29:38] [INF] [pipeline.go:67] pipeline: process_excel started with 1 instances\n[05-05 15:29:38] [INF] [api.go:147] api server listen at: http://0.0.0.0:2900\n[05-05 15:29:38] [INF] [ui.go:149] http server listen at: http://127.0.0.1:9001\n[05-05 15:29:38] [INF] [import_sql.go:87] sql execute success, 1 rows affected, lastInsertID: 1\n[05-05 15:29:38] [INF] [import_sql.go:87] sql execute success, 1 rows affected, lastInsertID: 1\n^C\n[CSV2SQL] got signal:interrupt, start shutting down\n                         |    |\n   _` |   _ \\   _ \\   _` |     _ \\  |  |   -_)\n \\__, | \\___/ \\___/ \\__,_|   _.__/ \\_, | \\___|\n ____/                             ___/\n[CSV2SQL] 0.1.0_SNAPSHOT, uptime:5.773621s\n\n```\n\n4. Now, check out the database, you will see 2 new records in two different table\n\n\u003cimg width=\"800\"  src=\"https://github.com/medcl/csv2sql/raw/master/doc/assets/img/Snip20180505_5.png\"\u003e\n\u003cimg width=\"800\"  src=\"https://github.com/medcl/csv2sql/raw/master/doc/assets/img/Snip20180505_6.png\"\u003e\n\n\nLicense\n=======\nReleased under the [Apache License, Version 2.0](https://github.com/medcl/csv2sql/blob/master/LICENSE) .\n\nPowered by [https://github.com/infinitbyte/framework](https://github.com/infinitbyte/framework)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmedcl%2Fcsv2sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmedcl%2Fcsv2sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmedcl%2Fcsv2sql/lists"}