{"id":16406600,"url":"https://github.com/odino/docsql","last_synced_at":"2025-03-23T05:31:22.018Z","repository":{"id":57560759,"uuid":"121291364","full_name":"odino/docsql","owner":"odino","description":"Import Google Docs' spreadsheets into a MySQL table","archived":false,"fork":false,"pushed_at":"2019-08-18T12:57:13.000Z","size":234,"stargazers_count":27,"open_issues_count":1,"forks_count":9,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-18T18:14:05.578Z","etag":null,"topics":["golang","google-docs","mysql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/odino.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-12T19:26:42.000Z","updated_at":"2024-04-30T08:03:24.000Z","dependencies_parsed_at":"2022-08-25T15:11:36.039Z","dependency_job_id":null,"html_url":"https://github.com/odino/docsql","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odino%2Fdocsql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odino%2Fdocsql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odino%2Fdocsql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odino%2Fdocsql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/odino","download_url":"https://codeload.github.com/odino/docsql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245061382,"owners_count":20554563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","google-docs","mysql"],"created_at":"2024-10-11T06:10:23.025Z","updated_at":"2025-03-23T05:31:21.626Z","avatar_url":"https://github.com/odino.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# docsql\n\nA tool to import spreadsheets hosted on Google Docs to a MySQL table.\n\n## Usage\n\nGrab a binary from the [releases' page](https://github.com/odino/docsql/releases)\nand start having some fun:\n\n``` bash\n$ docsql \\\n--doc \"https://docs.google.com/spreadsheets/d/1vyVxaYgfZ2Tka7reg4whg99kRlWqpg6cKvEa1QFArZI/export?format=tsv\" \\\n--table my_sample \\\n--connection \"root:@tcp(localhost:3308)/test?charset=utf8\u0026allowAllFiles=true\"\n\n2018/02/12 23:20:31 Downloading https://docs.google.com/spreadsheets/d/1vyVxaYgfZ2Tka7reg4whg99kRlWqpg6cKvEa1QFArZI/export?format=tsv ...\n2018/02/12 23:20:32 Doc downloaded in my_sample_1518463231621589126.csv\n2018/02/12 23:20:32 Connecting to MySQL...\n2018/02/12 23:20:32 Creating table 'my_sample_1518463231621589126'...\n2018/02/12 23:20:32 Connecting to MySQL...\n2018/02/12 23:20:32 Loading data into 'my_sample_1518463231621589126'...\n2018/02/12 23:20:32 Connecting to MySQL...\n2018/02/12 23:20:32 Swapping 'my_sample' with 'my_sample_1518463231621589126'\n2018/02/12 23:20:32 Connecting to MySQL...\n2018/02/12 23:20:32 Creating table 'my_sample'...\n2018/02/12 23:20:32 Connecting to MySQL...\n2018/02/12 23:20:32 Clearing old tables...\n2018/02/12 23:20:32 All done\n```\n\n![doc](https://raw.githubusercontent.com/odino/docsql/master/images/doc.png)\n\n![select](https://raw.githubusercontent.com/odino/docsql/master/images/docsql.png)\n\n## Advanced\n\n### Spreadsheet\n\nYour spreadsheet will need to be shared publicly (*anyone with the link can access*),\nand the URL you need to feed to docsql takes the form of `https://docs.google.com/spreadsheets/d/$DOCID/export?format=tsv`\nwhere `$DOCID` is the unique ID of the Google Doc.\n\nBy default, docsql will download the first sheet in the doc, but if you need to\nimport other sheets you can simply append the `gid` of the sheet at the end of the URL\n(`https://docs.google.com/spreadsheets/d/$DOCID/export?format=tsv\u0026gid=$GID`).\n\nPlease note that the export format must be **tsv** because, well, [it's just\neasier than csv](https://en.wikipedia.org/wiki/Tab-separated_values)\n\n### MySQL\n\nInstead of passing the connection string to MySQL as a flag you can export it as\nenvironment variable -- this makes sure you don't leave credentials on the CLI:\n\n``` bash\n$ export $CONNECTION=...\n\n$ docsql \\\n--doc \"https://docs.google.com/spreadsheets/d/1vyVxaYgfZ2Tka7reg4whg99kRlWqpg6cKvEa1QFArZI/export?format=tsv\" \\\n--table my_sample  \n\n2018/02/12 23:27:30 Downloading https://docs.google.com/spreadsheets/d/1vyVxaYgfZ2Tka7reg4whg99kRlWqpg6cKvEa1QFArZI/export?format=tsv ...\n2018/02/12 23:27:33 Doc downloaded in my_sample_1518463650997899367.csv\n2018/02/12 23:27:33 Connecting to MySQL...\n2018/02/12 23:27:33 Creating table 'my_sample_1518463650997899367'...\n2018/02/12 23:27:33 Connecting to MySQL...\n2018/02/12 23:27:33 Loading data into 'my_sample_1518463650997899367'...\n2018/02/12 23:27:33 Connecting to MySQL...\n2018/02/12 23:27:33 Swapping 'my_sample' with 'my_sample_1518463650997899367'\n2018/02/12 23:27:33 Connecting to MySQL...\n2018/02/12 23:27:33 Creating table 'my_sample'...\n2018/02/12 23:27:33 Connecting to MySQL...\n2018/02/12 23:27:33 Clearing old tables...\n2018/02/12 23:27:33 All done\n```\n\nBe aware that `LOAD DATA LOCAL INFILE` must be available on the MySQL server,\nand you will need to end your connection string with `allowAllFiles=true` so that\nthe Go MySQL driver is allowed to process local files.\n\n### Keeping old tables\n\ndocsql is (probably) meant to run as a cron, or everytime you make an update to\nyour spreadsheet -- whenever it runs, it nukes the previous version of the output\ntable and imports the new contents of the spreadsheet.\n\nYou can customize how many (old) tables to keep with the `--keep` flag. For example,\n`docsql ... --keep 5` will keep 5 version of the old table in MySQL:\n\n``` bash\nmysql\u003e SHOW TABLES;\n+---------------------------------------+\n| Tables_in_test                        |\n+---------------------------------------+\n| my_sample                             |\n| my_sample_1518463163413558194_archive |\n| my_sample_1518463168405819860_archive |\n| my_sample_1518463173716215291_archive |\n| my_sample_1518463231621589126_archive |\n| my_sample_1518463650997899367_archive |\n+---------------------------------------+\n```\n\n### Table structure\n\ndocsql will make a few opinionated assumptions for you:\n\n* all fields in the table are `VARCHAR(255)`\n* it creates an `docsql_id` field used as a primary key\n* it adds an `docsql_created_at` with the timestamp when the rows were loaded into the table\n* will sanitize column names (taken from the spreadsheet) filtering out non alphanumeric characters\n\nThere are plans to make all of these configurable in the future through flags...\n...PRs are more than welcome!\n\n### Other stuff?\n\nIt might be a good idea to run `docsql --help` to have a look at what's available.\n\n## Contributing\n\ndocsql is being developed through docker because... ...well, don't always have\nthe Go toolchain with me!\n\nAnyhow, it should be fairly straighforward to get running:\n\n* `make build_docker`, will build the docker container used to develop\n* `make test ARGS=\"go run main.go -d $YOUR_TEST_DOC -t $TABLE -c $MYSQL_CONNECTION_STRING\"` will build and run docsql on the fly\n* `make release` when you want to generate a release binary (under `builds/`)\n\nFeel free to rant or, even better, fix some of my crappy code through a [pull request](https://github.com/odino/docsql/pulls)!\n\n## Tests\n\n![tommy](https://raw.githubusercontent.com/odino/docsql/master/images/tommy.png)\n\n## Ideas\n\n* if a column ends in `:index` it should be indexed\n* ability to alter the `CREATE TABLE` via flags\n* abort if some basic checks don't pass (ie. minmum number of rows when someone nukes the doc by mistake)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fodino%2Fdocsql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fodino%2Fdocsql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fodino%2Fdocsql/lists"}