{"id":15871142,"url":"https://github.com/sgreben/csv-dl","last_synced_at":"2025-04-01T22:25:53.324Z","repository":{"id":80988905,"uuid":"167751509","full_name":"sgreben/csv-dl","owner":"sgreben","description":"csv-dl reads CSV on stdin, and downloads linked files. single binary, no dependencies. linux, osx, windows. #golang","archived":false,"fork":false,"pushed_at":"2019-01-27T14:49:54.000Z","size":186,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-07T14:40:37.942Z","etag":null,"topics":["cli","csv","download","scraper"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sgreben.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-26T23:47:51.000Z","updated_at":"2023-12-12T12:06:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"930c96f3-3d7e-4952-bf6c-e6c15207312b","html_url":"https://github.com/sgreben/csv-dl","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sgreben%2Fcsv-dl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sgreben%2Fcsv-dl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sgreben%2Fcsv-dl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sgreben%2Fcsv-dl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sgreben","download_url":"https://codeload.github.com/sgreben/csv-dl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246719710,"owners_count":20822790,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","csv","download","scraper"],"created_at":"2024-10-06T00:41:57.049Z","updated_at":"2025-04-01T22:25:53.307Z","avatar_url":"https://github.com/sgreben.png","language":"Go","readme":"# csv-dl\n\n`csv-dl` reads CSV on stdin, and downloads linked files.\n\n![screenshot](doc/shot.png)\n\n- [Examples](#examples)\n  - [Field names from CSV header](#field-names-from-csv-header)\n  - [Field values by column index](#field-values-by-column-index)\n  - [Field names from CLI args](#field-names-from-cli-args)\n  - [Relative/incomplete links](#relativeincomplete-links)\n  - [Multiple links per row](#multiple-links-per-row)\n- [Get it](#get-it)\n- [Usage](#usage)\n\n## Examples\n\n### Field names from CSV header\n\n**input.csv**\n```csv\n\"app\",\"version\",\"url\"\ncsv-dl,0.0.1,https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip\n```\n\n```sh\n$ \u003cinput.csv csv-dl -u -l '{{field \"url\"}}'\n2019/01/27 12:25:41 https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip\n\n$ ls\ncsv-dl_0.0.1_linux_x86_64.zip\n```\n\n### Field values by column index\n\n**input.csv**\n```csv\ncsv-dl,0.0.1,https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip\n```\n\n```sh\n$ \u003cinput.csv csv-dl -l '{{column 2}}'\n2019/01/27 12:25:41 https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip\n\n$ ls\ncsv-dl_0.0.1_linux_x86_64.zip\n```\n\n### Field names from CLI args\n\n**input.csv**\n```csv\ncsv-dl,0.0.1,https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip\n```\n\n```sh\n$ \u003cinput.csv csv-dl -s ',,url' -l '{{field \"url\"}}'\n2019/01/27 12:25:41 https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip\n\n$ ls\ncsv-dl_0.0.1_linux_x86_64.zip\n```\n\n### Relative/incomplete links\n\n**input.csv**\n```csv\ncsv-dl,0.0.1,\"0.0.1/csv-dl_0.0.1_linux_x86_64.zip\"\n```\n\n```sh\n$ \u003cinput.csv csv-dl -s ',,link' -l 'https://github.com/sgreben/csv-dl/releases/download/{{field \"link\"}}'\n2019/01/27 12:25:41 https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip\n\n$ ls\ncsv-dl_0.0.1_linux_x86_64.zip\n```\n\n### Multiple links per row\n\n**input.csv**\n```csv\nname,title,release,source\ncsv-dl,0.0.1,https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip,https://github.com/sgreben/csv-dl/archive/0.0.1.tar.gz\n```\n\n```sh\n$ \u003cinput.csv csv-dl -u -l '{{field \"release\"}}' -l '{{field \"source\"}}'\n2019/01/27 12:42:23 https://github.com/sgreben/csv-dl/releases/download/0.0.1/csv-dl_0.0.1_linux_x86_64.zip\n2019/01/27 12:42:23 https://github.com/sgreben/csv-dl/archive/0.0.1.tar.gz\n\n$ ls\n0.0.1.tar.gz\ncsv-dl_0.0.1_linux_x86_64.zip\n```\n\n## Get it\n\nUsing go get:\n\n```bash\ngo get -u github.com/sgreben/csv-dl\n```\n\nOr [download the binary](https://github.com/sgreben/csv-dl/releases/latest) from the releases page.\n\n```bash\n# Linux\ncurl -LO https://github.com/sgreben/csv-dl/releases/download/1.3.0/csv-dl_1.3.0_linux_x86_64.zip\nunzip csv-dl_1.3.0_linux_x86_64.zip\n\n# OS X\ncurl -LO https://github.com/sgreben/csv-dl/releases/download/1.3.0/csv-dl_1.3.0_osx_x86_64.zip\nunzip csv-dl_1.3.0_osx_x86_64.zip\n\n# Windows\ncurl -LO https://github.com/sgreben/csv-dl/releases/download/1.3.0/csv-dl_1.3.0_windows_x86_64.zip\nunzip csv-dl_1.3.0_windows_x86_64.zip\n```\n\n## Usage\n\n```text\nUsage of csv-dl:\n  -H value\n    \t(alias for -http-header)\n  -dry-run\n    \tonly print URLs to stdout (one per line), do not download\n  -f\t(alias for -force-overwrite)\n  -force-overwrite\n    \toverwrite existing files\n  -http-header value\n    \ta HTTP header KEY:VALUE\n  -l value\n    \t(alias for -link)\n  -link value\n    \ta link to download, may use go {{template}} syntax and refer to data columns by index (column i) or name (field \"f\")\n  -n\t(alias for -dry-run)\n  -p int\n    \t(alias for -parallel) (default 8)\n  -parallel int\n    \tnumber of parallel connections (default 8)\n  -q\t(alias for -quiet)\n  -quiet\n    \tsuppress all logging\n  -r duration\n    \t(alias for -rate-limit)\n  -rate-limit duration\n    \tat most one-request-per-$duration (0 = off)\n  -s string\n    \t(alias for -schema)\n  -schema string\n    \tuse the given CSV expression as the table schema\n  -skip-csv-header\n    \tassume the first row is the CSV header, skip it\n  -u\t(alias for -use-csv-header)\n  -use-csv-header\n    \tassume the first row is the CSV header, use it as a schema\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsgreben%2Fcsv-dl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsgreben%2Fcsv-dl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsgreben%2Fcsv-dl/lists"}