https://github.com/maxharlow/csv-fetch

📚 Downloads a list of URLs from a CSV file
https://github.com/maxharlow/csv-fetch

csv

Last synced: about 2 months ago
JSON representation

📚 Downloads a list of URLs from a CSV file

• Host: GitHub
• URL: https://github.com/maxharlow/csv-fetch
• Owner: maxharlow
• Created: 2021-08-05T11:02:20.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2024-06-26T10:51:51.000Z (7 months ago)
• Last Synced: 2024-09-28T09:17:41.268Z (3 months ago)
• Topics: csv
• Language: JavaScript
• Homepage:
• Size: 60.5 KB
• Stars: 5
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
CSV Fetch

=========

Download a list of URLs from a CSV file.

Installing

----------

    $ npm install -g csv-fetch

Alternatively, don't install it and just prepend the below commands with `npx`.

Completions for Zsh will also be installed if a directory exists:

    $ mkdir -p /usr/local/share/zsh/site-functions

    $ chown -R $(whoami) /usr/local/share/zsh/site-functions

Usage

-----

    $ csv-fetch    

Where `` is the name of your CSV, `` is the name of a directory where the files will be downloaded to, `` is a column from the CSV file with unique identifiers for each row, and `` is a column from the CSV file with the URLs of the files to be fetched.

Give a suffix, such as a file extension, to be added to all filenames with `-s`. Specify a column to be used to generate subdirectories with `-d`.

Request headers can be specified using the `-h` flag, in `key: value` format. Specify the flag multiple times to specify multiple headers.

Note the identifiers in the name column should be unique, otherwise files will be overwritten -- or skipped, if the `-c` flag is given.

Requests are made as fast as possible by default. The `-l` flag lets you set a rate limit for the maximum number that should be made per second.

HTTP requests are automatically retried if they fail, five times by default, but this can be adjusted with the `-r` flag.

Request checking can be turned on with the `-c` flag. This will check to see whether there is an existing file in your depository for each row, and skip making the request if so. Alternatively, `-C` will use a faster cache which takes less time than checking the file actually exists.