https://github.com/simonw/conditional-get

CLI tool for fetching data using HTTP conditional get
https://github.com/simonw/conditional-get

git-scraping http

Last synced: 22 days ago
JSON representation

CLI tool for fetching data using HTTP conditional get

• Host: GitHub
• URL: https://github.com/simonw/conditional-get
• Owner: simonw
• License: apache-2.0
• Created: 2020-04-29T09:02:29.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2021-07-14T03:28:02.000Z (over 3 years ago)
• Last Synced: 2024-10-06T20:50:57.804Z (about 1 month ago)
• Topics: git-scraping, http
• Language: Python
• Size: 17.6 KB
• Stars: 14
• Watchers: 3
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-starred - simonw/conditional-get - CLI tool for fetching data using HTTP conditional get (http)

README

        
# conditional-get

[![PyPI](https://img.shields.io/pypi/v/conditional-get.svg)](https://pypi.org/project/conditional-get/)

[![Changelog](https://img.shields.io/github/v/release/simonw/conditional-get?include_prereleases&label=changelog)](https://github.com/simonw/conditional-get/releases)

[![Tests](https://github.com/simonw/conditional-get/workflows/Test/badge.svg)](https://github.com/simonw/conditional-get/actions?query=workflow%3ATest)

[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/conditional-get/blob/main/LICENSE)

CLI tool for fetching data using [HTTP conditional get](https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests).

## Installation

    pip install conditional-get

## Usage

The first time you run this command it will download the file and store the ETag (if one was returned) in a file called `etags.json`.

The second time you run this command against the same URL it will use that ETag, potentially resulting in a `304 Not Modified` response which saves bandwidth by not re-downloading the file.

    # First run - will fetch the file

    conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png

    # Second run - will only fetch the file if it has changed

    conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png

The filename will be derived from the URL. You can customize the filename using the `-o` option:

    conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png -o til.png

By default the ETags for the retrieved URLs will be stored in a `etags.json` file in the current directory. You can use the `--etags otherfile.json` to store that file somewhere else:

    conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png --etags my-etags.json

Use the `-v` option to get debug output showing what is happening:

    $ conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png -v

    Response status code: 200

    [####################################]  100%

    $ ls

    Simon_Willison__TIL.png	etags.json

    $ cat etags.json 

    {

        "https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png": "\"d65b78782dfa93213c99099e0e2181d8\""

    }

    $ conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png -v

    Existing ETag: "d65b78782dfa93213c99099e0e2181d8"

    Response status code: 304

The key used to store the ETag in `etags.json` defaults to the URL. You can specify a custom key using the `--key` option:

    conditional-get --key til https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png

    cat etags.json

    {

        "til": "\"d65b78782dfa93213c99099e0e2181d8\""

    }

This is useful if the URL to the file changes even though the file contents stays the same - for example if you are downloading files from URLs that include an expiring signature.