https://github.com/pythoninthegrass/download
Feed requests a URL, extension, and local directory to download files scraped from a website.
https://github.com/pythoninthegrass/download
python
Last synced: 5 months ago
JSON representation
Feed requests a URL, extension, and local directory to download files scraped from a website.
- Host: GitHub
- URL: https://github.com/pythoninthegrass/download
- Owner: pythoninthegrass
- License: unlicense
- Created: 2023-09-24T21:01:47.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-23T16:49:30.000Z (about 2 years ago)
- Last Synced: 2025-01-11T16:50:31.144Z (about 1 year ago)
- Topics: python
- Language: Makefile
- Homepage:
- Size: 5.89 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE.md
Awesome Lists containing this project
README
# download
[**Download**](https://www.youtube.com/watch?v=T5jHO4jMPxA)
## Summary
Feed `requests` a URL, extension, and local directory to download files scraped from a website.
Cache initial response to avoid too many requests to server.
**Table of Contents**
* [download](#download)
* [Summary](#summary)
* [Setup](#setup)
* [Quickstart](#quickstart)
* [Demo](#demo)
* [Usage](#usage)
* [Development](#development)
* [TODO](#todo)
* [Further Reading](#further-reading)
## Setup
* Dependencies
* make
* [Linux](https://www.gnu.org/software/make/)
* [macOS](https://www.freecodecamp.org/news/install-xcode-command-line-tools/)
* [editorconfig](https://editorconfig.org/)
* Install python and tooling
```bash
# install python and dependencies (e.g., git, ansible, etc.)
./bootstrap install
```
## Quickstart
### Demo
### Usage
```bash
# run script with defaults
./main.py
# add environment variables for url, ext, and path
cp .env.example .env
# fill out (e.g., 'EXT=".csv"')
# run with overrides
./main.py
```
## Development
```bash
# install tools and runtimes (cf. xcode, brew, asdf, poetry, etc.)
./bootstrap # dev only runs plays w/tags and is verbose
# install git hooks
./bootstrap install-precommit
# update git hooks
./bootstrap update-precommit
```
## TODO
* [Open Issues](https://github.com/pythoninthegrass/download/issues)
## Further Reading
* [python](https://www.python.org/)
* [asdf](https://asdf-vm.com/guide/getting-started.html#_2-download-asdf)
* [poetry](https://python-poetry.org/docs/)
* [docker-compose](https://docs.docker.com/compose/install/)
* [pre-commit hooks](https://pre-commit.com/)