https://github.com/sloev/commoncrawler_fetch_example
https://github.com/sloev/commoncrawler_fetch_example
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/sloev/commoncrawler_fetch_example
- Owner: sloev
- Created: 2020-07-15T22:57:16.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2020-07-15T23:01:08.000Z (about 6 years ago)
- Last Synced: 2025-12-26T13:49:44.409Z (7 months ago)
- Language: Python
- Size: 1.95 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# example of commoncrawler html news downloader
uses commoncrawler to search for news site prefix (dr.dk/indland/*) and downloads all html plus extracts articles from it
## install
```
pip install cdx_toolkit requests trafilatura
```
## usage
```
python test.py > output.json
```