Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/cak/foot
Foot is a library that fetches a list of URLs and silly walks through each site to gather information.
https://github.com/cak/foot
bugbounty crawler scraping
Last synced: 27 days ago
JSON representation
Foot is a library that fetches a list of URLs and silly walks through each site to gather information.
- Host: GitHub
- URL: https://github.com/cak/foot
- Owner: cak
- License: mit
- Created: 2018-11-22T10:28:42.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2018-11-22T10:36:48.000Z (almost 6 years ago)
- Last Synced: 2024-09-24T02:30:04.709Z (about 1 month ago)
- Topics: bugbounty, crawler, scraping
- Language: Python
- Homepage: https://pypi.org/project/foot/
- Size: 8.79 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.rst
- Contributing: CONTRIBUTING.rst
- License: LICENSE
Awesome Lists containing this project
README
# foot
Web Crawling Project
Foot is a library that fetches a list of URLs and silly walks through each site to gather information.
## Usage
### Install
`pip install foot`
#### foot (cli)
- `-u` URL(s) (encapsulated in quotes, separated by commas)
- `-f` filename (list of urls on new lines)
##### Options (not required)
- `-c` : Specify chunk size (default=10)
- `--recursive` : Enable one level of recursion
##### Examples:
Array of URLS:
`foot -u 'http://example.com, http://example1.com'`
Array of URLS with Options:
`foot 'http://example.com, http://example1.com' -c 5 --recursive`
File:
`foot -f 'example.txt`
File with Options:
`foot -f 'example.txt' -c 5 --recursive`
## Module
Import:
`import foot`
### Functions
The `get` function takes an array of URLS and options.
```javascript
foot.get(["http://www.example`.com", "http://www.example2.com"], options);
=> Data output in ./foot/url.json and foot-date.txt
```
The `file` function takes a filename and options.
```python
foot.file("./test.txt", recursive=True)
=> Data output in ./foot/url.json and foot-date.txt
```