Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/h4r7w3l1/http_file_prober
Simple a Bash tool to parse URLs size and content-type
https://github.com/h4r7w3l1/http_file_prober
bash calculate-size prober remote-file web-size webscraping
Last synced: 5 days ago
JSON representation
Simple a Bash tool to parse URLs size and content-type
- Host: GitHub
- URL: https://github.com/h4r7w3l1/http_file_prober
- Owner: h4r7w3l1
- Created: 2021-04-14T01:21:49.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-04-14T01:24:56.000Z (over 3 years ago)
- Last Synced: 2024-02-13T21:54:49.638Z (9 months ago)
- Topics: bash, calculate-size, prober, remote-file, web-size, webscraping
- Language: Shell
- Homepage:
- Size: 1000 Bytes
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# http_file_prober
Simple `bash` tool for **parsing size** and type of URLs.
Each request **only gets first KB** of data and parses **content length**+**type** & **headers**.
The output will be saved in execution path, appends the results to a file.
---
## Install Debian/Ubuntu
```bash
▶ sudo apt install ripgrep curl
```
## Install MacOS
```bash
▶ brew install ripgrep curl
```
### Options
```bash
# Usage of http_file_prober.sh:
▶ http_file_prober.sh [url] [url2] [urlX..]
Put one or more space-delimited urls/domains as arguments
▶ cat urls.txt | xargs -n10 -P25 /fullpath/http_file_prober.sh
Run multiple process "-P25" - process count. Check out `man xargs` for help
```
### Usage sample
```bash
▶ ./http_file_prober.sh https://pastebin.com/raw/{8XD3uxYQ,cydhSP1v,g5wTvVq7}
https://pastebin.com/raw/8XD3uxYQ | Null | text/plain
https://pastebin.com/raw/cydhSP1v | Null | text/plain
https://pastebin.com/raw/g5wTvVq7 | Null | text/plain
▶ ./http_file_prober.sh https://yandex.ru/news/quotes/2002.html https://yandex.ru/support/common/troubleshooting/main.html
https://yandex.ru/news/quotes/2002.html | 6856 | text/html
https://yandex.ru/support/common/troubleshooting/main.html | 266851 | text/html
```