https://github.com/thunderpoot/scdx
A simple tool for querying the Common Crawl CDX
https://github.com/thunderpoot/scdx
Last synced: over 1 year ago
JSON representation
A simple tool for querying the Common Crawl CDX
- Host: GitHub
- URL: https://github.com/thunderpoot/scdx
- Owner: thunderpoot
- License: mit
- Created: 2024-02-27T16:00:40.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-26T17:59:25.000Z (about 2 years ago)
- Last Synced: 2024-06-26T22:32:23.825Z (about 2 years ago)
- Language: Rust
- Size: 9.77 KB
- Stars: 3
- Watchers: 4
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# `scdx`
## Simple CDX
A tool for querying the Common Crawl CDX Index. Versions in both Python and Rust are included in this repository. The command–line syntax is identical in both versions.
### Installation:
1. Clone this repository
2. To run the Rust version, compile and run via:
```
$ cargo build --release
```
```
$ cd target/release
$ chmod +x scdx
```
### Usage:
```bash
$ scdx --sleep 2 --domain commoncrawl.org --crawls CC-MAIN-2021-04 CC-MAIN-2024-10
```
```bash
$ scdx -s 10 -d '*.wikipedia.org' -c CC-MAIN-2023-50
```
```bash
$ scdx -l -d apple.com
```
The program will display a progress bar and output a file with a timestamp (e.g `2024-02-27_18-34-50_output.jsonl`) to the working directory, unless the `-o` or `--output` options are used.
The default sleep time is 2 seconds. Please be polite! Polling multiple times a second will make the index server sad. See the CCF system status [here](https://status.commoncrawl.org).
If no crawls are specified, all crawls will be queried. Use the `-l` or `--latest` flag to only query the latest crawl.
The API used supports two methods of wildcarding, like the (more advanced and mature) [cdx-toolkit](https://github.com/cocrawler/cdx_toolkit) by Greg Lindahl.
- **Prefixed asterisk**
The query `*.example.com`, in CDX jargon sets `matchType='domain'`, and will return captures for `blog.example.com`, `support.example.com`, etc.
- **Appended asterisk**
The query `example.com/*` will return captures for any page on `example.com`.
The Python version uses [`tqdm`](https://tqdm.github.io/) to display a progress bar, and the Rust version uses [`indicatif`](https://docs.rs/indicatif/latest/indicatif/).
### Licence
[MIT License](LICENSE)
### Thanks
- **[Greg Lindahl](https://github.com/wumpus)**
- **[Pedro Ortiz Suarez](https://github.com/pjox)**