{"id":51640740,"url":"https://github.com/tamnd/ccrawl-cli","last_synced_at":"2026-07-13T19:02:51.779Z","repository":{"id":364482389,"uuid":"1267996847","full_name":"tamnd/ccrawl-cli","owner":"tamnd","description":"A fast, friendly command line for Common Crawl: URL index search, WARC fetch, Parquet columnar queries, and dataset building.","archived":false,"fork":false,"pushed_at":"2026-07-11T03:54:38.000Z","size":712,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-11T05:14:01.703Z","etag":null,"topics":["cli","common-crawl","dataset","duckdb","golang","open-data","scraper","warc","web-archive","web-scraping"],"latest_commit_sha":null,"homepage":"https://ccrawl-cli.tamnd.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tamnd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-13T03:38:18.000Z","updated_at":"2026-07-11T03:53:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tamnd/ccrawl-cli","commit_stats":null,"previous_names":["tamnd/ccrawl-cli"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/tamnd/ccrawl-cli","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamnd%2Fccrawl-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamnd%2Fccrawl-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamnd%2Fccrawl-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamnd%2Fccrawl-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tamnd","download_url":"https://codeload.github.com/tamnd/ccrawl-cli/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamnd%2Fccrawl-cli/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35432821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-13T02:00:06.543Z","response_time":119,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","common-crawl","dataset","duckdb","golang","open-data","scraper","warc","web-archive","web-scraping"],"created_at":"2026-07-13T19:02:51.057Z","updated_at":"2026-07-13T19:02:51.771Z","avatar_url":"https://github.com/tamnd.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ccrawl\n\nA fast, friendly command line for [Common Crawl](https://commoncrawl.org).\nOne binary that finds pages in the URL index, fetches the exact bytes Common Crawl saw, streams WARC/WAT/WET archives, queries the columnar Parquet index, looks up domain ranks, and builds datasets.\n\n![ccrawl resolving the latest crawl, reading a page as text, searching the URL index, and generating columnar SQL](docs/static/demo.gif)\n\nFull documentation: [ccrawl-cli.tamnd.com](https://ccrawl-cli.tamnd.com).\n\n## What is Common Crawl\n\n[Common Crawl](https://commoncrawl.org) is a nonprofit that has been crawling the web since 2008 and publishes the result as a free, openly licensed dataset.\nEvery month or so it releases a new crawl of billions of pages, hosted on `data.commoncrawl.org` and mirrored in Amazon S3.\nThere are no API keys and nothing to pay for, you just need to know where to look.\n\nThat last part is the catch.\nUsing the data by hand means juggling the [CDX index API](https://index.commoncrawl.org), S3 paths, multi-member gzip WARC files, and a pile of glue code.\nccrawl puts all of it behind one tool with sane defaults, real output formats, and commands that pipe into each other.\n\n\u003e [!IMPORTANT]\n\u003e This project is an independent client.\n\u003e It is not affiliated with or endorsed by the Common Crawl Foundation.\n\u003e Please read and follow the [Common Crawl terms of use](https://commoncrawl.org/terms-of-use) when you use the data.\n\n## Install\n\n```sh\ngo install github.com/tamnd/ccrawl-cli/cmd/ccrawl@latest\n```\n\nOr grab a prebuilt binary from the [releases page](https://github.com/tamnd/ccrawl-cli/releases).\nThe binary is pure Go with no runtime dependencies.\nDuckDB is optional and only needed to run the columnar index queries locally (see [Bulk questions](#bulk-questions-the-columnar-index)).\n\nBuild from source:\n\n```sh\ngit clone https://github.com/tamnd/ccrawl-cli\ncd ccrawl-cli\nmake build      # produces ./bin/ccrawl\n```\n\n## Quick start\n\n```sh\nccrawl crawls latest                  # newest crawl ID, for example CC-MAIN-2026-25\nccrawl get example.com --text         # the readable text of the latest capture\nccrawl get example.com --markdown     # the same page as Markdown\nccrawl search 'example.com/*'         # every capture under a path\nccrawl search 'example.com/*' -o url  # just the URLs, one per line\n```\n\nRun `ccrawl \u003ccommand\u003e --help` for the full flag list on any command.\nEvery command speaks the same global flags, so `-o`, `-c`, `-n`, and `--fields` work everywhere.\n\n\u003e [!TIP]\n\u003e ccrawl is built to compose.\n\u003e Any command that lists records can pipe into another, so `search` finds captures, `fetch` pulls their bytes, and a shell pipe (`|`) is the glue.\n\n## Find a page: `get`\n\n`get` is curl for Common Crawl.\nIt resolves the newest capture of a URL in the index, pulls that one record with an HTTP byte-range request, and prints what Common Crawl saw.\n\n```sh\nccrawl get example.com                # the captured HTML\nccrawl get example.com --text         # readable plain text\nccrawl get example.com --markdown     # HTML converted to Markdown\nccrawl get example.com --headers      # the captured HTTP response headers\nccrawl get example.com --links -o url # outbound links, one per line\nccrawl get example.com --at 2023-06   # the capture nearest a date\nccrawl get example.com --all -o jsonl # every capture across crawls\n```\n\nThe `extract` subcommands are shortcuts for a single transform:\n\n```sh\nccrawl extract text example.com\nccrawl extract links example.com\nccrawl extract title example.com\nccrawl extract markdown example.com -O page.md\n```\n\n## Search the index: `search`\n\n`search` queries the URL index (the CDX server) for captures of a URL or pattern.\nThe match type is inferred from wildcards, and you can always override it with `--match`.\n\n```sh\nccrawl search example.com                    # exact URL, every timestamp\nccrawl search 'example.com/*'                # every URL under a path (prefix)\nccrawl search '*.example.com'                # the domain and all its subdomains\nccrawl search '*.example.com' --status 200   # only successful captures\nccrawl search example.com --mime text/html   # filter by detected MIME type\nccrawl search example.com -c all -n 50        # across every crawl, newest first\nccrawl search example.com --at 2022-06        # the capture nearest a date, per URL\nccrawl search example.com --estimate          # approximate record count per crawl\n```\n\nNarrow the columns or shape each row:\n\n```sh\nccrawl search example.com --fields url,status,timestamp\nccrawl search example.com --template '{{.URL}} {{.Status}}'\n```\n\n## Fetch by location: `fetch`\n\n`fetch` retrieves WARC records by their exact byte location.\nThe point is composition: anything that emits location records (filename, offset, length) can pipe straight into it.\n`search --locations` and `columnar locations` both do.\n\n```sh\n# Every PDF Common Crawl saw on a domain, downloaded one file per record:\nccrawl search 'example.com/*' --mime application/pdf --locations \\\n  | ccrawl fetch - --dir --out-dir pdfs/\n\n# Read text from records the columnar index found:\nccrawl columnar locations --domain example.com -o jsonl \\\n  | ccrawl fetch - --text\n```\n\nYou can also fetch a single record by hand:\n\n```sh\nccrawl fetch --file crawl-data/CC-MAIN-2026-25/.../x.warc.gz --offset 123456 --length 4567 --text\n```\n\n## Bulk questions: the columnar index\n\nThe columnar (Parquet) index is the fastest way to answer crawl-wide questions like \"every PDF on .gov domains\" without touching a single WARC.\nccrawl builds the SQL for you and runs it against the public Parquet files using a local `duckdb` binary if one is on your `PATH`.\n\n```sh\nccrawl columnar urls --tld gov --mime application/pdf -o url\nccrawl columnar count --domain example.com\nccrawl columnar langs --tld jp\nccrawl columnar mimes --domain example.com\n```\n\n\u003e [!NOTE]\n\u003e DuckDB is optional.\n\u003e If it is not on your `PATH`, ccrawl prints ready-to-run SQL instead of running it, so nothing is required to get a useful answer.\n\nAsk for the SQL directly with `--print`, then paste it into [DuckDB](https://duckdb.org), Athena, Spark, or Trino:\n\n```sh\nccrawl columnar sql --tld gov --mime application/pdf --print\n```\n\n```sql\nSELECT url, url_host_registered_domain, fetch_status, content_mime_detected, ...\nFROM read_parquet('https://data.commoncrawl.org/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026-25/subset=warc/*.parquet', hive_partitioning=1)\nWHERE url_host_tld = 'gov'\n  AND content_mime_detected = 'application/pdf'\n```\n\nInstall DuckDB from [duckdb.org](https://duckdb.org/docs/installation) to run the queries directly.\nThe ccrawl binary never links DuckDB, so installs stay small and pure Go.\n\n## Raw archives: `paths`, `download`, `parse`, `convert`\n\nEach crawl ships its data as three archive kinds, all on [data.commoncrawl.org](https://data.commoncrawl.org):\n\n- **WARC** holds the full HTTP request and response,\n- **WAT** holds extracted metadata and links,\n- **WET** holds plain text.\n\nList the archive paths, download them, decode them, and convert them:\n\n```sh\nccrawl paths warc -c 2026-25                   # every WARC path for a crawl\nccrawl paths wet -n 1 | ccrawl download -      # download the first WET file\nccrawl download warc -n 5                       # the first 5 WARC files of the latest crawl\nccrawl parse local.warc.gz --type response -o table -n 20\nccrawl convert local.warc.wet.gz --to parquet -O out.parquet\n```\n\n## CC-NEWS\n\n[CC-NEWS](https://commoncrawl.org/blog/news-dataset-available) is a separate, continuously updated dataset of news articles.\nIt has no URL index, so a host search streams the month's WARC files and keeps matching records.\n\n```sh\nccrawl news list --year 2026 --month 5\nccrawl news search bbc.co.uk --year 2026 --month 5 -n 50\n```\n\n## Output formats\n\nEvery list command renders through the same formatter.\nPick a format with `-o`, or let ccrawl choose: a table when writing to a terminal, JSONL when piped.\n\n```sh\nccrawl search example.com -o table   # aligned columns for reading\nccrawl search example.com -o jsonl   # one JSON object per line, for piping\nccrawl search example.com -o json    # a single JSON array\nccrawl search example.com -o csv     # spreadsheet friendly\nccrawl search example.com -o url     # just the URL column\nccrawl search example.com -o parquet \u003e out.parquet  # columnar, for analytics\n```\n\n## Building a dataset library\n\nFor bulk work you want the archive files in one tidy place you can come back to, not scattered across ad-hoc download dirs.\nThe `--library` flag gives the data files a home and extends the commands you already know to list, download, and process them in place.\nEverything lands under `~/notes/ccrawl` by default (`CCRAWL_LIBRARY` or `--library-dir` to move it), keyed by crawl and kind:\n\n```sh\nccrawl download wet -n 20 --library -c 2026-25    # pull 20 WET files into the library\nccrawl paths    wet --library -c 2026-25          # list the WET files you have locally\nccrawl parse    wet --library --lang eng -o jsonl # decode every local WET file, eng only\nccrawl convert  wet --library --to parquet         # write parquet beside the raw files\n```\n\nRaw archives go to `\u003ccrawl\u003e/\u003ckind\u003e/`, processed output to `\u003ccrawl\u003e/\u003cformat\u003e/\u003ckind\u003e/`, so a directory listing tells you exactly what you have.\nRe-running `download` only fetches what is missing, so the corpus grows incrementally.\nThe library is separate from the scratch data dir, so clearing scratch state never touches it.\n\n## How it works\n\nccrawl uses the index to find a capture, then fetches just that record with an HTTP byte-range request.\nA WARC file is a stream of gzip members, one per record, so a single record decompresses on its own without downloading the whole file.\nThat is what makes `ccrawl get` feel instant even though the file it lives in may be a gigabyte.\n\nccrawl is also a polite client.\nIt rate-limits itself and retries 403, 429, and 5xx responses with exponential backoff, honoring a `Retry-After` header when the CDN sends one.\n\n\u003e [!TIP]\n\u003e If requests keep failing, check the [Common Crawl status page](https://status.commoncrawl.org) to see whether the data service is having problems before digging into your own setup.\n\n## Configuration\n\nccrawl keeps all of its state under one tree, `~/data/ccrawl` by default: the cache, downloaded archives, converted Parquet, and the local DuckDB file.\nPoint it somewhere else with `CCRAWL_DATA_DIR`.\nThe curated dataset library is a separate tree, `~/notes/ccrawl` by default (`CCRAWL_LIBRARY` or `--library-dir`), so scratch state and the corpus you keep never mix.\n\nSee the resolved paths and settings any time:\n\n```sh\nccrawl config show\n```\n\nUseful global flags (all have sensible defaults):\n\n| Flag | Meaning |\n| --- | --- |\n| `-c, --crawl` | Crawl ID, year, `latest`, `all`, an integer for the newest N, or a comma list (default `latest`) |\n| `-o, --output` | Output format, including `parquet` (default auto) |\n| `-n, --limit` | Maximum results (`0` means unlimited) |\n| `-j, --workers` | Concurrency for downloads and scans |\n| `--source` | Bulk data source: `https` or `s3` |\n| `--rate` | Minimum delay between requests, to stay polite |\n| `--no-cache` | Bypass the on-disk cache |\n\n## Commands\n\n| Command | What it does |\n| --- | --- |\n| `crawls` | List, resolve, and inspect the monthly crawls |\n| `search` | Query the URL index (CDX) for captures of a URL or pattern |\n| `get` | Fetch what Common Crawl captured for a URL (curl for Common Crawl) |\n| `fetch` | Retrieve WARC records by explicit location, or from stdin |\n| `extract` | Pull text, links, title, or Markdown from a captured page |\n| `export` | Write matching captures into WARC files with provenance |\n| `download` | Download whole archive files for a crawl |\n| `paths` | List the archive file paths for a crawl |\n| `parse` | Decode a local WARC/WAT/WET file into records |\n| `convert` | Convert WARC/WAT/WET archives to Parquet or JSONL |\n| `columnar` | Query the columnar Parquet index (alias `table`, `athena`) |\n| `news` | Work with the continuous CC-NEWS dataset |\n| `rank` | Look up host and domain ranks from the web graph |\n| `db` | Build and query a local DuckDB database |\n| `stats` | Show the shape of a crawl: file counts per archive kind |\n| `config` | Show resolved configuration and data paths |\n| `cache` | Inspect and clear the on-disk cache |\n\n## Development\n\n```sh\nmake test    # run the test suite\nmake vet     # go vet\nmake build   # build ./bin/ccrawl\n```\n\nThe code is layered.\n`cli/` is the command tree built on Cobra.\n`ccrawl/` is the library it sits on: the collection list, the CDX index, the columnar index, downloads, ranks, and CC-NEWS.\nThe archive format parsers live in their own small packages under `pkg/`, each importable on its own:\n\n| Package | What it reads |\n| --- | --- |\n| `pkg/warc` | WARC records, plus the HTTP header/body split helpers |\n| `pkg/wat` | WAT metadata: status, title, meta tags, and links |\n| `pkg/wet` | WET extracted plain text |\n\n`pkg/wat` and `pkg/wet` build on `pkg/warc`, and none of them depend on `ccrawl/` or the CLI, so you can pull just the parser you need into your own program.\n\n## License\n\n[Apache 2.0](LICENSE).\n\nCommon Crawl data is provided by the [Common Crawl Foundation](https://commoncrawl.org) under their [terms of use](https://commoncrawl.org/terms-of-use).\nThis project is an independent client and is not affiliated with the foundation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftamnd%2Fccrawl-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftamnd%2Fccrawl-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftamnd%2Fccrawl-cli/lists"}