{"id":51701514,"url":"https://github.com/mnot/cc-lint","last_synced_at":"2026-07-16T11:34:46.828Z","repository":{"id":368512351,"uuid":"1283656591","full_name":"mnot/cc-lint","owner":"mnot","description":"Lint HTTP responses across Common Crawl and report aggregate findings","archived":false,"fork":false,"pushed_at":"2026-06-30T23:13:32.000Z","size":1713,"stargazers_count":0,"open_issues_count":5,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-16T11:34:36.254Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://projects.mnot.net/cc-lint/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mnot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-29T05:54:02.000Z","updated_at":"2026-06-30T23:11:08.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mnot/cc-lint","commit_stats":null,"previous_names":["mnot/cc-lint"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mnot/cc-lint","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Fcc-lint","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Fcc-lint/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Fcc-lint/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Fcc-lint/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mnot","download_url":"https://codeload.github.com/mnot/cc-lint/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Fcc-lint/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35542951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-16T02:00:06.687Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-16T11:34:46.139Z","updated_at":"2026-07-16T11:34:46.823Z","avatar_url":"https://github.com/mnot.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Common Crawl Response Linter\n\nRun [httplint](https://github.com/mnot/httplint) against the\n[Common Crawl](https://commoncrawl.org/) WAT archives and report aggregate\nstatistics on the HTTP-level issues found in real-world responses.\n\nThere are two ways to run it, sharing the same linting and reporting code\n— only the input distribution and result aggregation differ:\n\n- **Local** — lint a handful of WAT files fetched over HTTP. No AWS\n  required; good for spot checks and development. Start here.\n- **Amazon EMR** — distribute an entire Common Crawl release across a\n  transient EMR cluster for a full-crawl analysis (~123M responses).\n  Needs an AWS account and costs money per run.\n\n---\n\n## Local run\n\nLint a few WAT files with no AWS setup. Requires Python 3.10+; the\nvirtualenv is created automatically on first run.\n\n```bash\n# One or more WAT paths, one per line:\necho crawl-data/CC-MAIN-2024-18/segments/.../warc/CC-MAIN-...warc.gz \u003e paths.txt\n\nmake report.html   # fetches the WATs, lints, writes report.html + report.md\n```\n\n`make report.html` wraps the `cc-lint lint` CLI: it fetches each WAT over\nHTTP from `data.commoncrawl.org`, runs httplint over the response\nmetadata, and renders the report. See `cc-lint lint --help` for the full\noption set.\n\n---\n\n## Running on Amazon EMR\n\nThe EMR path scales the same linting pipeline across an entire\nCommon Crawl release. It uses mrjob to launch a transient EMR\ncluster, distributes WARC paths across mappers, runs the linter in\nfork-isolated child processes (so a single bad WARC cannot take a\nmapper down), and aggregates the per-mapper `StatsCollector` dicts\ninto one summary record.\n\n**EMR jobs cost money on your AWS account.** Always confirm the\ncluster terminated in the EMR console after a run.\n\n### One-time configuration\n\n1. Install and configure the AWS CLI (region `us-east-1` is closest\n   to Common Crawl's S3 bucket).\n\n2. Create your config files. `make setup` prompts for your S3 bucket\n   name and generates all three from the tracked `*.example` templates:\n\n   ```bash\n   make setup\n   ```\n\n   It writes `cc-lint.mk` (your `OUTPUT_DIR` / `PATHS_PREFIX` /\n   `WHEEL_S3_PATH`) plus `mrjob.conf` and `mrjob-test.conf` (the\n   `cloud_log_dir` where mrjob preserves EMR cluster logs, so\n   `make emr-timing EMR_LOG_CLUSTER_ID=j-...` has something to\n   download for postmortems). All three are gitignored, so your bucket\n   never lands in a commit. `make setup` refuses to overwrite an\n   existing file — delete it first to regenerate.\n\n   To do it by hand instead, `cp` each `*.example` to its real name and\n   replace `YOUR-BUCKET`. Every Make target reads `cc-lint.defaults.mk`\n   first, then your `cc-lint.mk` overrides (or any\n   `CONFIG=/path/to/another.mk`).\n\n3. Build and upload the dependency wheel bundle. The bootstrap\n   installs packages from `/tmp/wheels` on each EMR node with\n   `--no-index`, so no PyPI traffic happens during a job:\n\n   ```bash\n   make wheels         # builds wheels in ./wheels via amazonlinux:2023 docker\n   make upload-wheels  # sync ./wheels to $(WHEEL_S3_PATH)\n   ```\n\n4. Cache the Tranco top-sites CSV locally — it is uploaded with each\n   job via `--files`:\n\n   ```bash\n   make tranco-cache\n   ```\n\n5. *(Optional)* Build the IP-to-ASN table for infrastructure\n   fingerprinting. Every report fingerprints the CDN / server /\n   framework behind each response from signal headers; supplying a\n   CAIDA pfx2as snapshot additionally resolves the crawl-time\n   `WARC-IP-Address` to an ASN, which catches CDNs that strip\n   identifying headers. Set `IPASN_V4_URL` (and optionally\n   `IPASN_V6_URL`) in your config to a snapshot near the crawl month,\n   then:\n\n   ```bash\n   make ipasn-cache\n   ```\n\n   Once built, the table is shipped to mappers (and used by the local\n   lint) automatically. Without it, fingerprinting is header-only.\n\n### Smoke test\n\n`make test-emr` runs the full pipeline against\n`tests/fixtures/warc.paths.txt` with a small instance fleet\n(`mrjob-test.conf`) and `LIMIT=1` so only a single WARC is processed.\nUse it to validate AWS plumbing, wheel availability, and bootstrap\ncorrectness before a full run:\n\n```bash\nmake test-emr\n```\n\nSuccessful runs land in `results/test-\u003cRUN_ID\u003e/` with `part-*`\nrecords from EMR plus the rendered `report.html` and `report.md`.\n\n### Full run\n\n```bash\nmake emr\n```\n\nThis pipeline does, in order:\n\n1. `cc_lint.emr.split_paths` reads\n   `s3://commoncrawl/crawl-data/$(CRAWL_ID)/warc.paths.gz` (requester\n   pays handled automatically) and uploads `MAP_TASKS` chunk files to\n   `$(PATHS_PREFIX)$(CRAWL_ID)-$(RUN_ID)/`.\n2. `cc_lint.emr.job` runs on EMR with `mrjob.conf` and\n   `REDUCES` reducers. Each mapper forks a child per WARC path,\n   pickles a `StatsCollector` snapshot back, and merges into a\n   single per-mapper dict.\n3. `aws s3 sync` pulls the reducer output into\n   `results/$(CRAWL_ID)-$(RUN_ID)/`.\n4. `cc_lint.emr.finalize` merges the sharded reducer records from the\n   `part-*` files — `globals`, per-note `note:*`, and the per-dimension\n   shards (`csp_sizes`, `vary`, `cache_control`, `value_histograms`,\n   `cooccur`, `note_cooccur`, `transition`) — and renders `report.html`\n   + `report.md`.\n\n### Re-rendering an existing run\n\nIf you already have a `results/\u003crun-name\u003e/` directory with `part-*`\nfiles synced from S3, regenerate the report without rerunning EMR:\n\n```bash\nmake report RESULTS_DIR=results/CC-MAIN-2026-12-20260520-101500\n# or:\nmake results/CC-MAIN-2026-12-20260520-101500/report.html\n```\n\n### Diagnosing slow or failed mappers\n\nIf you preserved EMR logs (`MRJOB_CLEANUP=NONE` or via the EMR\nconsole), pass the cluster id to surface per-WARC timings and\nfailures:\n\n```bash\nmake emr-timing EMR_LOG_CLUSTER_ID=j-XXXXXXXX\n```\n\nThis downloads `stderr.gz` from S3, parses the structured\n`INFO: finished WARC ...` lines emitted by the mapper, and prints a\nMarkdown summary of total/process/iterator times, top-N slow WARCs,\nand any child-process failures.\n\n### Configuration reference\n\n`make show-config` prints the effective values. Important knobs:\n\n| Variable | Purpose |\n| --- | --- |\n| `CRAWL_ID` | Common Crawl release to process (e.g. `CC-MAIN-2026-12`) |\n| `TOP_N`, `LOCAL_TOP_N` | Tranco top-N filter for full / local runs |\n| `SAMPLE_TOP_N` | Tranco ceiling for collecting per-note sample URLs |\n| `RECORD_LIMIT` | Max records per WARC (0 = all) |\n| `MAP_TASKS`, `REDUCES` | Full-run cluster sizing |\n| `TEST_MAP_TASKS`, `TEST_REDUCES`, `LIMIT` | Smoke-test sizing |\n| `OUTPUT_DIR`, `PATHS_PREFIX`, `WHEEL_S3_PATH` | S3 locations |\n| `MRJOB_CLEANUP` | Set to `NONE` to keep cluster + logs for postmortem |\n\n---\n\n## Contributing\n\nDevelopment setup, the test / typecheck / lint workflow, commit\nconventions, and a tour of the project layout live in\n[CONTRIBUTING.md](CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnot%2Fcc-lint","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmnot%2Fcc-lint","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnot%2Fcc-lint/lists"}