{"id":51701468,"url":"https://github.com/mnot/feed-survey","last_synced_at":"2026-07-16T11:34:25.754Z","repository":{"id":357033063,"uuid":"1235012170","full_name":"mnot/feed-survey","owner":"mnot","description":"Looking at how the Web uses feeds","archived":false,"fork":false,"pushed_at":"2026-06-28T12:02:30.000Z","size":10481,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-16T11:34:24.113Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://projects.mnot.net/feed-survey/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mnot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-10T23:28:08.000Z","updated_at":"2026-06-13T05:27:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mnot/feed-survey","commit_stats":null,"previous_names":["mnot/feed-survey"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mnot/feed-survey","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Ffeed-survey","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Ffeed-survey/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Ffeed-survey/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Ffeed-survey/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mnot","download_url":"https://codeload.github.com/mnot/feed-survey/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnot%2Ffeed-survey/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35542951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-16T02:00:06.687Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-16T11:34:24.941Z","updated_at":"2026-07-16T11:34:25.747Z","avatar_url":"https://github.com/mnot.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Feed Survey (`feed-survey`)\n\nA high-performance, distributed survey of RSS/Atom feed usage, autodiscovery, and quality in Common Crawl using AWS EMR.\n\n## Overview\n\n`feed-survey` uses MapReduce to process Common Crawl WARC files in parallel. It measures feed autodiscovery, parses RSS/Atom feed candidates, and renders HTML and Markdown reports from the aggregated results.\n\nThe HTML report is intended for visual exploration. The Markdown sibling uses\nplain sections and tables so the same run can be re-read, compared, or ingested\nby analysis tools without scraping the visual report.\n\n## Key Features\n\n- **Distributed MapReduce**: Built on `mrjob` for seamless scaling from a few instances to hundreds of nodes on AWS EMR.\n- **Python 3.12 on EMR**: Uses modern Python syntax and efficient libraries (`fastwarc`, `lxml`) for maximum throughput.\n- **Automatic Result Sync**: The build system automatically syncs results from S3 back to your local machine upon completion.\n- **Tranco Filtering**: Built-in support for filtering analysis to the Tranco Top-1M, using Tranco's subdomain-inclusive list by default and Public Suffix List site normalization.\n- **Platform Fingerprints**: Conservative CMS/framework hints from HTML pages, feed headers, and feed generator elements, with report-time quality comparisons.\n- **OPML Feed-List Reports**: Local reporting for a user's own OPML subscription list, using the same feed parsing, quality, autodiscovery, and HTML/Markdown report machinery as crawl reports.\n\n## Install\n\nFor the standalone [CLI tools](#cli-tools) (`feed-survey-probe`,\n`feed-survey-opml`), the base install is lightweight and pipx-friendly:\n\n```bash\npipx install feed-survey\n```\n\nThe Common Crawl / EMR pipeline is driven by `make`, not the installed CLI:\nit needs the repository's `Makefile`, `mrjob.conf`, and local `feed-survey.mk`\nconfig. To run it, clone the repo and follow [Quick Start (EMR)](#quick-start-emr)\nbelow; `make venv` installs the `[dev]` extra, which pulls in the `[emr]`\nextra automatically.\n\n## CLI Tools\n\nTwo standalone commands ship with the base install and don't require AWS,\nCommon Crawl, or any local config. Both reuse the same parser, quality\nscoring, autodiscovery, and HTML/Markdown report machinery as the EMR\npipeline.\n\n### `feed-survey-probe` — diagnose one URL\n\nFetch a single URL and print Markdown diagnostics. HTML responses report\nRSS/Atom autodiscovery links; feed responses report parser output,\nlanguage/date/content signals, extensions, fingerprints, and the operational\nquality score.\n\n```bash\n# A single feed\nfeed-survey-probe https://example.com/feed.xml\n\n# An HTML page, plus the feeds it advertises\nfeed-survey-probe --recursive https://example.com/\n```\n\nRecursive probing follows only RSS/Atom autodiscovery links, capped at 10\nunique feeds by default (`--max-feeds N` to change).\n\n### `feed-survey-opml` — report on an OPML subscription list\n\nTurn an OPML subscription file into a full HTML and Markdown report. Useful\nfor \"how healthy are the feeds I subscribe to?\" and ecosystem audits, without\ntouching Common Crawl.\n\n```bash\nfeed-survey-opml subscriptions.opml --output feeds-report.html\n```\n\nOPML `xmlUrl` values are the primary feed inputs. When an outline also has\n`url` or `htmlUrl`, that page is fetched as HTML and reported with its\nautodiscovery properties so the report can distinguish explicitly-listed\nfeeds from feeds the linked site advertises. Pass `--skip-html` to fetch\nonly the `xmlUrl` feeds, `-q`/`--quiet` to silence progress, `--concurrency\nN` to tune parallelism (default 32), and `--max-bytes N` to change the 10\nMiB per-fetch cap (`0` to disable).\n\nThe same command is also available through make:\n\n```bash\nmake opml-report OPML=subscriptions.opml OPML_REPORT=feeds-report.html\n```\n\n## Quick Start (EMR)\n\n### 1. Prerequisites\n- **AWS CLI**: Installed and configured (`aws configure`).\n- **EMR Roles**: Create the default roles once per account:\n  ```bash\n  aws emr create-default-roles\n  ```\n- **Local Cache**: The tool requires the Tranco list locally to upload to workers. `make emr` and `make test-emr` populate it automatically when missing; to do it explicitly:\n  ```bash\n  make tranco-cache\n  ```\n\n### 2. Local Setup\n```bash\n# Clone and set up a virtual environment\ngit clone https://github.com/mnot/feed-survey.git\ncd feed-survey\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -e \".[dev]\"   # includes the emr extra\n\n# Create your local run configuration before using EMR targets.\ncp feed-survey.example.mk feed-survey.mk\n```\n\n### Local Usage\nYou can run the analysis on your own machine for debugging. This uses the `local` runner and does not require AWS.\n\n```bash\n# Run a one-WARC local analysis and render test_report.html and test_report.md.\nmake local-report\n```\n\nFor ad-hoc URL/OPML inspection, see the [CLI Tools](#cli-tools) section\nabove — those commands work from the same checkout.\n\n### 3. Run a Smoke Test (EMR)\nThe `test-emr` target runs a single WARC file through a small EMR cluster to verify your AWS environment is ready.\n```bash\nmake test-emr\n```\nTo run a larger sample, set `LIMIT`, e.g. `make test-emr LIMIT=50`.\n*Results will be automatically downloaded to `results/test-XXXXXXXX/`.*\n\n### 4. Run the Full Crawl\nOnce validated, launch the full analysis across the current Common Crawl index.\n```bash\nmake emr\n```\n\n## Configuration\n\n### Make Configuration\n`feed-survey.defaults.mk` contains safe defaults for local development and\nnon-secret tuning. `feed-survey.mk` is your local, ignored configuration file\nfor AWS buckets and account-specific choices. Create it from the example:\n\n```bash\ncp feed-survey.example.mk feed-survey.mk\n```\n\nEdit `feed-survey.mk`, or pass another make fragment with\n`CONFIG=/path/to/config.mk`.\n\nRun `make show-config` to print the effective settings before starting an EMR\nrun.\n\n- **`CRAWL_ID`**: The Common Crawl index to process.\n- **`TOP_N`**: Tranco cutoff for EMR runs, applied to registrable sites after Public Suffix List normalization. Private suffixes such as `blogspot.com` and `github.io` make hosted sub-sites count independently.\n- **`TRANCO_LIST`**: Tranco ranking flavor for `TOP_N` scoping. Defaults to `subdomains`, which uses Tranco's list with subdomains included before normalizing to registrable sites. Set `TRANCO_LIST=standard` to use Tranco's domain-only Top-1M.\n- **`OUTPUT_DIR` / `PATHS_PREFIX` / `WHEEL_S3_PATH`**: S3 locations for EMR results, split WARC path inputs, and dependency wheels.\n- **`MAP_TASKS` / `REDUCES`**: Full-run map chunking and reducer count.\n- **`TEST_MAP_TASKS` / `TEST_REDUCES`**: Smoke-test map chunking and reducer count.\n- **`MRJOB_CONFIG` / `MRJOB_TEST_CONFIG`**: mrjob cluster configuration files.\n- **`MRJOB_CLEANUP`**: mrjob cleanup mode after successful EMR runs. Defaults to `TMP`, which removes temporary working data but keeps logs available for timing/debugging. Set `MRJOB_CLEANUP=ALL` to restore mrjob's default successful-run cleanup.\n- **`EMR_LOG_CLUSTER_ID` / `EMR_LOG_DIR`**: Inputs for `make emr-timing`, which downloads preserved mapper stderr logs and summarizes WARC timing counters.\n- **`TRANCO_CACHE_DIR`**: Local cache directory used by `make tranco-cache`; the selected Tranco CSV is normalized to registrable sites once locally and uploaded to EMR workers as `top-1m-sites.csv`.\n- **`MOCK_REPORT` / `RESULTS_DIR`**: Local report output and re-render inputs.\n\n### `mrjob.conf`\nControl EMR cluster shape and instance types. The make targets supply bootstrap\ncommands, dependency-wheel location, and the Tranco upload file from the make\nconfiguration.\n\n- **`TargetOnDemandCapacity`**: The default full run uses 30 core xlarge instances plus one master, leaving a little headroom below a 128 vCPU on-demand quota.\n- **`instance_fleets`**: Defines the mix of m5, r5, and c5 instances EMR can choose from.\n\n### `Makefile`\nThe Makefile is the command surface. It loads `feed-survey.defaults.mk`, then\noptionally loads `feed-survey.mk` or the file named by `CONFIG=...`. Generated\nreports stay under `results/` unless a target explicitly writes a local scratch\nreport.\n\nRun `make help` for the local development, report, EMR, and wheel targets.\n\n## Project Structure\n\n- `feed_survey/emr/`: EMR orchestration, WARC input, and MapReduce wire-format code.\n- `feed_survey/analysis/`: Core logic for parsing WARC records and extracting feed metadata.\n- `feed_survey/report/`: Report-time aggregation, quality scoring, and HTML/Markdown rendering.\n- `feed_survey/probe.py`: Single-URL Markdown diagnostics for feeds and HTML autodiscovery.\n- `feed_survey/opml.py`: OPML input path for local feed-list reports.\n- `feed_survey/commoncrawl.py`: Common Crawl metadata and WARC path discovery.\n- `feed_survey/tranco.py`: Tranco list loading for top-site scoping.\n- `feed_survey/url.py`: URL normalization, host extraction, and registrable-site helpers.\n- `feed_survey/download.py`: Shared download and cache helpers.\n- `tests/`: Unit tests and integration tests.\n- `tests/fixtures/`: Small local fixtures and profiling helpers used by tests and smoke runs.\n- `docs/`: Research notes and plans for future analysis dimensions.\n- `feed-survey.defaults.mk`: Tracked make defaults for crawl, EMR sizing, and cache paths.\n- `feed-survey.example.mk`: Example local configuration with placeholder S3 paths.\n- `mrjob.conf`: EMR orchestration settings (Python 3.12, dependencies, instance fleets).\n- `.mrjobignore`: Prevents local virtual environments and caches from being uploaded to workers.\n\n## Local Development \u0026 Testing\n\nUseful local development targets:\n\n```bash\nmake test\nmake typecheck\nmake lint\nmake mock-report\nmake check\nmake emr-timing EMR_LOG_CLUSTER_ID=j-...\n```\n\n## Cost Notes\n\nRuntime and cost depend on the selected crawl, EMR instance mix, regional pricing,\nspot availability, and whether slow WARC files dominate the tail of the job. Use\n`make test-emr LIMIT=\u003cn\u003e` to check throughput in your account before starting a\nfull run, and use the EMR console or Cost Explorer for current pricing.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnot%2Ffeed-survey","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmnot%2Ffeed-survey","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnot%2Ffeed-survey/lists"}