{"id":15649179,"url":"https://github.com/edjopato/website-stalker","last_synced_at":"2026-06-13T02:01:06.914Z","repository":{"id":36955820,"uuid":"367358964","full_name":"EdJoPaTo/website-stalker","owner":"EdJoPaTo","description":"Track changes on websites via git","archived":false,"fork":false,"pushed_at":"2026-06-13T00:03:53.000Z","size":920,"stargazers_count":71,"open_issues_count":6,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-06-13T01:19:36.770Z","etag":null,"topics":["change-alert","change-detection","git","monitoring","scraper","self-hosted","url-monitor","web-scraping","website-change-detector","website-change-monitor","website-change-tracker","website-monitor","website-monitoring"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-2.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EdJoPaTo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-05-14T12:41:25.000Z","updated_at":"2026-06-12T23:50:32.000Z","dependencies_parsed_at":"2024-11-16T05:06:03.486Z","dependency_job_id":"7389b999-a58a-4933-b29d-837088adb526","html_url":"https://github.com/EdJoPaTo/website-stalker","commit_stats":{"total_commits":585,"total_committers":5,"mean_commits":117.0,"dds":0.3401709401709402,"last_synced_commit":"e42ccda7534731d9c2b7701a49e5b6cd75b67cb5"},"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"purl":"pkg:github/EdJoPaTo/website-stalker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EdJoPaTo%2Fwebsite-stalker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EdJoPaTo%2Fwebsite-stalker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EdJoPaTo%2Fwebsite-stalker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EdJoPaTo%2Fwebsite-stalker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EdJoPaTo","download_url":"https://codeload.github.com/EdJoPaTo/website-stalker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EdJoPaTo%2Fwebsite-stalker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34269364,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-13T02:00:06.617Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["change-alert","change-detection","git","monitoring","scraper","self-hosted","url-monitor","web-scraping","website-change-detector","website-change-monitor","website-change-tracker","website-monitor","website-monitoring"],"created_at":"2024-10-03T12:28:32.846Z","updated_at":"2026-06-13T02:01:06.902Z","avatar_url":"https://github.com/EdJoPaTo.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Website Stalker\n\n\u003e Track changes on websites via git\n\nThis tool checks all the websites listed in its config.\nWhen a change is detected, the new site is added to a git commit.\nIt can then be inspected via normal git tooling.\n\nBasically it's `curl`, [`sed`++](#editors) and then `git commit` in a neat package.\n\nSee it [in action](https://github.com/EdJoPaTo/website-stalker-example) (literally in GitHub **Action**s).\n\n## Install\n\n- [GitHub Releases](https://github.com/EdJoPaTo/website-stalker/releases)\n- [Arch Linux User Repository (AUR)](https://aur.archlinux.org/packages/website-stalker/)\n- [Docker Hub Image](https://hub.docker.com/r/edjopato/website-stalker)\n- Via rust and cargo: Clone → `cargo install --path .`\n\n## Usage\n\n### GitHub Actions\n\nCheck out [website-stalker-example](https://github.com/EdJoPaTo/website-stalker-example) which runs within GitHub actions.\n\n### Locally\n\n- First create a new folder / git repository for tracking website changes\n\n    ```bash\n    mkdir personal-stalker\n    cd personal-stalker\n    git init\n    website-stalker example-config \u003e website-stalker.yaml\n    ```\n\n- Add your favorite website to the configuration file `website-stalker.yaml`.\n    Also make sure to set the value of [from](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/From) to an email address of yours.\n\n    ```bash\n    website-stalker example-config \u003e website-stalker.yaml\n    nano website-stalker.yaml\n    ```\n\n- Run your newly added website. If you added `https://apple.com/newsroom` use something like this to test if everything works like you want:\n\n    ```bash\n    website-stalker run apple\n    ```\n\n- Set up a cronjob / [`systemd.timer`](systemd) executing the following command occasionally\n\n    ```bash\n    website-stalker run --all --commit\n    ```\n\n### Config Example\n\nThe config describes a list of sites.\nEach site has a URL.\nAdditionally, each site can have editors which are used before saving the file.\nEach [editor](#editors) manipulates the content of the URL.\n\n```yaml\n# This is an example config\n# The filename should be `website-stalker.yaml`\n# and it should be in the working directory where you run website-stalker.\n#\n# For example run `website-stalker example-config \u003e website-stalker.yaml`.\n# Adapt the config to your needs and set the FROM email address which is used as a request header:\n# https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/From\n#\n# And then do a run via `website-stalker run --all`.\n---\nfrom: my-email-address\nsites:\n  - url: \"https://edjopato.de/post/\"\n    editors:\n      - css_select: article\n      - css_remove: a\n      - html_prettify\n      - regex_replace:\n          pattern: \"(Lesezeit): \\\\d+ \\\\w+\"\n          replace: $1\n  - url: \"https://edjopato.de/robots.txt\"\n```\n\nThere is a bigger [config](https://github.com/EdJoPaTo/website-stalker-example/blob/main/website-stalker.yaml) in my [example repository](https://github.com/EdJoPaTo/website-stalker-example).\nThe example repository is also used by me to detect changes of interesting sites.\n\n### Global Options\n\nOptions which are globally configured at the root level of the configuration file `website-stalker.yaml`.\n\n#### `from`\n\nUsed as the [`From` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/From) in the web requests.\nIt is a required field.\n\nThe idea here is to provide a way for a website host to contact whoever is doing something to their web server.\nAs this tool is self-hosted and can be run as often as the user likes this can annoy website hosts.\nWhile this tool is named \"stalker\" and is made to track websites it is not intended to annoy people.\n\nThis tool sets the [`User-Agent` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent) to `website-stalker/\u003cversion\u003e https://github.com/EdJoPaTo/website-stalker` and the [`From` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/From) to the user configured value.\nThis way both the creator and the user of this tool can be reached in case of problems.\n\n```yaml\nfrom: my-email-address\n```\n\nAlternatively you can specify FROM via environment variable\n\n```bash\nexport WEBSITE_STALKER_FROM=my-email-address\n```\n\n### Per Site Options\n\nOptions available per site besides the [editors](#editors) which are explained below.\n\n#### `url`\n\nOne or multiple URLs can be specified.\nThe simple form is a single URL:\n\n```yaml\nsites:\n  - url: \"https://edjopato.de/\"\n  - url: \"https://edjopato.de/post/\"\n```\n\nIt's also possible to specify multiple URL at the same time.\nThis is helpful when multiple sites are sharing the same options (like editors).\n\n```yaml\nsites:\n  - url:\n      - \"https://edjopato.de/\"\n      - \"https://edjopato.de/post/\"\n```\n\n#### `accept_invalid_certs`\n\nAllows HTTPS connections with self-signed or invalid / expired certificates.\n\nFrom [`reqwests` documentation](https://docs.rs/reqwest/0.11.26/reqwest/struct.ClientBuilder.html#method.danger_accept_invalid_certs):\n\n\u003e You should think very carefully before using this method. If\n\u003e invalid certificates are trusted, *any* certificate for *any* site\n\u003e will be trusted for use. This includes expired certificates. This\n\u003e introduces significant vulnerabilities, and should only be used\n\u003e as a last resort.\n\nDo you have a need for self-signed certificates or the usage of the system certificate store?\nPlease share about it in [Issue #39](https://github.com/EdJoPaTo/website-stalker/issues/39).\n\n```yaml\nsites:\n  - url: \"https://edjopato.de/post/\"\n    accept_invalid_certs: true\n```\n\n#### `http1_only`\n\nOnly use HTTP/1 for the web request.\n\nBack-ends might use HTTP/2 fingerprinting which could result in different or unusable output depending on what the back-end assumes about the client.\nHTTP/1 is a simpler protocol which does not allow such kinds of back-end optimizations.\n\n```yaml\nsites:\n  - url: \"https://edjopato.de/post/\"\n    http1_only: true\n```\n\n#### `ignore_error`\n\nOnly show warning when the site errors.\n\nThis is useful for buggy services which are sometimes just gone or for pages which will exist in the future but are not there yet.\nPersonal example: A bad DNS configuration which lets the website appear nonexistent for some time.\n\nThis setting also skips errors from editors.\n\n```yaml\nsites:\n  - url: \"https://edjopato.de/might-appear-in-the-future\"\n    ignore_error: true\n```\n\n#### `filename`\n\nOverrides the URL based default filename of the site.\n\nNormally the filename is automatically derived from the URL.\nFor the following example it would be something like `de-edjopato-api-token-0123456789-action-enjoy-20weather.html`.\nWith the `filename` options it is saved as `de-edjopato-api-weather.html` instead.\n\n```yaml\nsites:\n  - url: \"https://edjopato.de/api?token=0123456789\u0026action=enjoy%20weather\"\n    filename: de-edjopato-api-weather\n```\n\n#### `headers`\n\nAdd additional [HTTP headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) to the request to the given site.\n\nThis is useful for sites that respond differently based on different headers.\nEach header Key/Value pair is supplied as YAML String separated with a `:` followed by a space.\n\nThis is the same syntax as HTTP uses which sadly collides with YAML.\nYAML assumes something with a `:` is an object.\nTherefor you have to make sure to quote the headers.\nUsing a YAML object / key/value pair is also not possible as some header keys are allowed multiple times.\n\n```yaml\nsites:\n  - url: \"https://edjopato.de/\"\n    headers:\n      - \"Cache-Control: no-cache\"\n      - \"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0\"\n```\n\n### Editors\n\nEditors are manipulating the content of a webpage to simplify comparing them later on.\n\nFor example: If you are interested in the content of a webpage the `\u003chead\u003e` with changing style-sheets isn't interesting to you.\nWhen keeping it, it will still create diffs which end up being commits.\nThis will create noise you're probably just going to ignore.\nThat's why editors exist.\n\nThink of editors like a pipeline, the next one gets the input of the one before.\nAs some editors are assuming HTML input, they won't work (well) with non HTML input.\nFor example its kinda useless to use `html_prettify` after `html_textify` as text won't end up being pretty HTML.\nFor this reason editors like `css_select` are still producing valid HTML output.\n\nThere are probably more tasks out there that might be useful as editors.\nFeel free to provide an issue for an editor idea or create a Pull Request with a new editor.\n\n#### `css_flatten`\n\nReplaces every matching HTML element with its child nodes and returns the HTML.\nInstead of [`css_remove`](#css_remove) this does not remove all the child nodes below.\n\nExamples:\n\n```yaml\neditors:\n  - css_flatten: div\n  - css_flatten: a[href^=\"#\"] # flatten all local links away (starting with a #)\n```\n\n#### `css_remove`\n\nTries to remove every instance of matching HTML elements and returns the remaining HTML.\nOpposite of [`css_select`](#css_select).\nWhen the child nodes should be kept, use [`css_flatten`](#css_flatten).\n\nExamples:\n\n```yaml\neditors:\n  - css_remove: article\n  - css_remove: h1 a\n  - css_remove: h1 \u003e a\n```\n\n#### `css_select`\n\nUse [CSS Selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors) to grab every instance of matching HTML elements and returns all of them.\n\nIf no matching HTML elements are found, this editor errors.\n\nExamples:\n\n```yaml\neditors:\n  - css_select: article\n  - css_select: h1 a\n  - css_select: h1 \u003e a\n```\n\n#### `css_sort`\n\nSort elements matching to the given [CSS Selector](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors).\nOther elements not matching are kept.\nElements below different parents are sorted independently.\n\nBasic example:\n\n```html\n\u003cdiv\u003e\u003cp\u003eC\u003c/p\u003e\u003cp\u003eB\u003c/p\u003e\u003c/div\u003e\n\u003cdiv\u003e\u003cp\u003eD\u003c/p\u003e\u003cp\u003eA\u003c/p\u003e\u003c/div\u003e\n```\n\nwith `p` as the selector will sort into this:\n\n```html\n\u003cdiv\u003e\u003cp\u003eB\u003c/p\u003e\u003cp\u003eC\u003c/p\u003e\u003c/div\u003e\n\u003cdiv\u003e\u003cp\u003eA\u003c/p\u003e\u003cp\u003eD\u003c/p\u003e\u003c/div\u003e\n```\n\nExamples:\n\n```yaml\neditors:\n  # Sort all articles\n  - css_sort:\n      selector: article\n```\n\nThe above example sorts by the whole element ([`outerHTML`](https://developer.mozilla.org/en-US/docs/Web/API/Element/outerHTML)).\nIn order to sort by something specific for a given HTML element, editors can be used.\n\n```yaml\neditors:\n  # Sort articles by their heading\n  - css_sort:\n      selector: article\n      sort_by: # the specified editors are applied to every selected HTML element independently\n        - css_select: h2\n```\n\nThis might still sort in surprising ways as things like attributes are still included (`\u003ch2 class=\"a\"\u003eZ\u003c/h2\u003e` is sorted before `\u003ch2 class=\"z\"\u003eA\u003c/h2\u003e`).\nTherefore, editors like [`html_textify`](#html_textify) or [`html_sanitize`](#html_sanitize) are likely a good idea to be used in `sort_by`.\n\nTip: [`debug_files`](#debug_files) can help you understand what is happening. But don't forget to remove it after you are done testing:\n\n```yaml\neditors:\n  - css_sort:\n      selector: article\n      sort_by:\n        - css_select: h2\n        - html_sanitize\n        - debug_files: /tmp/website-stalker/\n```\n\nYou can also reverse the sorting:\n\n```yaml\neditors:\n  - css_sort:\n      selector: article\n      reverse: true\n```\n\n#### `css_tag_replace`\n\nReplace [HTML tags](https://developer.mozilla.org/en-US/docs/Glossary/Tag) matching a given [CSS Selector](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors).\n\nFor example, the following config will replace all `h3` tags with `h2` tags.\n\n```yaml\neditors:\n  - css_tag_replace:\n      selector: h3\n      replace: h2\n```\n\n```diff\n \u003chtml\u003e\n \u003chead\u003e\u003c/head\u003e\n \u003cbody\u003e\n-  \u003ch3 class=\"green\"\u003e\n+  \u003ch2 class=\"green\"\u003e\n     Hello\n-  \u003c/h3\u003e\n+  \u003c/h2\u003e\n   World\n \u003c/body\u003e\n \u003c/html\u003e\n```\n\nThis can be helpful to ensure some kind of structure especially when editors like [`html_markdownify`](#html_markdownify) are used.\nThink about a website where only some subsections are of interest and selected via the [`css_select`](#css_select).\nWhile the `header` contains some `h1` the selected part skips the `h2` headings and continues with `h3` headings.\nAlso, `\u003cstrong\u003e` are (incorrectly) used as subheadings.\nParsing this to Markdown results in less optimal structure as `h2` are skipped and `\u003cstrong\u003e` don't result in headings.\nIdeal would be a single `h1` and then continuous depending on the depth `h2`, `h3` and so on.\nThe following can help with that:\n\n```yaml\neditors:\n  # Select the header and some interesting sections\n  - css_select: header, main section.interesting\n  # First migrate the h3 tags to h2 so there is no gap\n  - css_tag_replace:\n      selector: h3\n      replace: h2\n  # Then migrate all strong tags to proper headings\n  - css_tag_replace:\n      selector: strong\n      replace: h3\n  # In the end parse to Markdown\n  - html_markdownify\n```\n\n#### `debug_files`\n\nThis editor passes its input through without modifying it.\nThe content is written to a file in the given directory.\nThe filename is created from the current UNIX Timestamp.\n\nThis is neat when looking at steps in between editors is of interest.\nEspecially for editors like [RSS](#rss) which use editors per item this can be handy to look at the steps in between.\n\nWarning: It's not recommended committing these files.\n`debug_files` should be removed before when committing the config.\nIt might have unintended side effects or might spam your repository with many potentially large files.\n\nExamples:\n\n```yaml\neditors:\n  - debug_files: /tmp/website-stalker/\n```\n\n#### `html_markdownify`\n\nFormats the input HTML as Markdown.\n\nExample:\n\n```yaml\neditors:\n  - html_markdownify\n```\n\n#### `html_prettify`\n\nFormats the input HTML as pretty HTML.\n\nExample:\n\n```yaml\neditors:\n  - html_prettify\n```\n\n#### `html_sanitize`\n\nStrip down HTML to its minimal form.\n\nExample:\n\n```yaml\neditors:\n  - html_sanitize\n```\n\n#### `html_textify`\n\nOnly returns text content of HTML elements within the input.\n\nExample:\n\n```yaml\neditors:\n  - html_textify\n```\n\n#### `html_url_canonicalize`\n\nParses the input HTML for URLs.\nURLs are parsed into their canonical, absolute form.\n\nExample:\n\n```yaml\neditors:\n  - html_url_canonicalize\n```\n\n#### `json_prettify`\n\nFormats the input JSON as pretty JSON.\n\nExample:\n\n```yaml\neditors:\n  - json_prettify\n```\n\n### `json_simple_select`\n\nSelects from an input JSON to narrow it down.\nInspired by [`jq`](https://jqlang.github.io/jq/) but with only very basic selection support.\n\nDue to the current, naive selector implementation it might support syntax `jq` doesnt support.\nSyntax that works with this editor and `jq` is intended behaviour.\nSyntax that only works with this editor and not `jq` is considered a bug and might change its behaviour on any release.\n\nThis editor is intentionally kept simple and does not intend to support the whole `jq` featureset.\nBut it does not mean it should stay in its current state.\nIf you think it should support other simple syntax of `jq`, feel free to create an issue or pull request and then lets discuss about it.\n\nExample:\n\n```yaml\neditors:\n  - json_simple_select: .foo[2].bar\n```\n\n#### `regex_replace`\n\nSearches the input with a Regex pattern and replaces all occurrences with the given replace phrase.\nGrouping and replacing with `$1` also works.\n\nExamples:\n\n```yaml\neditors:\n  # Remove all occurrences of that word\n  - regex_replace:\n      pattern: \"tree\"\n      replace: \"\"\n  # Remove all numbers\n  - regex_replace:\n      pattern: \"\\\\d+\"\n      replace: \"\"\n  # Find all css files and remove the extension\n  - regex_replace:\n      pattern: \"(\\\\w+)\\\\.css\"\n      replace: $1\n```\n\n#### `rss`\n\nCreates an RSS 2.0 Feed from the input.\nAn RSS item is generated for every `item_selector` result.\nThe other selectors can be used to find relevant information of the items.\nThe content is the full result of the `item_selector`.\nIt can be further edited with every available [editor](#editors).\n\nDefaults:\n\n- `title`: When a `\u003ctitle\u003e` exists, it will be used. Otherwise, it's empty.\n- `item_selector`: `article`\n- `title_selector`: `h2`\n- `link_selector`: `a`\n- `content_editors` can be omitted when empty\n\nExamples:\n\n```yaml\n  # Fully specified example\n  - url: \"https://edjopato.de/post/\"\n    editors:\n      - rss:\n          title: EdJoPaTos Blog\n          item_selector: article\n          title_selector: h2\n          link_selector: a\n          content_editors:\n            - css_remove: \"h2, article \u003e a, div\"\n            - html_textify\n\n  # Minimal working example\n  - url: \"https://edjopato.de/post/\"\n    editors:\n      - rss: {}\n```\n\n## Alternatives\n\n- [Website Changed Bot](https://github.com/EdJoPaTo/website-changed-bot) is a Telegram Bot which might potentially use this tool later on\n- [bernaferrari/ChangeDetection](https://github.com/bernaferrari/ChangeDetection) is an Android app for this\n- [dgtlmoon/changedetection.io](https://github.com/dgtlmoon/changedetection.io) can be self-hosted and configured via web interface\n- [Feed me up, Scotty!](https://gitlab.com/vincenttunru/feed-me-up-scotty) creates RSS feeds from websites\n- [htmlq](https://github.com/mgdm/htmlq) command line tool to format / select HTML (like `jq` for HTML)\n- [urlwatch](https://thp.io/2008/urlwatch/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedjopato%2Fwebsite-stalker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedjopato%2Fwebsite-stalker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedjopato%2Fwebsite-stalker/lists"}