{"id":16318104,"url":"https://github.com/scorphus/walt","last_synced_at":"2026-04-27T08:32:25.188Z","repository":{"id":54516225,"uuid":"336866060","full_name":"scorphus/walt","owner":"scorphus","description":"🕵️‍♂️ Website Availability Monitor","archived":false,"fork":false,"pushed_at":"2021-02-13T21:43:37.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-13T17:16:41.659Z","etag":null,"topics":["kafka","measurements","monitoring","network","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scorphus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-07T19:01:43.000Z","updated_at":"2021-02-13T21:39:45.000Z","dependencies_parsed_at":"2022-08-13T18:20:51.578Z","dependency_job_id":null,"html_url":"https://github.com/scorphus/walt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/scorphus/walt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scorphus%2Fwalt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scorphus%2Fwalt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scorphus%2Fwalt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scorphus%2Fwalt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scorphus","download_url":"https://codeload.github.com/scorphus/walt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scorphus%2Fwalt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32329463,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T23:26:28.701Z","status":"online","status_checked_at":"2026-04-27T02:00:06.769Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","measurements","monitoring","network","postgresql"],"created_at":"2024-10-10T22:09:59.270Z","updated_at":"2026-04-27T08:32:25.164Z","avatar_url":"https://github.com/scorphus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# walt [![Build Status][build-badge]][action-link] [![Coverage Status][codecov-badge]][codecov-link] [![Maintainability][codeclimate-badge]][codeclimate-link] [![Code Quality][codacy-badge]][codacy-link]\n\nwalt — Website Availability Monitor\n\nCheck target websites and keep all verification results on your database! walt\ncoordinates data flow with Kafka and stores results on Postgres.\n\nwalt takes a collection of URLs and their optional regular expression patterns\nand checks for:\n\n-   HTTP response time\n-   HTTP status code\n-   Occurrence of regexp pattern in the page\n\nVerification results are either a `result`:\n\n    walt=\u003e SELECT * FROM result LIMIT 10;\n     result_id |                    url                    |    response_time    | status_code |  pattern  |         timestamp          \n    -----------+-------------------------------------------+---------------------+-------------+-----------+----------------------------\n             1 | https://duckduckgo.com/?q=walt            | 0.08444564199999993 |         200 | FOUND     | 2021-02-07 21:24:24.583+00\n             2 | https://www.google.com/search?q=walt      |  1.8681911579999997 |         200 | FOUND     | 2021-02-07 21:24:26.378+00\n             3 | https://duckduckgo.com/?q=doge+meme       | 0.01953169999999993 |         200 | NOT_FOUND | 2021-02-07 21:24:26.606+00\n             4 | https://duckduckgo.com/?q=walt            | 0.03456038500000069 |         200 | FOUND     | 2021-02-07 21:24:28.641+00\n             5 | https://www.google.com/search?q=doge+meme |  0.6267991500000001 |         200 | FOUND     | 2021-02-07 21:24:29.01+00\n             6 | https://duckduckgo.com/?q=doge+meme       | 0.02907168100000046 |         200 | NOT_FOUND | 2021-02-07 21:24:31.042+00\n             7 | https://www.google.com/search?q=walt      |  1.1037337520000001 |         200 | FOUND     | 2021-02-07 21:24:31.749+00\n             8 | https://duckduckgo.com/?q=walt            |          0.03649068 |         200 | FOUND     | 2021-02-07 21:24:33.083+00\n             9 | https://www.google.com/search?q=doge+meme |  0.0821932520000015 |         200 | FOUND     | 2021-02-07 21:24:33.832+00\n            10 | https://duckduckgo.com/?q=doge+meme       | 0.01893998200000091 |         200 | NOT_FOUND | 2021-02-07 21:24:35.107+00\n    (10 rows)\n\nOr an `error`, for cases where there was no response:\n\n    walt=\u003e SELECT * FROM error LIMIT 10;\n     error_id |                      url             |     error     |         timestamp          \n    ----------+--------------------------------------+---------------+----------------------------\n            1 | https://www.google.com/search?q=walt | TIMEOUT_ERROR | 2021-02-07 21:38:50.293+00\n            2 | https://foobar.m.pipedream.net/p/179 | TIMEOUT_ERROR | 2021-02-07 21:42:59.343+00\n            3 | https://foobar.m.pipedream.net/p/04  | TIMEOUT_ERROR | 2021-02-07 21:42:59.343+00\n            4 | https://foobar.m.pipedream.net/p/01  | TIMEOUT_ERROR | 2021-02-07 21:42:59.344+00\n            5 | https://foobar.m.pipedream.net/p/359 | TIMEOUT_ERROR | 2021-02-07 21:42:59.343+00\n            6 | https://foobar.m.pipedream.net/p/13  | TIMEOUT_ERROR | 2021-02-07 21:42:59.343+00\n            7 | https://foobar.m.pipedream.net/p/02  | TIMEOUT_ERROR | 2021-02-07 21:42:59.344+00\n            8 | https://foobar.m.pipedream.net/p/11  | TIMEOUT_ERROR | 2021-02-07 21:42:59.344+00\n            9 | https://foobar.m.pipedream.net/p/10  | TIMEOUT_ERROR | 2021-02-07 21:42:59.344+00\n           10 | https://foobar.m.pipedream.net/p/05  | TIMEOUT_ERROR | 2021-02-07 21:42:59.343+00\n    (10 rows)\n\n## Installation\n\n### System requirement\n\n-   Python \u003e= 3.8\n\n### Install with pip\n\nClone the repo and install with `pip`:\n\n    $ git clone git@github.com:scorphus/walt.git\n    $ cd walt\n    $ pip install\n\nVerify the installation:\n\n    $ walt --version\n\n## Usage\n\nAfter installation, a script is created in your local Python `bin` path. Check\nhow to use it with:\n\n    $ walt --help\n\n## Configuration\n\nwalt is configured with a [TOML][] file. Either base off of\n[`config.sample.toml`][config.sample.toml] or generate a new sample:\n\n    $ walt generate_config_sample \u003e config.toml\n\nThen change the configuration accordingly. These are the default values:\n\n```toml\nlog_level = \"INFO\" # Logging level\nconcurrent = 2 # Number of concurrent workers checking URLs\ninterval = 2 # Interval between consecutive checks by the same worker\ntimeout = 30 # Timeout for connections\n\n# A map of URLs and their respective regular expressions: URL = regexp pattern\n[url_map]\n\"https://duckduckgo.com/?q=walt\" = \"Walt Disney\"\n\"https://www.google.com/search?q=walt\" = \"Walt Disney\"\n\"https://duckduckgo.com/?q=doge+meme\" = \"Kabosu\"\n\"https://www.google.com/search?q=doge+meme\" = \"Kabosu\"\n\n[kafka]\nuri = \"localhost:9092\" # Kafka server URI\ncafile = \"\" # Certificate Authority file path\ncertfile = \"\" # Client Certificate file path\nkeyfile = \"\" # Client Private Key file path\ntopic = \"walt\" # Default topic\n\n[postgres]\nhost = \"localhost\" # Database host address\nport = 5432 # Connection port number\nuser = \"postgres\" # User name used to authenticate\npassword = \"mysecretpassword\" # Password used to authenticate\ndbname = \"walt\" # Database name\n\n```\n\nYou don't need to write all entries in the TOML file. The above, for instance,\ndoes not specify the user agent and the HTTP headers:\n\n```toml\nuser_agent = \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36\"\n\n[headers]\nPragma = \"no-cache\"\n```\n\n### Environment Variables\n\nApart from `url_map`, all other config values can be set with environment\nvariables. Declare them in UPPER CASE prefixed with `WALT_`. For example:\n\n    $ env WALT_LOG_LEVEL=DEBUG WALT_TIMEOUT=17 walt generate_config_sample_from_env\n\n## Running\n\n### Setting up\n\nOnce you provisioned your infrastructure and configured walt accordingly, you\ncan start by creating the database and/or tables:\n\n    $ walt -c config.toml create_database  # skip if the database already exists\n    $ walt -c config.toml create_tables\n\nCheck [walt.tf][] if you plan to use walt with [Aiven][] database services.\n\n### Consuming/Producing\n\nStart the consumer with the following:\n\n    $ walt -c config.toml consume\n\nStart the producer with the following:\n\n    $ walt -c config.toml produce\n\n## Development\n\n### Requirements\n\n-   Python \u003e= 3.8\n-   An activated virtual environment\n-   [pre-commit][]\n\n### Create a development environment\n\n1.  Start by creating a new Python virtual environment with the tool of your\n    choice (we recommend [pyenv][])\n\n2.  Install pre-commit (we recommend [installing][pre-commit-install] it not as\n    part of the virtual environment — use your system's package manager)\n\n3.  Install walt in editable mode with all required dependencies:\n\n        $ make setup\n\n### Run tests\n\nOnce you have a working development environment:\n\n1.  Run tests\n\n        $ make tests\n\n2.  Check code coverage\n\n        $ make coverage\n        $ open htmlcov/index.html\n\n3.  Lint the code:\n\n        $ make lint\n\n### Run locally\n\nTo help with local development, the repository includes a `docker-compose.yml`\nfile that can be used to run the both Kafka and Postgres — default configuration\nvalues work with them.\n\n    $ docker-compose up -d\n\nHave fun!\n\n## License\n\nCode in this repository is distributed under the terms of the BSD 3-Clause\nLicense (BSD-3-Clause).\n\nSee [LICENSE][] for details.\n\n[build-badge]: https://github.com/scorphus/walt/workflows/Python/badge.svg\n[action-link]: https://github.com/scorphus/walt/actions?query=workflow%3APython\n[codecov-badge]: https://codecov.io/gh/scorphus/walt/branch/main/graph/badge.svg\n[codecov-link]: https://codecov.io/gh/scorphus/walt\n[codeclimate-badge]: https://api.codeclimate.com/v1/badges/1a6687203d55505d015d/maintainability\n[codeclimate-link]: https://codeclimate.com/github/scorphus/walt/maintainability\n[codacy-badge]: https://app.codacy.com/project/badge/Grade/06b3ee97b12b45abbe47bf92169b65be\n[codacy-link]: https://www.codacy.com/gh/scorphus/walt/dashboard?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=scorphus/walt\u0026amp;utm_campaign=Badge_Grade\n[toml]: https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f#summary\n[config.sample.toml]: config.sample.toml\n[walt.tf]: https://github.com/scorphus/walt.tf\n[aiven]: https://aiven.io/\n[pre-commit]: https://pre-commit.com\n[pre-commit-install]: https://pre-commit.com/#install\n[pyenv]: https://github.com/pyenv/pyenv\n[license]: LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscorphus%2Fwalt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscorphus%2Fwalt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscorphus%2Fwalt/lists"}