{"id":27258289,"url":"https://github.com/buanzo/webmon","last_synced_at":"2025-04-11T03:20:16.232Z","repository":{"id":40984734,"uuid":"162305900","full_name":"buanzo/webmon","owner":"buanzo","description":"A scalable, plugin-based url asset monitoring system based on zeromq, flask-restful, redis and docker-compose","archived":false,"fork":false,"pushed_at":"2024-04-12T21:46:39.000Z","size":54,"stargazers_count":14,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-26T09:41:48.169Z","etag":null,"topics":["asset-monitoring","docker","docker-compose","flask-restful","osint","osint-framework","osint-python","python3","redis","security","url-monitor","web-monitoring"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/buanzo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-18T15:09:31.000Z","updated_at":"2024-02-01T05:59:12.000Z","dependencies_parsed_at":"2022-07-08T19:34:28.192Z","dependency_job_id":null,"html_url":"https://github.com/buanzo/webmon","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buanzo%2Fwebmon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buanzo%2Fwebmon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buanzo%2Fwebmon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buanzo%2Fwebmon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/buanzo","download_url":"https://codeload.github.com/buanzo/webmon/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248333930,"owners_count":21086282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asset-monitoring","docker","docker-compose","flask-restful","osint","osint-framework","osint-python","python3","redis","security","url-monitor","web-monitoring"],"created_at":"2025-04-11T03:20:15.595Z","updated_at":"2025-04-11T03:20:16.214Z","avatar_url":"https://github.com/buanzo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Web Monitoring\n--------------\n\nWebmon executes plugins against assets (URLs, originally).\n\nThe API will contain groups of calls, divided in asset management,\nstatistics gathering and execution.\n\nAsset Management involves URL management, adding, modifying and removing\nitems (i.e \"https://somesite.net, http://some.other.server/somespecificURI,\netc\").\n\nStatistics Gathering will provide statistical information about said assets.\n\nExecution will provide the necessary means to push jobs to the work queue.\n\nStorage is implemented in Redis\n\n\nWork Queue\n----------\n\nThe Jobs API (A Ronald Class) will send jobs to a Messaging Queue (zeromq). \nThis Queue will be read by the Dispatcher, which is a standalone module. \n\n\nStatistics\n----------\n\nThe statistics API will provide information about assets. Information will\ncome from the different plugins that the Executor enables. All plugins will\nbe executed for each Asset.\n\n\nExecutor Module (standalone)\n----------------------------\n\nThe Executor Module will be a Task Worker, using the standard zeromq\nStreamer: http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/devices/streamer.html\n\nDispatcher Module (standalone)\n------------------------------\n\nThe Dispatcher Module will act as a broker. The Jobs API class will use zeromq\nto communicate with the Dispatcher, indicating Job IDs.\n\nProtocols\n---------\n\nEach executor will have access to all plugins, directly, hence no need to do\nunsafe things like transporting code. The executor is NOT a generic 'worker'\nnode. The transport between Jobs API \u003c-\u003e Dispatcher \u003c-\u003e Executor is zeromq.\n\nThe transport between Executor and Storage is Redis.\n\n\nTO DO LIST\n----------\n\nIn consideration to https://redis.io/topics/persistence, snapshots will be\ntriggered after each important API call, and whenever the Dispatcher gets a\n\"SNAPSHOT NOW\" command over its zeromq control channel.\n\nThe AOF (Append-Only-File) is a persistence method for\nRedis Key-Value noSQL storage system). It is comparable to a Changelog.\n\nThe RDB (Redis Database) is the snapshot-style persistence method. This is\nwhat we will be using.\n\nhttps://www.fullstackpython.com/blog/install-redis-use-python-3-ubuntu-1604.html\n\nhttps://redis.io/commands/save\n\nhttps://redis.io/commands/bgsave\n\nPLUGINS\n-------\n\n* HtmlStats: uses numpy to create standard deviation statistical\n  calculations to detect excessive changes between different parameters such\n  as html tags, pagesize in bytes, etc.\n\nThere are other plugins included.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuanzo%2Fwebmon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbuanzo%2Fwebmon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuanzo%2Fwebmon/lists"}