https://github.com/buanzo/webmon
A scalable, plugin-based url asset monitoring system based on zeromq, flask-restful, redis and docker-compose
https://github.com/buanzo/webmon
asset-monitoring docker docker-compose flask-restful osint osint-framework osint-python python3 redis security url-monitor web-monitoring
Last synced: about 1 year ago
JSON representation
A scalable, plugin-based url asset monitoring system based on zeromq, flask-restful, redis and docker-compose
- Host: GitHub
- URL: https://github.com/buanzo/webmon
- Owner: buanzo
- Created: 2018-12-18T15:09:31.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2024-04-12T21:46:39.000Z (about 2 years ago)
- Last Synced: 2025-02-26T09:41:48.169Z (over 1 year ago)
- Topics: asset-monitoring, docker, docker-compose, flask-restful, osint, osint-framework, osint-python, python3, redis, security, url-monitor, web-monitoring
- Language: Python
- Size: 52.7 KB
- Stars: 14
- Watchers: 2
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Web Monitoring
--------------
Webmon executes plugins against assets (URLs, originally).
The API will contain groups of calls, divided in asset management,
statistics gathering and execution.
Asset Management involves URL management, adding, modifying and removing
items (i.e "https://somesite.net, http://some.other.server/somespecificURI,
etc").
Statistics Gathering will provide statistical information about said assets.
Execution will provide the necessary means to push jobs to the work queue.
Storage is implemented in Redis
Work Queue
----------
The Jobs API (A Ronald Class) will send jobs to a Messaging Queue (zeromq).
This Queue will be read by the Dispatcher, which is a standalone module.
Statistics
----------
The statistics API will provide information about assets. Information will
come from the different plugins that the Executor enables. All plugins will
be executed for each Asset.
Executor Module (standalone)
----------------------------
The Executor Module will be a Task Worker, using the standard zeromq
Streamer: http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/devices/streamer.html
Dispatcher Module (standalone)
------------------------------
The Dispatcher Module will act as a broker. The Jobs API class will use zeromq
to communicate with the Dispatcher, indicating Job IDs.
Protocols
---------
Each executor will have access to all plugins, directly, hence no need to do
unsafe things like transporting code. The executor is NOT a generic 'worker'
node. The transport between Jobs API <-> Dispatcher <-> Executor is zeromq.
The transport between Executor and Storage is Redis.
TO DO LIST
----------
In consideration to https://redis.io/topics/persistence, snapshots will be
triggered after each important API call, and whenever the Dispatcher gets a
"SNAPSHOT NOW" command over its zeromq control channel.
The AOF (Append-Only-File) is a persistence method for
Redis Key-Value noSQL storage system). It is comparable to a Changelog.
The RDB (Redis Database) is the snapshot-style persistence method. This is
what we will be using.
https://www.fullstackpython.com/blog/install-redis-use-python-3-ubuntu-1604.html
https://redis.io/commands/save
https://redis.io/commands/bgsave
PLUGINS
-------
* HtmlStats: uses numpy to create standard deviation statistical
calculations to detect excessive changes between different parameters such
as html tags, pagesize in bytes, etc.
There are other plugins included.