{"id":16654338,"url":"https://github.com/cyrbil/datalog","last_synced_at":"2025-04-05T02:13:57.273Z","repository":{"id":141938867,"uuid":"170783200","full_name":"cyrbil/datalog","owner":"cyrbil","description":"Datalog HTTP logs monitoring","archived":false,"fork":false,"pushed_at":"2021-06-08T08:25:52.000Z","size":131,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-02-10T10:21:27.192Z","etag":null,"topics":["http","logs","monitoring","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cyrbil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-15T01:30:17.000Z","updated_at":"2022-05-13T15:14:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"f29d9652-d78b-4526-a116-3874a6ba8b9a","html_url":"https://github.com/cyrbil/datalog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyrbil%2Fdatalog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyrbil%2Fdatalog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyrbil%2Fdatalog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyrbil%2Fdatalog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cyrbil","download_url":"https://codeload.github.com/cyrbil/datalog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276188,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http","logs","monitoring","python3"],"created_at":"2024-10-12T09:49:30.733Z","updated_at":"2025-04-05T02:13:57.268Z","avatar_url":"https://github.com/cyrbil.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DataLog HTTP Monitoring\r\n\r\nThis tool parses http log files and display statistics about it.\r\n\r\n![](demo.png)\r\n\r\n\r\n## Install\r\n\r\n\u003e Python 3.7 is required to run the project (3.6 might run okay though)\r\n\r\n    $ python setup.py install\r\n    $ datalog -h\r\n    \r\n    \r\n## Run\r\n\r\nFor a simple demo (will generate random logs):\r\n\r\n    $ datalog --demo\r\n    \r\nWith real log files:\r\n\r\n    $ datalog /path/to/http.log\r\n    \r\n \r\n## Docker\r\n\r\nA container is available and will run the command line directly\r\n\r\n    $ docker build --rm --tag datalog .\r\n    $ docker run --rm -ti datalog --help\r\n\r\n\r\n## Command line\r\n\r\nThis tool can also be called from command line and offers various options:\r\n\r\n    $ datalog --help\r\n    usage: run.py [-h] [--period PERIOD] [--alert THRESHOLD]\r\n              [--alert-period ALERT_PERIOD] [--alert-file ALERT_FILE]\r\n              [--refresh REFRESH] [--no-curses] [--demo] [--debug]\r\n              [--debug-file FILE] [--debug-color]\r\n              [LOGFILE [LOGFILE ...]]\r\n\r\n    Collect logs and display realtime formatted statistics\r\n    \r\n    positional arguments:\r\n      LOGFILE               Files to collect logs from (default: ['/tmp/access.log'])\r\n    \r\n    optional arguments:\r\n      -h, --help            show this help message and exit\r\n      --period PERIOD       monitoring period to display statistics (default: 10)\r\n      --alert THRESHOLD     minimum number of requests to trigger alert mode (default: 10)\r\n      --alert-period ALERT_PERIOD\r\n                            period to look for threshold alert (default: 120)\r\n      --alert-file ALERT_FILE\r\n                            where to store alerts details (default: /tmp/access.log)\r\n      --refresh REFRESH     statistics display refresh delay (default: 0.5)\r\n      --no-curses           fallback to simple print for display\r\n      --demo                auto generate logs for debugging purpose\r\n      --debug               show application debug information\r\n      --debug-file FILE     write application debug information to this file (default: stderr)\r\n      --debug-color         colorize application debug information (implies --debug)\r\n\r\n\r\n## Testing\r\n\r\nYou can simply run the tests using:\r\n\r\n    pip install pytest \r\n    pytest tests\r\n\r\n\r\n## Coverage\r\n\r\nTo run tests with coverage:\r\n\r\n    pip install pytest-cov\r\n    pytest --cov\r\n\r\n## Roadmap\r\n\r\n  - [x] Consume an actively written-to w3c-formatted HTTP access log (https://www.w3.org/Daemon/User/Config/Logging.html). It should default to reading /tmp/access.log and be overrideable\r\n  - [x] Display stats every 10s about the traffic during those 10s: the sections of the web site with the most hits, as well as interesting summary statistics on the traffic as a whole. A section is defined as being what's before the second '/' in the resource section of the log line. For example, the section for \"/pages/create\" is \"/pages\"\r\n  - [x] Make sure a user can keep the app running and monitor the log file continuously\r\n  - [x] Whenever total traffic for the past 2 minutes exceeds a certain number on average, add a message saying that “High traffic generated an alert - hits = {value}, triggered at {time}”.\r\n  - [x] The default threshold should be 10 requests per second, and should be overridable.\r\n  - [x] Whenever the total traffic drops again below that value on average for the past 2 minutes, add another message detailing when the alert recovered.\r\n  - [x] Make sure all messages showing when alerting thresholds are crossed remain visible on the page for historical reasons.\r\n  - [x] Write a test for the alerting logic.\r\n  - [x] Explain how you’d improve on this application design.\r\n  - [x] If you have access to a linux docker environment, we'd love to be able to docker build and run your project! If you don't though, don't sweat it.\r\n\r\n  \r\n## Improvements\r\n\r\n  - [x] Use curses\r\n  - [x] Pipfile\r\n  - [x] Log generator\r\n  - [x] Create setup.py for command install\r\n  - [x] Coverage\r\n  - [ ] Better unittests\r\n  - [ ] Better documentation\r\n  - [ ] Better logging (not flooding like hell)\r\n  - [ ] Avoid UI overflow\r\n  - [ ] Add log files names to UI\r\n  - [ ] Detect and handle log rotation\r\n  - [ ] ~~Maybe asyncio instead of processes~~ (Nope, aiofiles is struggling)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyrbil%2Fdatalog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyrbil%2Fdatalog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyrbil%2Fdatalog/lists"}