{"id":22964879,"url":"https://github.com/susji/lilmon","last_synced_at":"2025-04-02T04:22:46.438Z","repository":{"id":50691976,"uuid":"516477146","full_name":"susji/lilmon","owner":"susji","description":"li'l monitor","archived":false,"fork":false,"pushed_at":"2024-06-26T19:34:38.000Z","size":2216,"stargazers_count":0,"open_issues_count":5,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-07T19:13:47.388Z","etag":null,"topics":["lilmon","monitoring","monitoring-application","time-series"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/susji.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-21T18:14:45.000Z","updated_at":"2024-04-01T17:56:47.000Z","dependencies_parsed_at":"2024-06-21T11:46:32.614Z","dependency_job_id":"60cbe643-5540-4d46-9343-0508df5f5266","html_url":"https://github.com/susji/lilmon","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susji%2Flilmon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susji%2Flilmon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susji%2Flilmon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/susji%2Flilmon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/susji","download_url":"https://codeload.github.com/susji/lilmon/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246753371,"owners_count":20828138,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lilmon","monitoring","monitoring-application","time-series"],"created_at":"2024-12-14T20:12:46.243Z","updated_at":"2025-04-02T04:22:46.420Z","avatar_url":"https://github.com/susji.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lilmon\n\n## What is it?\n\nlilmon is a small program for collecting numeric values on your UNIX-like system\nand displaying them as time series in a browser view. If you stretch the\ndefinition a bit, it is a minimalistic monitoring tool.\n\nlilmon is currently very experimental software.\n\n## Why does it exist?\n\nI needed a small monitoring tool for my own use. I use it to monitor individual\nUNIX-like hosts. I'm looking at parameters like available diskspace, CPU load,\nnetwork performance, host temperatures, connection quality and so on. There are\nvery good and professional tools for this purpose, but I wanted something small\nand simple that requires very little setup.\n\n## What does it do?\n\nlilmon has two modes of operation.\n\nIn the first mode, `measure`, it periodically executes commands for its metrics\nand gathers their numeric values into a SQLite database.\n\nIn the second mode, `serve`, it displays the recorded values with a dynamic HTML\npage.\n\n\n## What does lilmon measure?\n\nlilmon measures numeric values. To do this, lilmon is given a set of *metrics*.\nEach metric has a\n\n- name\n- description\n- graphing options\n- command\n\nlilmon uses raw shell-commands to obtain these numeric values for each specified\nmetric. Commands are shell-expanded like\n\n    $ /bin/sh -c '\u003cmetric-command\u003e'\n\nand as a result, lilmon expects to receive a single value back via `stdout`. The\nvalue is interpreted as a `float64`. Integers are also fine. Whitespace is\ntrimmed before any interpretation is attempted.\n\nA minimalistic example of a metric command would then be `echo 123` which would\nresult in a static value of `123` on each measurement.\n\n## How does it look like?\n\nThe graphs are drawn using `gonum.org/v1/plot`. Currently lilmon produces graphs\nlike this:\n\n![screenshot of lilmon\nUI](https://github.com/susji/lilmon/raw/main/misc/lilmon.png \"lilmon v0.x.y\")\n\n## How do you configure lilmon?\n\nSee [the example file](lilmon.ini.example) for inspiration. Each definition of a\nmetric consists of the following four fields:\n\n    \u003cname\u003e|\u003cdescription\u003e|\u003cgraph-options\u003e|\u003craw-shell-command\u003e\n\nThe shell command may contain `|` characters -- it will not affect configuration\nparsing.\n\n`\u003cgraph-options\u003e` may contain the following `,` separated parameters:\n\n  - `deriv`: The time series is numerically differentiated with respect to time\n  - `no_ds`: The time series is not downsampled at all\n  - `y_min=\u003cfloat64\u003e`: Graph's minimum Y value\n  - `y_max=\u003cfloat64\u003e`: Graph's maximum Y value\n  - `kibi` and `kilo`: Y values are rendered with unit prefixes in base-2 or base-10, respectively\n\n`deriv` is useful if your metric is, for example, measuring transmitted or\nreceived bytes for a network interface. By using `deriv`, the UI will then\ndisplay transfer rates (bytes/second) instead of bytes.\n\n`no_ds` may be useful if you want to produce an exact averaging of some metric's\ndata. For example, if your data may contain abrupt changes in individual\nmeasurements and you want to be sure they are included when the time series is\nbeing binned, then you should enable `no_ds`. Do note that this makes the\ngeneration of the specific graph considerably slower.\n\n`kibi` and `kilo` will make larger values much more easier to read.\n\n### What if my metric command contains `;`?\n\nThis will be a problem for the configuration parser because it assumes that a\nlonesome `;` begins a comment. To avoid this, quote your metric definition like\nthis:\n\n```\nmetric=\"n_subshell_constant|Plain silly||{ echo -n \\\"one\\\"; echo -n two; }|wc -c\"\n```\n\n## Show me some example metrics!\n\nThese are some metrics I use. They may fail in cases I have not thought about.\nThere are many ways to obtain similar results. The primary reason for including\nthese here is to give you inspiration on how to use lilmon.\n\nAs is stressed elsewhere in this README, to be safer, avoid running any metrics\nas a privileged user. If greater privileges are required, use something which\nselectively gives you just enough capabilities to measure your value.\n\n### TX \u0026 RX speed of some interface\n\nUse the correct interface name in place of `if-name`. If you need to measure\nthan one interface, define more similar metrics with different metric name and\n`if-name`. Silly but simple!\n\nAlso, note the `deriv` in the graphing options, which means that the raw byte\ncounts are numerically differentiated when the graph is drawn. The result is\nthen a decent approximation of TX \u0026 RX speed.\n\n#### Linux\n\n```\nmetric=bytes_wifi_rx|Wifi RX|y_min=0,deriv,kilo|cat /proc/net/dev|fgrep if-name|awk '{print $2}'\nmetric=bytes_wifi_tx|Wifi TX|y_min=0,deriv,kilo|cat /proc/net/dev|fgrep if-name|awk '{print $10}'\n```\n\n#### OpenBSD\n\n```\nmetric=bytes_wired_rx|Wired RX|deriv,y_min=0,kilo|netstat -n -i -b|fgrep if-name|fgrep Link|awk '{print $5}'\nmetric=bytes_wired_tx|Wired TX|deriv,y_min=0,kilo|netstat -n -i -b|fgrep if-name|fgrep Link|awk '{print $6}'\n```\n\n### Temperature sensor\n\n#### Linux\n\nThe example here makes use of `jq` to search the JSON dump produced by `sensors\n-j`. See what `sensors -j` displays for you and accomodate the `jq` filter. You\nmay of course produce the same sensor value by just parsing and filtering the\nregular text dump.\n\n```\nmetric=temp_cpu|CPU temperature|y_min=30,y_max=90|sensors -j|jq '.[\"dev::temp1::temp1_input\"]'\n```\n\n#### OpenBSD\n\nLook at the output of `sysctl hw.sensors` and figure out the the exact path for\nyour device. If it has something other than a raw float value, filter the rest\nout.\n\n```\nmetric=cpu_temp|CPU temperature|y_min=40,y_max=90|sysctl hw.sensors.km0.temp0|cut -d '=' -f2|cut -d ' ' -f 1\n```\n\n### Ping round-trip time for a well-known target\n\n#### Linux\n\nNote that in this example we use the `-w 10` option to define a hard deadline of\n10 seconds. This is not fully portable, so see your `man 8 ping` for more\ndetails. Something like the `timeout` command is available on many platforms,\nand it works well for making sure programs time out.\n\n```\nmetric=ping_google|PING Google|y_min=0,kilo|ping -q -w 10 -c 2 8.8.8.8|tail -1|cut -d'=' -f2|cut -d '/' -f2\n```\n\n### System load (1 min)\n\n#### OpenBSD\n\n```\nmetric=load_1|1 minute CPU LOAD|y_min=0|uptime|grep -E -o 'averages: [\\.0-9]+'|cut -d ' ' -f2\n```\n\n### Free memory\n\n#### OpenBSD\n\nFor an example's sake, we go through some trouble to dig out some bytes. Perhaps\nwe are lucky and `top` always prints megabytes?\n\n```\nmetric=free_mem|Free memory|y_min=0,kilo|echo $((1024 * 1024 * $(top -b|egrep -o 'Free: [0-9]+'|cut -d ' ' -f2)))\n```\n\n### Wi-Fi clients (stations) connected to hostapd\n\nNote that this invocation probably requires privileged execution. See the\ndiscussion regarding `sudo` and `doas`. This command also assumes that there is\nonly one station or that the station of interest has index `0`.\n\n```\nmetric=n_sta|hostapd clients|y_min=0|sudo /usr/sbin/hostapd_cli status|fgrep 'num_sta[0]'|cut -d '=' -f 2\n```\n\n## Does lilmon do alerting?\n\nNo. Its intended purpose is to record numeric values and display them with a\nbare bones UI. However, as everything is recorded into a SQLite database, a\ndifferent program can easily follow the metrics and do alerting based on that.\nFor details, see the next question.\n\n## How do I access the values lilmon has gathered?\n\nFirst make sure you have `sqlite3` installed. Then you can do something like\nthe following to get the 10 latest measurements for metric `NAME`.\n\n    $ sqlite3 'file:/var/lilmon/db/lilmon.sqlite?mode=ro' \\\n          'SELECT * FROM lilmon_metric_NAME ORDER BY timestamp DESC LIMIT 10'\n\nNote the `mode=ro` part for read-only.\n\n## Will lilmon have a configuration UI?\n\nNo.\n\n## The graphs look terrible!\n\n~~Yes. I'll probably make them less terrible in future.~~\nMuch better now, right?\n\n## The graphs look too terrible!\n\nThis is enough for me. However, `measure` and `serve` are fairly well decoupled\nso an alternative UI is quite easy to build based on the data gathered by\n`measure`.\n\n## `lilmon serve` refuses to start and says it cannot open the database!\n\nAre you running `serve` without a pre-existing database? In that case, you will\nhave to start `lilmon measure` first because it will create the missing\ndatabase.\n\n## How to proceed after changing the metrics in the configuration file?\n\nRestart both processes but do restart `lilmon measure` first. It is responsible\nfor creating new database tables and their indexes for new or renamed metrics.\n\n## Will lilmon support monitoring more than one machine?\n\nAs all lilmon metrics are just columns in a SQLite table, they can be\ntransferred outside their host of origin with relative ease. It's just not\nsomething I'm especially interested in.\n\n## How to run measurement commands which require privileged execution?\n\nIt is **not** required or recommended to run lilmon as a privileged user.\n\nFor the measure mode, it is wiser to make use of `sudo`, `doas`, or something\nsimilar with limited capabilities to obtain privileged metrics. You would then\nrun these exactly as the non-privileged ones, except through `doas` like here:\n\n```\n[metrics]\nmetric=n_id_chars|Characters output by privileged id|y_min=0|doas /usr/bin/id|wc -c\n```\n\n### doas\n\nWith `doas` you may permit the `lilmon` user to run `/usr/bin/id` without any\narguments as `root` like this:\n\n```doas\npermit nopass lilmon as root cmd /usr/bin/id args\n```\n\nDo note that the lone keyword `args` suffixed to the command means only\nexecution without any arguments.\n\n### sudo\n\n`sudo` also permits specifying that the command can only be run without any\narguments, however as `sudo` also supports very rich logic for specifying how\nand what commands can be run, please exercise caution, stick to simple\ndefinitions, and give the relevant manuals a careful read. In this example, the\nuser `lilmon` is permitted to run `/usr/bin/id` without any arguments as `root`\nand without a password on all hosts where the configuration file is active:\n\n```sudo\nlilmon ALL=(root) NOPASSWD: /usr/bin/id \"\"\n```\n\nDo note that if you do not specify the `\"\"` suffix, then the above invocation\nwould permit `lilmon` to execute `/usr/bin/id` with arbitrary arguments.\n\n## Can you edit the browser UI?\n\nYes, just use [the example as basis](lilmon.template.example) and have at it.\n\n## What is required to run lilmon?\n\n**NOTE**: lilmon is currently **very experimental software** and it is not yet\npackaged in any reasonable manner. Your usage experience will be mildly tedious.\nBefore trying to perform an install with the attached `Makefile`, convince\nyourself that it is doing the right thing. At this stage, performing a manual\ninstall may be a better idea.\n\nThe installation is for the most part condensed into `make install`, but the\ncreation of the non-privileged user is platform-dependent. We also must give\nthat user a chance to write its database in the directory. For GNU/Linux it\nlooks like this\n\n```\n# make install\n# adduser --disabled-login --system --no-create-home --group lilmon\n# chown lilmon:lilmon /var/lilmon/db\n# sudo -u lilmon /usr/local/bin/lilmon measure\n# sudo -u lilmon /usr/local/bin/lilmon serve\n```\n\nWhen you are starting lilmon fresh without a pre-existing database, the first\nrun of `lilmon measure` will create it. As `lilmon serve` opens the database in\na read-only mode, it cannot initialize the database. Thus make sure have\nsuccessfully ran `measure` at least once before running `serve`.\n\nAlso note that by default `lilmon serve` listens only on localhost. You may want\nto set the listening adress to something else such as a suitable interface's IP.\nIf you want it to listen on all interfaces, use `0.0.0.0:15515` but please do\nnot expose the lilmon browser view to any untrusted networks. As suggested\nbelow, you may in any case wish to provide the actual access via a suitable\nreverse proxy.\n\n## Do I need timeouts for my commands?\n\nIt does not hurt, but lilmon tries to cancel measurement commands which take\n`$TOO_LONG` to complete. See `metrics.go` for the details.\n\n## What about TLS, rate limiting, authentication...?\n\nI strongly recommend a reverse proxy for handling these things.\n\n## How does lilmon treat the data when time series are produced?\n\n### Short answer\n\nIt produces an averaged view which may contain some quantitative accuracy.\n\n### Long answer\n\nThere are two basic steps: obtaining the `(timestamp, value)` pairs from the\ndatabase and producing a binned view on them. I'm guessing there is a smarter\nway to achieve the same result with some SQL wizardry.\n\n#### Random sampling of measurements\n\nlilmon automatically does downsampling when it thinks that the query may result\nin a large amount of samples. Here we make two assumptions:\n\n1. Measurements are evenly distributed\n2. Neglecting individual samples at random is OK\n\nIn practice we use SQLite's `RANDOM()` to produce a coinflip when samples are\n`SELECT`ed from the metric tables.\n\nThe behavior can be turned off for individual metrics with the `no_ds` graphing\noption and the global behavior may be adjusted with the `downsampling_scale`\noption. The greater the value is, the less effect downsampling has. For details,\nsee `db.go`.\n\n#### Averaging of samples to individual bins\n\nEach lilmon graph contains some amount of bins. The exact amount is defined by\nthree variables:\n\n1. graph time range (from user)\n2. bin width (from configuration)\n3. maximum amount of bins (from configuration)\n\nIn the first step, we collected a bunch of samples and the here in the second\nstep we distribute them among the bins. The resulting bin value is then an\naverage of the all the values placed in the bin. For details, see `graph.go`.\n\n## Known limitations\n\n- If a metric is disabled by removing it from the configuration file, its\n  historical data will not be automatically pruned after the retention period\n\n## TODO\n\n- [ ] support units for smart Y labels (eg. \"bytes\")\n- [ ] some end-to-end testing for `serve`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsusji%2Flilmon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsusji%2Flilmon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsusji%2Flilmon/lists"}