Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/lifailon/shell-metrics
Select metrics for outputs to log file and influxdb
https://github.com/lifailon/shell-metrics
bash cpu-monitoring curl devops directory-monitoring file-monitoring influx influxdb ios iostat linux log logging metrics monitoring process-monitor process-monitoring scripts shell units
Last synced: about 2 months ago
JSON representation
Select metrics for outputs to log file and influxdb
- Host: GitHub
- URL: https://github.com/lifailon/shell-metrics
- Owner: Lifailon
- Created: 2023-09-04T10:12:39.000Z (over 1 year ago)
- Default Branch: rsa
- Last Pushed: 2023-09-20T10:35:40.000Z (over 1 year ago)
- Last Synced: 2024-01-29T09:46:07.183Z (about 1 year ago)
- Topics: bash, cpu-monitoring, curl, devops, directory-monitoring, file-monitoring, influx, influxdb, ios, iostat, linux, log, logging, metrics, monitoring, process-monitor, process-monitoring, scripts, shell, units
- Language: Shell
- Homepage:
- Size: 1.44 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
**Content:**
- [dir-monitor-log](#dir-monitor-log)
- [top-metrics-log](#top-metrics-log)
- [iostat-metrics-log](#iostat-metrics-log)
- [logrotate](#logrotate)
- [iostat-to-influxdb](#iostat-metrics-log)
- [proc-stats-to-influxdb](#proc-stats-to-influxdb)
- [InfluxDB](#influxdb)
## dir-monitor-log
Directory monitoring for get metric to **file count, size and modify.** When the number of files is changed, a diff of changes is logging.
**Service path:** /etc/systemd/system/[dir-monitor-log.service](https://github.com/Lifailon/shell-metrics/blob/rsa/service/dir-monitor-log.service) \
**Script path:** /root/[dir-monitor-log.sh](https://github.com/Lifailon/shell-metrics/blob/rsa/scripts/dir-monitor-log.sh) \
Script variables: \
**path_mon** - target directory for monitoring (example: **/var/lib/jenkins**) \
**path_log** - path to log file (example: **/var/log/metrics/dir-monitor.log**)
**Example:**
```bash
root@devops-01:~# systemctl daemon-reload
root@devops-01:~# systemctl enable dir-monitor-log.service
root@devops-01:~# systemctl start dir-monitor-log.service
root@devops-01:~# touch /var/lib/jenkins/test /var/lib/jenkins/test2
root@devops-01:~# sleep 10
root@devops-01:~# rm /var/lib/jenkins/test /var/lib/jenkins/test2
root@devops-01:~# sleep 10
root@devops-01:~# cat "/var/log/dir-monitor.log" | tail -n 15
4 Sep 01:40:41 Size: 333M File Count: 12107 Modify: 4 Sep 13:35
4 Sep 01:40:52 Size: 333M File Count: 12107 Modify: 4 Sep 13:35
4 Sep 01:41:02 Size: 333M File Count: 12107 Modify: 4 Sep 13:35
4 Sep 01:41:12 Size: 333M File Count: 12107 Modify: 4 Sep 13:35
4 Sep 01:41:22 Size: 333M File Count: 12107 Modify: 4 Sep 13:35
4 Sep 01:41:32 Size: 333M File Count: 12107 Modify: 4 Sep 13:35
4 Sep 01:41:43 Size: 333M File Count: 12107 Modify: 4 Sep 13:35
4 Sep 01:41:53 Size: 333M File Count: 12109 Modify: 4 Sep 13:41
+ 0 /var/lib/jenkins/test2
+ 0 /var/lib/jenkins/test
4 Sep 01:42:03 Size: 333M File Count: 12107 Modify: 4 Sep 13:42
- 0 /var/lib/jenkins/test2
- 0 /var/lib/jenkins/test
4 Sep 01:42:13 Size: 333M File Count: 12107 Modify: 4 Sep 13:42
4 Sep 01:42:24 Size: 333M File Count: 12107 Modify: 4 Sep 13:42
```
## top-metrics-log
Collecting metrics from software **top** for write to the log file and **logging of high-load process**.
**Script path:** /root/[top-metrics-log.sh](https://github.com/Lifailon/shell-metrics/blob/rsa/scripts/top-metrics-log.sh) \
Script variables: \
**path** - path to log file (example: **/var/log/metrics/top-metrics.log**) \
**trigger** - CPU load percentage (example: **20 %**) for logging for logging high-load process
**Initialization unit to systemd:** \
Creat service to path: /etc/systemd/system/[top-metrics-log.service](https://github.com/Lifailon/shell-metrics/blob/rsa/service/top-metrics-log.service) \
`systemctl daemon-reload` \
`systemctl enable top-metrics-log.service` \
`systemctl start top-metrics-log`
```bash
root@devops-01:~# systemctl status top-metrics-log.service
● top-metrics-log.service - Metrics from software top for write to the log file
Loaded: loaded (/etc/systemd/system/top-metrics-log.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-09-04 09:45:32 MSK; 3h 31min ago
Main PID: 2023242 (bash)
Tasks: 2 (limit: 4515)
Memory: 7.0M
CPU: 6min 20.318s
CGroup: /system.slice/top-metrics-log.service
├─2023242 /bin/bash /root/top-metrics-log.sh
└─2404579 sleep 2
Sep 04 13:16:34 devops-01 bash[2404195]: 4 Sep 01:16:34 CPU(user): 14.3 % CPU(system): 7.1 % Proc: 1/239 Users: 2 MEM: 1598.5/3876.4 MB>
Sep 04 13:16:34 devops-01 bash[2404219]: Warning: 4 Sep 01:16:34 Top Process: mysqld Time: 86:30.06 CPU: 12.5 % MEM: 18.3 MB User: mys>
Sep 04 13:16:36 devops-01 bash[2404278]: 4 Sep 01:16:36 CPU(user): 7.1 % CPU(system): 7.1 % Proc: 1/239 Users: 2 MEM: 1598.7/3876.4 MB >
Sep 04 13:16:39 devops-01 bash[2404334]: 4 Sep 01:16:39 CPU(user): 52.9 % CPU(system): 5.9 % Proc: 2/239 Users: 2 MEM: 1598.9/3876.4 MB>
Sep 04 13:16:39 devops-01 bash[2404358]: Warning: 4 Sep 01:16:39 Top Process: java Time: 5:10.34 CPU: 50.0 % MEM: 13.3 MB User: jenkins
Sep 04 13:16:41 devops-01 bash[2404416]: 4 Sep 01:16:41 CPU(user): 58.8 % CPU(system): 0.0 % Proc: 2/239 Users: 2 MEM: 1598.9/3876.4 MB>
Sep 04 13:16:41 devops-01 bash[2404440]: Warning: 4 Sep 01:16:41 Top Process: java Time: 5:11.27 CPU: 56.2 % MEM: 13.4 MB User: jenkins
~
~
~
root@devops-01:~# cat /var/log/top-metrics.log
4 Sep 01:16:34 CPU(user): 14.3 % CPU(system): 7.1 % Proc: 1/239 Users: 2 MEM: 1598.5/3876.4 MB SWAP: 0.8/3889.0 MB
Warning: 4 Sep 01:16:34 Top Process: mysqld Time: 86:30.06 CPU: 12.5 % MEM: 18.3 MB User: mysql
4 Sep 01:16:36 CPU(user): 7.1 % CPU(system): 7.1 % Proc: 1/239 Users: 2 MEM: 1598.7/3876.4 MB SWAP: 0.8/3889.0 MB
4 Sep 01:16:39 CPU(user): 52.9 % CPU(system): 5.9 % Proc: 2/239 Users: 2 MEM: 1598.9/3876.4 MB SWAP: 0.8/3889.0 MB
Warning: 4 Sep 01:16:39 Top Process: java Time: 5:10.34 CPU: 50.0 % MEM: 13.3 MB User: jenkins
4 Sep 01:16:41 CPU(user): 58.8 % CPU(system): 0.0 % Proc: 2/239 Users: 2 MEM: 1598.9/3876.4 MB SWAP: 0.8/3889.0 MB
Warning: 4 Sep 01:16:41 Top Process: java Time: 5:11.27 CPU: 56.2 % MEM: 13.4 MB User: jenkins
4 Sep 01:16:43 CPU(user): 60.0 % CPU(system): 6.7 % Proc: 1/239 Users: 2 MEM: 1599.3/3876.4 MB SWAP: 0.8/3889.0 MB
Warning: 4 Sep 01:16:43 Top Process: java Time: 5:12.71 CPU: 95.0 % MEM: 13.6 MB User: jenkins
4 Sep 01:16:46 CPU(user): 6.2 % CPU(system): 6.2 % Proc: 1/238 Users: 2 MEM: 1599.3/3876.4 MB SWAP: 0.8/3889.0 MB
4 Sep 01:16:48 CPU(user): 6.7 % CPU(system): 6.7 % Proc: 1/242 Users: 2 MEM: 1599.3/3876.4 MB SWAP: 0.8/3889.0 MB
```
## iostat-metrics-log
Collection metrics **iostat** from the set sysstat for **write log file**
**Service path:** /etc/systemd/system/[iostat-metrics-log.service](https://github.com/Lifailon/shell-metrics/blob/rsa/service/iostat-metrics-log.service) \
**Script path:** /root/[iostat-metrics-log.sh](https://github.com/Lifailon/shell-metrics/blob/rsa/scripts/iostat-metrics-log.sh) \
Script variables: \
**path** - path to log file (example: **/var/log/metrics/iostat-metrics.log**)
```bash
root@devops-01:~# systemctl daemon-reload
root@devops-01:~# systemctl enable iostat-metrics-log.service
root@devops-01:~# systemctl start iostat-metrics-log.service
root@devops-01:~# systemctl status iostat-metrics-log.service
● iostat-metrics-log.service - Directory monitoring for get metric to file count, size and modify
Loaded: loaded (/etc/systemd/system/iostat-metrics-log.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-09-05 12:44:33 MSK; 1min 20s ago
Main PID: 724914 (bash)
Tasks: 3 (limit: 4515)
Memory: 14.7M
CPU: 977ms
CGroup: /system.slice/iostat-metrics-log.service
├─ 724914 /bin/bash /root/iostat-metrics-log.sh
├─ 729062 sleep 5
└─2644388 iostat -yh /dev/sda 1
Sep 05 12:45:35 devops-01 bash[728116]: 5 Sep 12:45:34 tps = 0 read/s = 0.0k write/s = 0.0k
Sep 05 12:45:41 devops-01 bash[728450]: 5 Sep 12:45:40 tps = 164 read/s = 0.0k write/s = 1.4M
Sep 05 12:45:41 devops-01 bash[728465]: 5 Sep 12:45:40 tps = 0 read/s = 0.0k write/s = 0.0k
Sep 05 12:45:41 devops-01 bash[728480]: 5 Sep 12:45:40 tps = 0 read/s = 0.0k write/s = 0.0k
Sep 05 12:45:47 devops-01 bash[728776]: 5 Sep 12:45:46 tps = 123 read/s = 0.0k write/s = 1.3M
Sep 05 12:45:47 devops-01 bash[728791]: 5 Sep 12:45:46 tps = 0 read/s = 0.0k write/s = 0.0k
Sep 05 12:45:47 devops-01 bash[728806]: 5 Sep 12:45:46 tps = 0 read/s = 0.0k write/s = 0.0k
Sep 05 12:45:54 devops-01 bash[729031]: 5 Sep 12:45:52 tps = 99 read/s = 0.0k write/s = 1.2M
Sep 05 12:45:54 devops-01 bash[729046]: 5 Sep 12:45:52 tps = 0 read/s = 0.0k write/s = 0.0k
Sep 05 12:45:54 devops-01 bash[729061]: 5 Sep 12:45:52 tps = 0 read/s = 0.0k write/s = 0.0k
```
## logrotate
Automated log file rotation for all metrics: /etc/logrotate.d/[logrotate_metrics.conf](https://github.com/Lifailon/shell-metrics/blob/rsa/logrotate_metrics.conf)
**Debug:**
``` bash
root@devops-01:~# logrotate -d /etc/logrotate.d/logrotate_metrics.conf
...
rotating pattern: /var/log/metrics/dir-monitor.log 10485760 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/metrics/dir-monitor.log
Creating new state
Now: 2023-09-05 21:31
Last rotated at 2023-09-05 21:00
log does not need rotating (log size is below the 'size' threshold)
rotating pattern: /var/log/metrics/top-metrics.log 10485760 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/metrics/top-metrics.log
Creating new state
Now: 2023-09-05 21:31
Last rotated at 2023-09-05 21:00
log does not need rotating (log size is below the 'size' threshold)
rotating pattern: /var/log/metrics/iostat-metrics.log 10485760 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/metrics/iostat-metrics.log
Creating new state
Now: 2023-09-05 21:31
Last rotated at 2023-09-05 21:00
log does not need rotating (log size is below the 'size' threshold)
```
## iostat-to-influxdb
**Service path:** /etc/systemd/system/[iostat-to-influxdb.service](https://github.com/Lifailon/shell-metrics/blob/rsa/service/iostat-to-influxdb.service) \
**Script path:** /root/[iostat-to-influxdb.sh](https://github.com/Lifailon/shell-metrics/blob/rsa/scripts/iostat-to-influxdb.sh) \
Script variables: \
**ip** - ip-address server InfluxDB \
**db** - Database name \
**table** - Measurement/Table name
```bash
root@devops-01:~# systemctl daemon-reload
root@devops-01:~# systemctl enable iostat-to-influxdb.service
root@devops-01:~# systemctl start iostat-to-influxdb.service
root@devops-01:~# systemctl status iostat-to-influxdb.service
● iostat-to-influxdb.service - Collection metrics iostat from the set sysstat for send to influxdb
Loaded: loaded (/etc/systemd/system/iostat-to-influxdb.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-09-05 12:43:10 MSK; 1h 44min ago
Main PID: 720656 (bash)
Tasks: 4 (limit: 4515)
Memory: 1.0M
CPU: 1min 22.648s
CGroup: /system.slice/iostat-to-influxdb.service
├─ 720656 /bin/bash /root/iostat-to-influxdb.sh
├─1039830 /bin/bash /root/iostat-to-influxdb.sh
├─1039831 iostat -ky /dev/sda /dev/sda1 /dev/sda2 /dev/sda3 /dev/sdb /dev/sdb1 /dev/sdc /dev/sdc1 1 1
└─1039832 grep -w sd.
Sep 05 14:27:30 devops-01 bash[1039591]: Dload Upload Total Spent Left Speed
Sep 05 14:27:30 devops-01 bash[1039591]: [158B blob data]
Sep 05 14:27:30 devops-01 bash[1039591]: HTTP/1.1 204 No Content
Sep 05 14:27:30 devops-01 bash[1039591]: Content-Type: application/json
Sep 05 14:27:30 devops-01 bash[1039591]: Request-Id: 33045ca9-4bdf-11ee-bc29-000c294f9f2b
Sep 05 14:27:30 devops-01 bash[1039591]: X-Influxdb-Build: OSS
Sep 05 14:27:30 devops-01 bash[1039591]: X-Influxdb-Version: 1.7.11
Sep 05 14:27:30 devops-01 bash[1039591]: X-Request-Id: 33045ca9-4bdf-11ee-bc29-000c294f9f2b
Sep 05 14:27:30 devops-01 bash[1039591]: Date: Tue, 05 Sep 2023 11:27:30 GMT
Sep 05 14:27:30 devops-01 bash[1039591]:
```
## proc-stats-to-influxdb
Creating metrics for monitoring consumption memory and CPU time process. \
Script variables: \
**process_name** - value for **Tag process** via space (example: **"mysqld" "jenkins"**)
```bash
root@devops-01:~# systemctl status proc-stats-to-influxdb
● proc-stats-to-influxdb.service - Process statisctics for send to influxdb
Loaded: loaded (/etc/systemd/system/proc-stats-to-influxdb.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-09-20 11:43:40 MSK; 3min 17s ago
Main PID: 285471 (bash)
Tasks: 2 (limit: 4515)
Memory: 804.0K
CPU: 8.171s
CGroup: /system.slice/proc-stats-to-influxdb.service
├─285471 /bin/bash /root/proc-stats-to-influxdb.sh
└─296048 sleep 5
Sep 20 11:46:34 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:34 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
Sep 20 11:46:40 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:40 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
Sep 20 11:46:45 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:45 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
Sep 20 11:46:50 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:50 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
Sep 20 11:46:56 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:56 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
```
## InfluxDB
### Get data iostat
`SELECT * FROM "iostat_metrics_table" WHERE time > now() - 5m`
**Key by name selection:**
`SELECT * FROM iostat_metrics_table WHERE disk = 'sda' and time > now() - 10m`
**Founction for output of the maximum tps value:**
`SELECT read_kb_sec,write_kb_sec,MAX(tps) FROM "iostat_metrics_table" GROUP BY instance`
### Get process
Process stats Jenkins **when running the pipeline**
