Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/shundhammer/y2log-util
Utilities for YaST Logs
https://github.com/shundhammer/y2log-util
Last synced: about 2 months ago
JSON representation
Utilities for YaST Logs
- Host: GitHub
- URL: https://github.com/shundhammer/y2log-util
- Owner: shundhammer
- License: gpl-2.0
- Created: 2021-03-03T09:14:45.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2024-04-24T16:18:22.000Z (8 months ago)
- Last Synced: 2024-10-13T00:53:03.319Z (3 months ago)
- Language: Shell
- Size: 33.2 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# y2log-util: Utilities for YaST Logs
_Author: Stefan Hundhammer _
_License: GPL V2_
This contains some scripts for handling and analyzing y2logs as generated by
YaST and the _save_y2logs_ script.
They are provided in the hope that they may be useful, but there is absolutely
no warranty and no support. Use at your own risk.
# Merging and Splitting
## y2log-merge
This merges all the parts of y2logs into one single file, so you no longer have
to deal with `y2log`, `y2log-1.gz`, `y2log-2.gz` etc. separately.
It generates one single file `y2log-full.log` from them.
Usage:
cd YaST2
y2log-merge
You will get a new file `y2log-full.log`.
## y2log-split
This splits one y2log (by default `y2log-full.log`) into several parts, one for
each `y2base` call (i.e. YaST start). The resulting files include a numeric
part for the chronological order of invocation and the name of the YaST module
called.
Usage:
cd YaST2
y2log-merge
y2log-split
or
cd YaST2
y2log-split y2log
You will get files like
- y2-00-installation.log
- y2-01-menu.log
- y2-02-sw_single.log
- y2-03-repositories.log
- y2-04-repositories.log
- y2-05-repositories.log
- y2-06-repositories.log
- y2-07-dasd.log
- y2-08-dasd.log
- y2-09-dasd.log
## y2log-headers
This shows the headers (the full `y2base` call) from split logs (that were
created by `y2log-split`):
% y2log-headers
y2-00-installation.log: 2021-02-26 22:29:37 y2base called with ["installation", "--arg", "initial", "ncurses"]
y2-01-menu.log: 2021-02-26 17:42:37 y2base called with ["menu", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-02-sw_single.log: 2021-02-26 17:42:40 y2base called with ["sw_single", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-03-repositories.log: 2021-02-26 17:42:52 y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-04-repositories.log: 2021-02-26 17:48:14 y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-05-repositories.log: 2021-02-26 17:51:45 y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-06-repositories.log: 2021-02-26 17:58:07 y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-07-dasd.log: 2021-02-26 18:06:22 y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-08-dasd.log: 2021-02-27 00:07:52 y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2-09-dasd.log: 2021-02-27 00:11:57 y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
## y2log-y2base-calls
This shows y2base calls from the raw (gzipped and not split) `y2log` and
`y2log-[1-9].gz` files:
% y2log-y2base-calls
y2log-1.gz: 2021-02-26 22:29:37 y2base called with ["installation", "--arg", "initial", "ncurses"]
y2log: 2021-02-26 17:42:37 y2base called with ["menu", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log: 2021-02-26 17:42:40 y2base called with ["sw_single", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log: 2021-02-26 17:42:52 y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log: 2021-02-26 17:48:14 y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log: 2021-02-26 17:51:45 y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log: 2021-02-26 17:58:07 y2base called with ["repositories", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log: 2021-02-26 18:06:22 y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log: 2021-02-27 00:07:52 y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
y2log: 2021-02-27 00:11:57 y2base called with ["dasd", "ncurses", "-name", "YaST2", "-icon", "yast"]
## y2log-rm-split-logs
This removes the files created by `y2log-split`, i.e. all files that match
`y2-[0-9][0-9]*.log`.
Caution: This does not ask for confirmation, it just does it.
# Cleaning and Stripping
The scripts in this section are intended for simplifying y2logs.
## y2log-clean
This is a wrapper script to strip off common cruft in y2logs, to simplify the
line header, and to downgrade misleading Cheetah error log lines from `<3>` to
`<1>`.
All operations are done in-place, so make sure to copy the original file if you
want to keep the untouched version.
This script can be used commonly for y2logs with or without debug logging.
Usage:
y2log-clean y2-??-*.log
## y2log-strip-irregular
This strips off irregular lines from one or more y2log files in-place. It
leaves only those that match the standard log line pattern:
2024-02-30 14:07:42 <0> myhost(4711) [Ruby]
This is a remedy against all that cruft from random places that fills up the
y2logs like this:
// global string String::Quote (string)
// global string String::UnQuote (string)
-------------------------------- accept ----------------------------
// filename: "/usr/share/YaST2/data/languages/language_af_ZA.ycp"
textdomain "languages_db";
return $["af_ZA":["Afrikaans", "Afrikaans", ".UTF-8", "", _("Afrikaans")], "keyboard":"english-us"];
It is hopeless that those who thought it would be a good idea to just dump that
stuff into the y2log will ever clean up their mess; that hasn't happened over
the last 15-20 years, so it won't happen in the next 15-20 years.
This `y2log-strip-irregular` script gets rid of it.
This is included in the more general `y2log-clean` script.
## y2log-strip-date-hostname
This strips off the date part of the log line headers, the hostname and the
PID, so it considerably shortens the line header:
From
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:42: Hello world!
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:43: ...
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:44: ...
To
14:07:42 <1> [Ruby] foobar.rb:42: Hello world!
14:07:42 <1> [Ruby] foobar.rb:43: ...
14:07:42 <1> [Ruby] foobar.rb:44: ...
This is included in the more general `y2log-clean` script.
The rationale is that those parts are always the same anyway for each log
line in all normal cases, so we can save some screen space; and, more
importantly, give Bugzilla less opportunities to butcher any pasted content,
like it loves to do.
## y2log-strip-date
This does only the date stripping part of `y2log-strip-date-hostname`:
From
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:42: Hello world!
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:43: ...
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:44: ...
To
14:07:42 <1> myhost(4711) [Ruby] foobar.rb:42: Hello world!
14:07:42 <1> myhost(4711) [Ruby] foobar.rb:43: ...
14:07:42 <1> myhost(4711) [Ruby] foobar.rb:44: ...
This might be useful if the hostname actually changes, or if multiple YaST
processes are logging into the same file at the same time.
## y2log-strip-hostname
This does only the hostname/PID stripping part of `y2log-strip-date-hostname`:
From
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:42: Hello world!
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:43: ...
2024-02-30 14:07:42 <1> myhost(4711) [Ruby] foobar.rb:44: ...
To
2024-02-30 14:07:42 <1> [Ruby] foobar.rb:42: Hello world!
2024-02-30 14:07:42 <1> [Ruby] foobar.rb:43: ...
2024-02-30 14:07:42 <1> [Ruby] foobar.rb:44: ...
This might be useful if the date actually changes in the same y2log, or if
multiple y2log files stretching over multiple dates are involved.
## y2log-strip-cruft
This strips common completely useless messages that only obscur important ones:
```
[Pkg] modules/PackageCallbacks.rb:9999 Pkg Builtin called: Callback...
[libstorage] SystemCmd.cc(doExecute):999 stopwatch ...
[libstorage] SystemCmd.cc(execute):999 timestamp ...
[libstorage] SystemCmd.cc(step_fork_and_exec):999 child_pid:9999
[libstorage] SystemCmd.cc(SystemCmd):999 constructor SystemCmd(...)
[libstorage] SystemCmd.cc(add_line):713 line stdout[99] 'loop99'
[agent-ini] IniParser.cc(getTimeStamp):999 Unable to stat '/etc/sysctl.conf'
[agent-ini] IniParser.cc(getTimeStamp):999 Unable to stat '/etc/install.inf'
```
This also shortens the boilerplate stdout output for libstorage and cheetah shell commands:
```
2024-04-16 17:36:56 <1> myhost(9999) [libstorage] SystemCmd.cc(add_line):713 line stdout[0] 'foo'
2024-04-16 17:36:56 <1> myhost(9999) [libstorage] SystemCmd.cc(add_line):713 line stdout[1] 'bar'
2024-04-16 17:36:56 <1> myhost(9999) [libstorage] SystemCmd.cc(add_line):713 line stdout[2] 'baz'
```
to
```
2024-04-16 17:36:56 <1> myhost(9999) 'foo'
2024-04-16 17:36:56 <1> myhost(9999) 'bar'
2024-04-16 17:36:56 <1> myhost(9999) 'baz'
```
```
2024-04-16 15:03:48 <1> myhost(9999) [Ruby] lib/cheetah.rb(log_stream_line):211 Standard output: foo
2024-04-16 15:03:48 <1> myhost(9999) [Ruby] lib/cheetah.rb(log_stream_line):211 Standard output: bar
2024-04-16 15:03:48 <1> myhost(9999) [Ruby] lib/cheetah.rb(log_stream_line):211 Standard output: baz
```
to
```
2024-04-16 15:03:48 <1> myhost(9999) foo
2024-04-16 15:03:48 <1> myhost(9999) bar
2024-04-16 15:03:48 <1> myhost(9999) baz
```
The context is still clear from the line immediately preceding the block of
output which contains the executed command.
## y2log-clean-cheetah
This cleans up misleading error log lines written by Cheetah which just logs
all regular stdout of commands that it invokes as errors. Because reasons. And
despite the bug report about this that was filed years ago.
This makes searching for real errors a royal PITA because a simple `dracut` run
will easily flood the y2log with some hundred error log lines, drowning out all
real errors.
This script simply downgrades all `<3>` (_error_) log lines to `<1>`
(_milestone_) if `cheetah` is found in the same line. This operation is
in-place in the file.
Usage:
y2log-clean-cheetah y2log
This is included in the more general `y2log-clean` script.
## y2log-strip-debug
This strips off debug lines from one or more y2log files in-place; it removes
lines with log level `<0>` like these:
2024-02-30 14:07:42 <0> myhost(4711) [Ruby]
14:07:42 <0> myhost(4711) [Ruby]
14:07:42 <0> [Ruby]
14:07:42 <0> myhost(4711) [Ruby]
Sometimes it's useful to have debug logging, but mostly it's just additional
noise that drowns out useful information.
Since this operates in-place, it is advised to copy the original file with
debug logging away to a safe place before using this script:
cp y2log y2log-debug
y2log-strip-debug y2log
It is advised to do this together with the other scripts here:
y2log-clean y2log
cp y2log y2log-debug
y2log-strip-debug y2log
Now there are two cleaned-up y2logs: `y2log` without debug logging and
`y2log-debug` with.
# Extracting Embedded Information
## y2log-get-xml-profile
This extracts an XML AutoYaST profile from an y2log and formats it in a
human-readable way, with \n translated to newlines etc. Typically, this needs
to be an y2log with debug logging.
Usage:
y2log-get-xml-profile y2log-debug >profile.xml
At the time of this writing, this uses the first XML profile that it
finds. This might change in the future.
## y2log-grep-xml-profile
This searches one or more y2log files for XML AutoYaST profiles. It cuts off
the output after a reasonable amount of characters to avoid flooding the
terminal window.
```
% y2log-grep-xml-profile y2log-debug
02:25:14 ScriptingAgent.cc(executeSubagentCommand):592 opt: "\n\n\n \n \n
showopts\n \n\n\n
\n\n\n
...
...
...
21:26:15 ../../libscr/src/include/scr/Y2AgentComponent.h(evaluate):100 Going to evaluate `Write
(.string, "/tmp/YaST2-06833-60VX5P/autoinst.xml", "\n\n\n \n\n\n
*** Number of profiles: 24
```
## y2log-grep-systemcmd
This extracts the external commands executed with the libstorage `SystemCmd` class to get a good
impression of storage operations:
y2log-grep-systemcmd
By default, this uses `y2log-full.log` (the result of `y2log-merge`), and if there is no file with
that name, `y2log`. You can also specify a file on the command line.
Since libstorage executes a lot of `udevadm settle` commands that are normally not very relevant, it
is usually a good idea to filter those out:
```
% y2log-grep-systemcmd | grep -v 'udevadm settle'
02:25:13 /usr/bin/uname -m
02:25:13 /usr/bin/test -d /sys/firmware/efi/efivars
02:25:13 /usr/bin/getconf PAGESIZE
02:25:13 /bin/ls -1 --sort=none /sys/block
02:25:13 /usr/bin/stat --format %f /dev/sda
02:25:13 /usr/bin/udevadm info /dev/sda
02:25:14 /usr/bin/uname -m
02:25:14 /usr/bin/test -d /sys/firmware/efi/efivars
02:25:14 /usr/bin/getconf PAGESIZE
21:26:15 /usr/bin/uname -m
21:26:15 /usr/bin/test -d /sys/firmware/efi/efivars
21:26:15 /usr/bin/getconf PAGESIZE
21:30:00 /sbin/multipath -d -v 2
21:30:00 /sbin/mdadm --assemble --scan
21:30:00 /sbin/dmraid --activate yes --no_partitions
21:30:00 /sbin/lvs --reportformat json --config log { command_names = 0 prefix = "" } --units b --nosuffix --all --options lv_name,lv_uuid,vg_name,vg_uuid,lv_role,lv_attr,lv_size,origin_size,segtype,stripes,stripe_size,chunk_size,pool_lv,pool_lv_uuid,origin,origin_uuid,data_lv,data_lv_uuid,metadata_lv,metadata_lv_uuid
21:30:01 /sbin/blkid -c /dev/null
21:30:01 /usr/bin/uname -m
21:30:01 /usr/bin/test -d /sys/firmware/efi/efivars
21:30:01 /usr/bin/getconf PAGESIZE
21:30:01 /bin/ls -1 --sort=none /sys/block
21:30:01 /usr/bin/stat --format %f /dev/sda
21:30:01 /usr/bin/udevadm info /dev/sda
21:30:01 /usr/bin/lsscsi --version
21:30:01 /usr/bin/lsscsi --transport
21:30:01 /sbin/multipath -d -v 2 -ll
21:30:01 /sbin/dmraid --sets=active -ccc
21:30:01 /sbin/blkid -c /dev/null
21:30:01 /usr/sbin/parted --version
21:30:01 /usr/sbin/parted --script --machine /dev/sda unit s print
```