{"id":13719858,"url":"https://github.com/Scribery/tlog","last_synced_at":"2025-05-07T12:30:32.159Z","repository":{"id":32704267,"uuid":"36293788","full_name":"Scribery/tlog","owner":"Scribery","description":"Terminal I/O logger","archived":false,"fork":false,"pushed_at":"2025-02-05T17:30:24.000Z","size":1470,"stargazers_count":324,"open_issues_count":110,"forks_count":53,"subscribers_count":25,"default_branch":"main","last_synced_at":"2025-02-05T18:46:37.763Z","etag":null,"topics":["elasticsearch","journald","json","log","playback","recording","rsyslog","session","stream","syslog","terminal"],"latest_commit_sha":null,"homepage":"http://scribery.github.io/tlog/","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Scribery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-05-26T12:16:37.000Z","updated_at":"2025-02-05T17:30:28.000Z","dependencies_parsed_at":"2024-01-29T16:06:46.545Z","dependency_job_id":"3af9c1a9-b0a1-4b20-a9d0-82949bbb2731","html_url":"https://github.com/Scribery/tlog","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Scribery%2Ftlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Scribery%2Ftlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Scribery%2Ftlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Scribery%2Ftlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Scribery","download_url":"https://codeload.github.com/Scribery/tlog/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252876268,"owners_count":21818152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","journald","json","log","playback","recording","rsyslog","session","stream","syslog","terminal"],"created_at":"2024-08-03T01:00:56.819Z","updated_at":"2025-05-07T12:30:29.269Z","avatar_url":"https://github.com/Scribery.png","language":"C","readme":"Tlog\n====\n\n[![Build Status](https://travis-ci.org/Scribery/tlog.svg?branch=master)](https://travis-ci.org/Scribery/tlog)\n[![Coverage Status](https://coveralls.io/repos/github/Scribery/tlog/badge.svg?branch=master)](https://coveralls.io/github/Scribery/tlog?branch=master)\n\n`Tlog` is a terminal I/O recording and playback package suitable for\nimplementing [centralized user session recording][session_recording].\n\nWhereas most other similar packages write the recorded data to a file in their\nown format, or upload it to a custom server, `tlog` sends it to a logging\nservice. Both the standard syslog and the journald interfaces are supported.\nThe recorded data is [encoded][log_format] in JSON in a way which keeps it\nhuman-readable and searchable as much as possible.\n\nThe primary purpose of logging in JSON format is to eventually deliver the\nrecorded data to a storage service such as [Elasticsearch][elasticsearch],\nwhere it can be searched and queried, and from where it can be played back.\n\n`Tlog` contains three tools: `tlog-rec` for recording terminal I/O of programs\nor shells in general, `tlog-rec-session` for recording I/O of whole terminal\nsessions, with protection from recorded users, and `tlog-play` for playing\nback the recordings. You can run `tlog-rec` for testing or recording specific\ncommands or shell sessions for yourself, or you can integrate it into another\nsolution. `Tlog-rec-session` is intended to be a user's login shell. It puts\nitself between the actual user's shell and the terminal upon user login,\nlogging everything that passes through. Lastly, `tlog-play` can playback\nrecordings from Elasticsearch or from a file, made with either `tlog-rec` or\n`tlog-rec-session`. There is no difference in log format between `tlog-rec`\nand `tlog-rec-session`.\n\nBuilding\n--------\n\nBuild dependencies are systemd, cURL, json-c, and libutempter, which development\npackages are `systemd-devel`, `json-c-devel`, `libcurl-devel`, and\n`libutempter-devel` on RPM-based distros, and `pkg-config`, `libjson-c-dev`,\n`libsystemd-journal-dev`/`libsystemd-dev`, `libcurl-*-dev`, and `libutempter-dev`\non Debian-based distros.\n\nTo build from Git you'll need `autoconf`, `automake` and `libtool` packages.\nFor creating RPM package `rpm-build` is also required.\n\nIf Systemd Journal support is not required, it can be disabled with\nconfigure's `--disable-journal` option, removing the systemd dependency as\nwell.\n\nUpdating the system utmp and wtmp files can be enabled with the\n`--enable-utempter` configure option, utilizing the libutempter library.\n\nIf you'd like to build `tlog` from the Git source tree, you need to first\ngenerate the build system files:\n\n    autoreconf -i -f\n\nAfter that, or if you're building a [release source tarball][releases], you\nneed to follow the usual configure \u0026 make approach:\n\n    ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var \u0026\u0026 make\n\nTo generate a source tarball:\n\n    ./configure --prefix=/usr --sysconfdir=/etc \u0026\u0026 make dist\n\nFrom a source tarball you can build an SRPM package:\n\n    rpmbuild -ts \u003ctarball\u003e\n\nOr an RPM package:\n\n    rpmbuild -tb \u003ctarball\u003e\n\nInstalling\n----------\n\nYou can use one of the [release binary RPM packages][releases] and install\nthem with your favorite tool. The RPM package does all the necessary setup for\nyou.\n\nOtherwise, if you built `tlog` from source, you can install it with the usual\n`make install`:\n\n    sudo make install\n\nIf you are recording other user sessions, and don't want them to be able to\naffect the recording process, make sure you use `tlog-rec-session` and its\nexecutable is SUID/SGID to separate and dedicated user and group. It doesn't\nrequire running as root and will be safer with a regular, dedicated user.\n\nYou will also need to create the session lock directory `/var/run/tlog` and\nmake it writable (only) for the user(s) `tlog-rec-session` runs as. On systems\nwhere (/var)/run is a tmpfs you will also need to make sure the session lock\ndirectory is recreated on the next boot. In that case, on systems with systemd\nyou'll need to create a tmpfiles.d configuration file, and on init.d systems -\na startup script, creating the directory for you.\n\nTesting\n-------\n\nYou can test if session recording and playback work in general with a freshly\ninstalled tlog, by recording a session into a file with `tlog-rec` and then\nplaying it back with `tlog-play`.\n\nUsage\n-----\n\n### Recording to a file\n\nTo record into a file, execute `tlog-rec` on the command line as such:\n\n    tlog-rec --writer=file --file-path=tlog.log\n\n### Playing back from a file\n\nBoth during, and after the recording you can play the session back with\n`tlog-play`:\n\n    tlog-play --reader=file --file-path=tlog.log\n\n### Recording to Systemd Journal\n\nTo record into the Systemd Journal, execute `tlog-rec` as such:\n\n    tlog-rec --writer=journal\n\nAlong with the regular JSON log messages, when recording to Journal, tlog\ncopies a few JSON fields to Journal fields (unless explicitly disabled) to aid\nsearching and extracting (parts of) particular recordings:\n\n* `TLOG_USER` - the user the recording was started as (`user` in JSON),\n* `TLOG_SESSION` - the audit session ID of the recording process\n   (`session` in JSON),\n* `TLOG_REC` - host-unique recording ID (`rec` in JSON),\n* `TLOG_ID` - log message ID within the recording (`id` in JSON).\n\n### Playing back from Systemd Journal\n\nIn general, selecting Journal log entries for playback is done using Journal\nmatches and timestamp limits, with `-M/--journal-match`, `-S/--journal-since`,\nand `-U/--journal-until` options.\n\nIn practice however, playback from Journal is usually done with a single match\nagainst the `TLOG_REC` Journal field. The `TLOG_REC` field contains a copy of\nthe `rec` field from the logged JSON data, which is a host-unique ID of the\nrecording. For example, to playback a recording which contains this message\n(output with `journalctl -o verbose` and abbreviated):\n\n    Mon 2018-01-22 10:51:48.463904 EET [s=87ea0a3f655a48cd80d7f49053860806;...\n        _AUDIT_LOGINUID=1000\n        _UID=1000\n        _GID=1000\n        _AUDIT_SESSION=2\n        _BOOT_ID=12ca5b356065453fb50adfe57007658a\n        _MACHINE_ID=2d8d017e2b1144cbbdd049a8a997911b\n        _HOSTNAME=bard\n        PRIORITY=6\n        _TRANSPORT=journal\n        _SYSTEMD_OWNER_UID=1000\n        TLOG_REC=12ca5b356065453fb50adfe57007658a-306a-26f2910\n        TLOG_USER=nkondras\n        TLOG_SESSION=2\n        TLOG_ID=1\n        MESSAGE={\"ver\":\"2.3\",\"host\":\"bard\",\"rec\":\"12ca5b356065453fb50adfe57007658a-306a-26f2910\",...\n        SYSLOG_IDENTIFIER=tlog-rec\n        _PID=12394\n        _COMM=tlog-rec\n        _SOURCE_REALTIME_TIMESTAMP=1516611108463904\n\nyou can take the ID either from the `TLOG_REC` field value directly, or from\nthe `MESSAGE` field (from the JSON `rec` field). You can then playback the\nwhole recording like this:\n\n    tlog-play -r journal -M TLOG_REC=12ca5b356065453fb50adfe57007658a-306a-26f2910\n\nWhen compiled with Systemd \u003e= 245 it is possible to read log entries from a specific\nJournal namespace using parameters `-N/--journal-namespace`.\n\n### Playing back ongoing recordings\n\nBy default, once `tlog-play` reaches the last message a recording has so far,\nit exits. However, it can be made to poll for new messages appearing with the\n`-f/--follow` option, which is useful for playing back ongoing recordings.\n\n### Playback controls\n\n`Tlog-play` accepts several command-line options affecting playback, including\n`-s/--speed` for setting playback speed multiplier, `-g/--goto` for specifying\nthe location the playback should be fast-forwarded to, and `-p/--paused` for\nstarting playback in paused state.\n\nSeveral control keys are also recognized during playback:\n\n* SPACE or `p` for pause/resume,\n* `{` and `}` for halving and doubling the playback speed,\n* `.` for stepping through the recording (on pause or during playback)\n* `G` for fast-forwarding to the end of the recording (useful with\n  `--follow`), or to the specified timestamp (see `tlog-play(8)` for details),\n* and `q` for quitting playback.\n\n### Rate-limiting recording\n\nBoth `tlog-rec` and `tlog-rec-session` can be setup to limit the rate at which\nrecording's messages are logged. `Tlog-rec` accepts three options:\n`--limit-rate=NUMBER`, `--limit-burst=NUMBER`, and `--limit-action=STRING`,\nwhich specify rate limit in bytes per second, burst limit in bytes, and the\nlimit action (pass/delay/drop), respectively. The same parameters can be\nchanged using `limit.rate`, `limit.burst`, and `limit.action` configuration\nparameters for both `tlog-rec` and `tlog-rec-session`.\n\nThe default `pass` limit action lets all the messages through unhindered,\neffectively disabling rate-limiting. You can throttle logging, and slow down\nthe user's terminal I/O using the `delay` limit action. Finally, you can\nsimply drop the captured I/O, going above the rate and burst limits, using the\n`drop` action. See `tlog-rec(8)`, `tlog-rec.conf(5)`, and\n`tlog-rec-session.conf(5)` for details.\n\n### Playing back partial recordings\n\nBy default `tlog-play` will terminate playback, if it notices out-of-order or\nmissing log messages. However, it is possible to make it ignore missing\nmessages with the `--lax` option for when you need to playback a partial\nor a corrupted recording.\n\n### Recording sessions of a user\n\nChange the shell of the user to be recorded to `tlog-rec-session`:\n\n    sudo usermod -s /usr/bin/tlog-rec-session \u003cuser\u003e\n\nLogin as the user on a text terminal. By default the recorded terminal data\nwill be delivered to Journal (if tlog was built with Journal support) or to\nsyslog with facility \"authpriv\". In both cases default priority will be\n\"info\". It will appear in Journal, if you use journald, or in\n`/var/log/auth.log` on Debian-based systems, and in `/var/log/secure` on\nFedora and derived systems.\n\nCustomize `tlog-rec-session` configuration in\n`/etc/tlog/tlog-rec-session.conf` as necessary (see `tlog-rec-session.conf(5)`\nfor details).\n\n#### Locale configuration issue on Fedora and RHEL\n\nFedora and RHEL (and some other distros) use an approach for configuring\nsystem locale, where the login shell is responsible for reading the locale\nconfiguration from a file (`/etc/locale.conf`) itself, instead of receiving it\nthrough the environment variables as most programs do. Since `su` clears\nenvironment when asked for imitation of a login shell (`su -`, or `su -l`),\nthe shell can only retrieve locale configuration from that file, in that case,\non these distros.\n\nBecause `tlog-rec-session` is not an actual shell and cannot read\n`/etc/locale.conf` file itself, it will use libc routines to read the\nenvironment, which will fall back to `ANSI_X3.4-1968` (ASCII) in these cases.\nSince nowadays pure ASCII is rarely used, `tlog-rec-session` assumes that\nlocale environment was lost, assumes the actual encoding is UTF-8, and prints\na warning.\n\nTo work that around, you can implement the approach Debian and derived distros\nuse. I.e. use PAM's pam_env.so module to read and set the locale environment\nvariables before shell or `tlog-rec-session` starts. To accomplish that on\nFedora or RHEL, put this into the `/etc/pam.d/system-auth` file, along with\nall other `session` lines:\n\n    session     required      pam_env.so readenv=1 envfile=/etc/locale.conf\n\nHowever, tlog only supports UTF-8 so far, so the above workaround only serves\nto silent the fallback warning.\n\n#### Configuring shell per-user using symlinks\n\nYou can create a symlink to `tlog-rec-sessions` containing the shell it should\nstart, in its name. E.g. if you create a symlink like this:\n\n    sudo ln -s tlog-rec-session /usr/bin/tlog-rec-session-shell-bin-zsh\n\nand execute `tlog-rec-session-shell-bin-zsh`, then `tlog-rec-session` will\nstart and execute `/bin/zsh` as the shell to record. See `tlog-rec-session(8)`\nfor details.\n\nSuch symlinks can then be assigned as login shells to certain users to have a\nspecific shell started for them, under recording.\n\n#### Configuring recording in SSSD\n\nSSSD starting with v1.16.0 allows configuring which users and/or groups should\nbe recorded (have `tlog-rec-session` started when they login), while also\npreserving the original user shell. See `sssd-session-recording(5)`.\n\n### Recording sessions to Elasticsearch\n\nRsyslog can be set up to deliver tlog messages to Elasticsearch. First of\nall, increase the maximum message size to be 1k more than the\n`tlog-rec-session` payload. The default payload is 2kB, so the `rsyslog`\nmaximum message size needs to be \"3k\" if the defaults are used:\n\n    $MaxMessageSize 3k\n\nThe line above needs to be above any network setup in `rsyslog.conf` (put it\nat the top to be safe).\n\nThen the Elasticsearch output module needs to be loaded:\n\n    $ModLoad omelasticsearch\n\n#### Massaging JSON\n\nBefore sending tlog messages to Elasticsearch they need to be reformatted\nand real time timestamp needs to be added, which can be done with this\n`rsyslog` template:\n\n    template(name=\"tlog\" type=\"list\") {\n        constant(value=\"{\")\n        property(name=\"timegenerated\"\n                 outname=\"timestamp\"\n                 format=\"jsonf\"\n                 dateFormat=\"rfc3339\")\n        constant(value=\",\")\n        property(name=\"msg\"\n                 regex.expression=\"{\\\\(.*\\\\)\"\n                 regex.submatch=\"1\")\n        constant(value=\"\\n\")\n    }\n\n#### Filtering out tlog messages\n\nThen, a rule routing messages originating from tlog to Elasticsearch\nneeds to be added. If you installed v4 or later tlog RPM package, or set up\n`tlog-rec-session` as SUID/SGID to a dedicated user yourself, then the rule\ncan use that user ID to filter genuine tlog messages securely.\n\nIf your rsyslog receives messages from journald, with the `imjournal` module,\nthen the rule condition should be:\n\n    if $!_UID == \"\u003cTLOG_UID\u003e\" then {\n        # ... actions ...\n    }\n\nNote that the above would only work with rsyslog v8.17.0 and later, due to an\nissue preventing it from parsing variable names starting with underscore.\n\nIf your rsyslog receives messages via syslog(1) socket by itself, with the\nimuxsock module, you need to enable the module's `Annotate` and `ParseTrusted`\noptions. E.g. like this:\n\n    module(load=\"imuxsock\" SysSock.Annotate=\"on\" SysSock.ParseTrusted=\"on\")\n\nAnd then the rule condition should be:\n\n    if $!uid == \"\u003cTLOG_UID\u003e\" then {\n        # ... actions ...\n    }\n\nThe `\u003cTLOG_UID\u003e` above should be replaced with the UID your `tlog-rec-session`\nruns as.\n\nOtherwise you'll need to filter by something else. For example the program\nname (`ident` argument to syslog(3)), which tlog specifies as `tlog`. In that\ncase the condition could be:\n\n    if $programname == \"tlog-rec-session\" then {\n        # ... actions ...\n    }\n\nHowever, note that any program is able to log with that program name and thus\nspoof tlog messages.\n\n#### Sending the messages\n\nOnce your rule condition is established, you can add the actual action sending\nthe messages to Elasticsearch:\n\n    action(name=\"tlog-elasticsearch\"\n           type=\"omelasticsearch\"\n           server=\"localhost\"\n           searchIndex=\"tlog-rsyslog\"\n           searchType=\"tlog\"\n           bulkmode=\"on\"\n           template=\"tlog\")\n\nThe action above would send messages formatted with the `tlog` template,\ndescribed above, to an Elasticsearch server running on localhost and default\nport, and would put them into index `tlog-rsyslog` with type `tlog`, using the\nbulk interface.\n\nAdd the following action if you want to also send tlog messages to a dedicated\nfile for debugging:\n\n    action(name=\"tlog-file\"\n           type=\"omfile\"\n           file=\"/var/log/tlog.log\"\n           fileCreateMode=\"0600\"\n           template=\"tlog\")\n\nFurther, if you don't want tlog messages delivered anywhere else you can add\nthe discard action (`~`) after both of those:\n\n    ~\n\nIf you'd like to exclude tlog messages from *any* other logs remember to put\nits rule before any other rules in `rsyslog.conf`.\n\nHere is a complete example of a rule matching messages arriving from\n`tlog-rec-session` running as user with UID 123, delivered from journald. It\nsends them to Elasticsearch running on localhost with default port, puts them\ninto `tlog-rsyslog` index with type `tlog`, using bulk interface, stores them\nin `/var/log/tlog.log` file, and then stops processing, not letting them get\nanywhere else.\n\n\tif $!_UID == \"123\" then {\n\t\taction(name=\"tlog-elasticsearch\"\n\t\t\t   type=\"omelasticsearch\"\n\t\t\t   server=\"localhost\"\n\t\t\t   searchIndex=\"tlog-rsyslog\"\n\t\t\t   searchType=\"tlog\"\n\t\t\t   bulkmode=\"on\"\n\t\t\t   template=\"tlog\")\n\t\taction(name=\"tlog-file\"\n\t\t\t   type=\"omfile\"\n\t\t\t   file=\"/var/log/tlog.log\"\n\t\t\t   fileCreateMode=\"0600\"\n\t\t\t   template=\"tlog\")\n\t\t~\n\t}\n\n### Playing back from Elasticsearch\n\nOnce you got tlog messages to Elasticsearch, you can play them back using the\nstill rudimentary `tlog-play` command-line tool. You will need to tell it to\nuse the Elasticsearch reader (`es`), supply it with the Elasticsearch base\nURL, and the query string, which would match the messages for your session.\n\nThe base URL should point to the `_search` endpoint for your type and index.\nE.g. a base URL for index `tlog-rsyslog` and type `tlog` on localhost would\nbe:\n\n    http://localhost:9200/tlog-rsyslog/tlog/_search\n\nThe query string should follow the [Elasticsearch query string\nsyntax][query_string_syntax]. E.g. to look for session #17 which happened in\nthe last week on host `server`, you can use this query string:\n\n    host:server AND timestamp:\u003e=now-7d AND session:17\n\nUse `--reader` (or just `-r`), `--es-baseurl` and `--es-query` options to\nspecify the reader, base URL, and the query string respectively. The full\ncommand for the above parameters could look like this:\n\n    tlog-play -r es \\\n              --es-baseurl=http://localhost:9200/tlog-rsyslog/tlog/_search \\\n              --es-query='host:server AND timestamp:\u003e=now-7d AND session:17'\n\nIf you're playing back an ongoing session, adding the `--follow` or `-f`\noption will make `tlog-play` wait for more messages after it plays back all\nthat were logged so far. Just like `tail -f` will wait for more lines to be\nadded to a file it's outputting.\n\nInterrupt `tlog-play` (e.g. press Ctrl-C) to stop the playback at any moment.\n\nInstead of specifying the reader and the base URL on the command line every\ntime, you can put them in `/etc/tlog/tlog-play.conf` configuration file.\n\n#### Limitations\n\nCurrently `tlog-play` functionality is limited. It doesn't provide a way to\nrewind playback, only to fast-forward. It polls files, Journal and\nElasticsearch for new messages, instead of asking to be updated when they\nappear, which impairs performance. Even though the messages contain recorded\nterminal window (re)sizes, it doesn't resize its own terminal to fit the\noutput better. Further development will be addressing these.\n\n### Contributing\nPlease read the [Contributing Guidelines](CONTRIBUTING.md) for more details\non the process for submitting pull requests.\n\n[session_recording]: http://scribery.github.io/\n[log_format]: doc/log_format.md\n[elasticsearch]: https://www.elastic.co/products/elasticsearch\n[releases]: https://github.com/Scribery/tlog/releases\n[query_string_syntax]: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax\n","funding_links":[],"categories":["C"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FScribery%2Ftlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FScribery%2Ftlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FScribery%2Ftlog/lists"}