{"id":18436060,"url":"https://github.com/codenotary/immudb-log-audit","last_synced_at":"2025-04-07T20:32:31.076Z","repository":{"id":159846119,"uuid":"615301245","full_name":"codenotary/immudb-log-audit","owner":"codenotary","description":null,"archived":false,"fork":false,"pushed_at":"2023-06-27T10:17:47.000Z","size":1431,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-05T20:33:14.586Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codenotary.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-17T11:46:59.000Z","updated_at":"2025-01-30T11:19:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"16a24910-7844-4e26-a655-eb2a0f3f010f","html_url":"https://github.com/codenotary/immudb-log-audit","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenotary%2Fimmudb-log-audit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenotary%2Fimmudb-log-audit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenotary%2Fimmudb-log-audit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenotary%2Fimmudb-log-audit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codenotary","download_url":"https://codeload.github.com/codenotary/immudb-log-audit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247725548,"owners_count":20985724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T06:10:13.197Z","updated_at":"2025-04-07T20:32:28.497Z","avatar_url":"https://github.com/codenotary.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Log auditing secured by immudb\n\nvault-log-audit is a simple service and cli tool to store json formatted log input and audit it later in [immudb Vault](https://vault.immudb.io). \n\nNote: the previous version using directly immudb is deprecated, nevertheless can still be used. More instructions can be found [here](./doc/immudb-log-audit.md)\n\nExample json input log source\n\n```json\n{\"field1\":1, \"field2\":\"abc\", \"field3\": \"2023-03-10T22:38:26.461908Z\", \"group\": {\"field4\":\"cde\"}}\n{\"field1\":2, \"field2\":\"cde\", \"field3\": \"2023-03-10T22:38:26.461908Z\", \"group\": {\"field4\":\"cde\"}}\n{\"field1\":3, \"field2\":\"efg\", \"field3\": \"2023-04-10T22:38:26.461908Z\", \"group\": {\"field4\":\"cde\"}}\n{\"field1\":4, \"field2\":\"ijk\", \"field3\": \"2023-05-10T22:38:26.461908Z\", \"group\": {\"field4\":\"cde\"}}\n```\n\nIn addition, vault-log-audit provides few predefined log line parsers, which make it easier to configure Vault and push data:\n- pgauditjsonlog, which configures Vault with predefined set of indexes and stores them in Vault (recommended).\n- pgaudit, which transforms pgaudit audit std logs into json representation and stores them in Vault. \n- wrap, which accepts any log line and wraps it into json adding uid and timestamp and stores them in Vault.\n- default, no parsing or predefined Vault configuration, everything is up to the user. \n\n\n## Installation\n\n### Building from source\n\nTo build vault-log-audit binary, clone this repository and run\n\n```bash\nmake\n```\n\n### Docker image\n\nThe simplest way is to pull image from [docker hub](https://hub.docker.com/r/codenotary/vault-log-audit)\n\n```bash\ndocker pull codenotary/vault-log-audit\n```\n\nTo build vault-log-audit docker image locally, clone this repository and run\n\n```bash\nmake docker\n```\n\n## Overview\nvault-log-audit uses [immudb Vault](https://vault.immudb.io) to store your log data. In general, it allows for easy parsing of any log file, including not JSON, and store its content securely in Vault.  \n\n### Vault connection\nTo send data, you need to configure immudb Vault api key:\n\n```bash\nexport VAULT_API_KEY=\u003cyour api key\u003e\n```\n\n### Storing data\nTo start storing data, you need to first [create a collection and define indexes](https://vault.immudb.io/docs/guide/immudb-vault/immudb_vault_API_technical#create-a-vault), or use one of available line parsers that have them predefined.\n\nTo create a custom Vault:\n\n```bash\n./vault-log-audit create --indexes '{\"fields\":[ {\"name\":\"field1\", \"type\": \"INTEGER\" } ], \"indexes\":[ { \"fields\": [ \"field1\" ] } ]}'\n```\n\nAfter creating a collection, data can be easily pushed using tail subcommand. Currently supported sources are file and docker container. Both can be used with --follow option, which in case of files will also handle rotation and automatically track monitored files to minimize possibility of logs duplication. \n\n```bash\n./vault-log-audit tail file path/to/your/file --follow\n```\n\n```bash\n./vault-log-audit tail docker container_name --follow --stdout --stderr\n```\n\nNote: adding --log-level trace will print what lines have been parsed and stored\n\n### Reading data\nBest way to view your data is to login to [immudb Vault](https://vault.immudb.io).\n![Vault Search](./doc/images/vault_search_screen.png)\n\nNevertheless, reading data from immudb Vault can be done also from command line:\n\n```bash\n./vault-log-audit read \n```\n\nIf you need to query based on [Vault indexes](https://vault.immudb.io/docs/guide/immudb-vault/immudb_vault_API_technical#search-documents)\n\n```bash\n./vault-log-audit read '{ \"expressions\": [ {\"fieldComparisons\": [ {\"field\": \"field1\", \"operator\": \"EQ\", \"value\": 1 } ] } ] }'\n```\n\n### Auditing data\nAuditing data from immudb Vault is based on document _id\n\n```bash\n./vault-log-audit audit 6498b5d40000000000000337cc15e225\n```\n\n## Storing pgaudit logs in immudb\n[pgaudit](https://github.com/pgaudit/pgaudit) is PostgreSQL extension that enables audit logs for the database. Any kind of audit logs should be stored in secure location. immudb is fullfiling this requirement with its immutable and tamper proof features.\n\nvault-log-audit can parse PostgreSQL pgaudit logs in following formats: \n\n- Jsonlog log parser (recommended)\n- Stderr log parser. It assumes that each log line has log_line_prefix of '%m [%p] '.\n\nFor more information about those formats, visit [PostgreSQL logging documentation](https://www.postgresql.org/docs/current/runtime-config-logging.html).\n\nTo start, you need to have an PostgreSQL running with pgaudit extension enabled. As the example, [bitnami postgresql](https://hub.docker.com/r/bitnami/postgresql) which already hase pgaudi extension can be used. \n\n![Vault Dashboard](./doc/images/vault_screen.png)\n\n### Pull postgres image\n\n```bash\ndocker pull docker.io/bitnami/postgresql:15\n```\n\n### jsonlog log format\n\nThe example pgaudit jsonlog log line looks like:\n```json\n{\"timestamp\":\"2023-05-13 21:09:08.502 GMT\",\"user\":\"postgres\",\"dbname\":\"postgres\",\"pid\":138,\"remote_host\":\"172.22.0.1\",\"remote_port\":58300,\"session_id\":\"645ffc74.8a\",\"line_num\":1,\"ps\":\"CREATE TABLE\",\"session_start\":\"2023-05-13 21:09:08 GMT\",\"vxid\":\"3/44\",\"txid\":736,\"error_severity\":\"LOG\",\"message\":\"AUDIT: SESSION,1,1,DDL,CREATE TABLE,,,\\\"create table if not exists audit_trail (id VARCHAR, ts TIMESTAMP, usr VARCHAR, action INTEGER, sourceip VARCHAR, context VARCHAR, PRIMARY KEY(id));\\\",\u003cnot logged\u003e\",\"backend_type\":\"client backend\",\"query_id\":0}\n```\n\npgauditjsonlog parser will convert each jsonlog log line into following json\n\n```json\n{\"audit_type\":\"SESSION\",\"statement_id\":1,\"substatement_id\":1,\"class\":\"DDL\",\"command\":\"CREATE TABLE\",\"statement\":\"create table if not exists audit_trail (id VARCHAR, ts TIMESTAMP, usr VARCHAR, action INTEGER, sourceip VARCHAR, context VARCHAR, PRIMARY KEY(id));\",\"parameter\":\"\\u003cnot logged\\u003e\",\"uid\":\"f233afdd-304b-44e8-90ee-a7757b46c49f\",\"server_timestamp\":\"2023-05-13T22:35:07.666574128Z\",\"timestamp\":\"2023-05-13 21:09:08.502 GMT\",\"user\":\"postgres\",\"dbname\":\"postgres\",\"remote_host\":\"172.22.0.1\",\"remote_port\":58300,\"session_id\":\"645ffc74.8a\",\"line_num\":1,\"ps\":\"CREATE TABLE\",\"session_start\":\"2023-05-13 21:09:08.000 GMT\"}\n```\n\nThe indexed fields for pgauditjsonlog are\n```\nuid, user, dbname, session_id, statement_id, substatement_id, server_timestamp, timestamp, audit_type, class, command\n```\n\n### stderr log format\n\nThe example pgaudit stderr log line looks like:\n\n```\n2023-02-03 21:15:01.851 GMT [294] LOG:  AUDIT: SESSION,61,1,WRITE,INSERT,,,\"insert into audit_trail(id, ts, usr, action, sourceip, context) VALUES ('134ff2d5-2db4-44d2-9f67-9c7f5ed64967', NOW(), 'user60', 1, '127.0.0.1', 'some context')\",\u003cnot logged\u003e\n```\n\npgaudit parser will convert each stderr log line into following json\n\n```json\n{\"uid\": \"234aa2d5-2db4-44d2-9f67-9c7f5eda4967\", \"timestamp\":\"2023-03-16T08:58:44.033611299Z\",\"log_timestamp\":\"2023-03-02T21:15:01.851Z\",\"audit_type\":\"SESSION\",\"statement_id\":61,\"substatement_id\":1,\"class\":\"WRITE\",\"command\":\"INSERT\",\"statement\":\"insert into audit_trail(id, ts, usr, action, sourceip, context) VALUES ('134ff2d5-2db4-44d2-9f67-9c7f5ed64967', NOW(), 'user60', 1, '127.0.0.1', 'some context')\",\"parameter\":\"\\u003cnot logged\\u003e\"}\n```\n\nThe indexed fields for stderr are\n```\nuid, statement_id, substatement_id, server_timestamp, timestamp, audit_type, class, command\n```\n\n### How to set up\n\nYou can use [docker-compose end-to-end example](./examples/vault/pgaudit) from this repository.\nBefore starting compose, configure your API key there.\n\n```bash\ndocker-compose -f examples/vault/pgaudit/docker-compose.yml up\n```\n\nNote: you can execute ```go run test/pgaudit/utils/psql.go``` to generate audit entries.\n\nTo manually create vault-log-audit collection for pgaudit logs\n\n```bash\n# jsonlog log format\n./vault-log-audit create --parser pgauditjsonlog\n\n# or stderr log format\n./vault-log-audit create --parser pgaudit\n```\n\nTail PostgreSQL docker container logs\n\n```bash\n# jsonlog log format\n./vault-log-audit tail docker psql-postgresql-1 --parser pgauditjsonlog --stdout --stderr --follow\n\n# or stderr log format\n./vault-log-audit tail docker psql-postgresql-1 --parser pgaudit --stdout --stderr --follow\n```\n\nOptionally, adding --log-level trace will print out all lines parsed and stored. \n\nRead\n\n```bash\n./vault-log-audit read \n./vault-log-audit read '{ \"expressions\": [ {\"fieldComparisons\": [ {\"field\": \"command\", \"operator\": \"EQ\", \"value\": \"INSERT\" } ] } ] }' \n```\n\nAudit\n\n```bash\n./vault-log-audit audit 6498b5d40000000000000337cc15e225\n```\n\n## Storing kubernetes audit logs in immudb Vault\nKubernetes allow audit logging showing the track of actions taken in the cluster. To enable kubernets audit, follow the [documentation](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/).\n\nKubernetes audit logs are stored as Json structured log, so can be easily parsed and stored in immudb with vault-log-audit. For this exercise, you can use example [k8s.log](test/k8s/k8s.log) from this repository.\n\nSample json log line\n\n```json\n{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"d4652481-193e-42f6-9b78-f8651cab5dfe\",\"stage\":\"RequestReceived\",\"requestURI\":\"/api?timeout=32s\",\"verb\":\"get\",\"user\":{\"username\":\"admin\",\"uid\":\"admin\",\"groups\":[\"system:masters\",\"system:authenticated\"]},\"sourceIPs\":[\"127.0.0.1\"],\"userAgent\":\"kubectl/v1.25.6 (linux/amd64) kubernetes/ff2c119\",\"requestReceivedTimestamp\":\"2023-03-10T22:38:26.382098Z\",\"stageTimestamp\":\"2023-03-10T22:38:26.382098Z\"}\n```\n\n### How to set up\n\nFirst, create k8saudit vault.\n\n```bash\n./vault-log-audit create  --indexes '{\"fields\":[ {\"name\":\"verb\", \"type\": \"STRING\" }, {\"name\":\"stage\", \"type\": \"STRING\" }, {\"kind\":\"stage\", \"type\": \"STRING\" }, {\"name\":\"user.username\", \"type\": \"STRING\" } ], \"indexes\":[ { \"fields\": [ \"verb\", \"stage\", \"user.username\" ] } ]}'\n```\n\nTail k8s.log \n\n```bash\n./vault-log-audit tail file test/k8s/k8s.log --follow\n```\n\nRead\n\n```bash\n./vault-log-audit read \n./vault-log-audit read '{ \"expressions\": [ {\"fieldComparisons\": [ {\"field\": \"user.username\", \"operator\": \"EQ\", \"value\": \"admin\" } ] } ] }' \n\n```\n\nAudit\n\n```bash\n./vault-log-audit audit 6498b5d40000000000000337cc15e225\n```\n\n## Storing unstructured logs in immudb\nvault-log-audit provides \"wrap\" parser, which wraps any log line with autogenerated uid and timestamp. In this example, given following syslog line:\n\n```bash\nJan  6 13:57:19 DESKTOP-BLRRBQO kernel: [    0.000000] Hyper-V: privilege flags low 0xae7f, high 0x3b8030, hints 0xc2c, misc 0xe0bed7b2\n```\n\nIt will convert it to:\n```json\n{\"uid\":\"6326acda-e254-481f-b030-0144141df091\",\"log_timestamp\":\"2023-03-16T10:23:25.554276817+01:00\",\"message\":\"Jan  6 13:57:19 DESKTOP-BLRRBQO kernel: [    0.000000] Hyper-V: privilege flags low 0xae7f, high 0x3b8030, hints 0xc2c, misc 0xe0bed7b2\"}\n```\n\n### How to set up\n[Syslog file](test/syslog/syslog) is used as an example, but source can be any file our log output from a docker container.\n\nFirst, create syslog collection with \"wrap\" parser.\n\n```bash\n./vault-log-audit create --parser wrap\n```\n\nTail syslog \n\n```bash\n./vault-log-audit tail file test/syslog/syslog\n```\n\nRead\n\n```bash\n ./vault-log-audit read\n```\n\nAudit\n\n```bash\n./vault-log-audit audit 6498b5d40000000000000337cc15e225\n```\n\n## Further ideas to develop\nSources:\n - equivalent of kubectl logs\n - syslog server\n - ubuntu/rhel login tracking, auth.log\n\nDeployment:\n - add dockerfile \n\nParsers:\n - apache access logs\n - traefik/nginx logs\n - mariadb audit https://mariadb.com/kb/en/mariadb-audit-plugin/\n - ubuntu/rhel auth.log parser \n - templating/pattern matching for unstructured logs \n\nOthers:\n - OpenPolicyAgent logs ?\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodenotary%2Fimmudb-log-audit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodenotary%2Fimmudb-log-audit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodenotary%2Fimmudb-log-audit/lists"}