{"id":17280425,"url":"https://github.com/umputun/dkll","last_synced_at":"2025-10-23T23:33:23.198Z","repository":{"id":55411874,"uuid":"188120482","full_name":"umputun/dkll","owner":"umputun","description":"Agent, server and client to collect and access logs in dockerized infrastructure ","archived":false,"fork":false,"pushed_at":"2025-03-01T06:04:38.000Z","size":7993,"stargazers_count":67,"open_issues_count":3,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-02T19:07:55.394Z","etag":null,"topics":["cli","docker","logging","server","syslog"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/umputun.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["umputun"]}},"created_at":"2019-05-22T22:11:25.000Z","updated_at":"2024-11-30T20:22:08.000Z","dependencies_parsed_at":"2024-05-10T04:31:01.735Z","dependency_job_id":"f98dbe16-f7ff-4c31-8411-88329c977a83","html_url":"https://github.com/umputun/dkll","commit_stats":{"total_commits":156,"total_committers":5,"mean_commits":31.2,"dds":0.03205128205128205,"last_synced_commit":"ce56183f17a69208e91a2aa67a8baaba231c7810"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umputun%2Fdkll","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umputun%2Fdkll/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umputun%2Fdkll/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umputun%2Fdkll/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/umputun","download_url":"https://codeload.github.com/umputun/dkll/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248112364,"owners_count":21049646,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","docker","logging","server","syslog"],"created_at":"2024-10-15T09:20:32.808Z","updated_at":"2025-10-23T23:33:23.130Z","avatar_url":"https://github.com/umputun.png","language":"Go","readme":"# DKLL [![build](https://github.com/umputun/dkll/workflows/build/badge.svg)](https://github.com/umputun/dkll/actions?query=workflow%3Abuild) [![Go Report Card](https://goreportcard.com/badge/github.com/umputun/dkll?12345)](https://goreportcard.com/report/github.com/umputun/dkll)  [![Coverage Status](https://coveralls.io/repos/github/umputun/dkll/badge.svg?branch=master)](https://coveralls.io/github/umputun/dkll?branch=master) [![Docker Automated build](https://img.shields.io/docker/automated/jrottenberg/ffmpeg.svg)](https://hub.docker.com/r/umputun/dkll/)\n\n\nLogging server, agent and CLI client for dockerized infrastructure. \n\n- each host runs `dkll agent` container collecting logs from all docker containers on the host.\n- the agent can store logs locally, or/and forward them to remote syslog server.\n- server (`dkll server`) container installed on another host, acts as syslog server and stores logs.\n- server also provides http api to access logs.\n- client (`dkll client`) is a binary command-line utility to read/filter/search and follow logs.\n\n_note: `dkll agent` can be used as a standalone container without server and client. In this case it will be a local collector of \nall logs and/or forwarder to any (external or internal) syslog server._\n \n\n## Build from the source\n\n1. clone this repo - `git clone https://github.com/umputun/dkll.git`\n2. build the logger - `cd dkll \u0026\u0026 docker build -t umputun/dkll .`\n\n_alternatively use provided `Makefile`, i.e. `make test lint docker`_\n\n\n## Server\n\nServer mode runs syslog server collecting records sent by dkll agent (see below). All records parsed, analyzed and stored\nin mongodb (capped collection). Optionally, records can be sent to `\u003chost\u003e/\u003ccontainer\u003e.log` files as well as to merged `dkll.log`\n file. All files rotated and compressed automatically.\n\n### Usage\n\n1. copy provided `compose-server.yml` to `docker-compose.yml` \n2. adjust `docker-compose.yml` if needed. \n3. pull containers - `docker-compose pull`\n4. start server - `docker-compose up -d`\n\n\ncommand line options and env params:\n\n```\ndkll [OPTIONS] server [server-OPTIONS]\n\nApplication Options:\n      --dbg                              show debug info [$DEBUG]\n\nHelp Options:\n  -h, --help                             Show this help message\n\n[server command options]\n      --api-port=                      rest server port (default: 8080) [$API_PORT]\n      --syslog-port=                   syslog server port (default: 5514) [$SYSLOG_PORT]\n      --mongo=                         mongo URL [$MONGO]\n      --mongo-timeout=                 mongo timeout (default: 5s) [$MONGO_TIMEOUT]\n      --mongo-size=                    max collection size (default: 10000000000) [$MONGO_SIZE]\n      --mongo-docs=                    max docs in collection (default: 50000000) [$MONGO_DOCS]\n      --backup=                        backup log files location [$BACK_LOG]\n      --merged                         enable merged log file [$BACK_MRG]\n\n    container:\n      --limit.container.max-size=      max log size, in megabytes (default: 100) [$MAX_SIZE]\n      --limit.container.max-backups=   max number of rotated files (default: 10) [$MAX_BACKUPS]\n      --limit.container.max-age=       max age of rotated files, days (default: 30) [$MAX_AGE]\n\n    merged:\n      --limit.merged.max-size=         max log size, in megabytes (default: 100) [$MAX_SIZE]\n      --limit.merged.max-backups=      max number of rotated files (default: 10) [$MAX_BACKUPS]\n      --limit.merged.max-age=          max age of rotated files, days (default: 30) [$MAX_AGE]\n```\n\n- mongo URL specify the standard [mongodb connection string](https://docs.mongodb.com/manual/reference/connection-string/) with `db` and `collection` extra parameters, e.g. `mongodb://localhost:27017/admin?db=dkll\u0026collection=logs` \n- if `backup` defined dkll server will make `host/container.log` files in `backup` directory\n- `merged` parameter produces a single `dkll.log` file with all received records.\n\nParameters can be set in `command` directive (see docker-compose.yml) or as environment vars. \n\n### API\n\nRecords format (response):\n\n```go\ntype LogEntry struct {\n\tID        string    `json:\"id\"`         // record ID\n\tHost      string    `json:\"host\"`       // host name\n\tContainer string    `json:\"container\"`  // container\n\tPid       int       `json:\"pid\"`        // process id\n\tMsg       string    `json:\"msg\"`        // log message\n\tTs        time.Time `json:\"ts\"`         // reported time \n\tCreatedTs time.Time `json:\"cts\"`        // creation time\n}\n```\n\n- `GET /v1/last` - get last records `LogEntry`\n- `POST /v1/find` - find records for given `Request`\n\n```go\ntype Request struct {\n\tLastID     string    `json:\"id\"`                   // get records after this id\n\tLimit      int       `json:\"max\"`                  // max size of response, i.e. number of messages one request can return\n\tHosts      []string  `json:\"hosts,omitempty\"`      // list of hosts, can be exact match or regex in from of /regex/\n\tContainers []string  `json:\"containers,omitempty\"` // list of containers, can be regex as well\n\tExcludes   []string  `json:\"excludes,omitempty\"`   // list of excluded containers, can be regex\n\tFromTS     time.Time `json:\"from_ts,omitempty\"`    \n\tToTS       time.Time `json:\"to_ts,omitempty\"`\n}\n```\n\n- `POST /v1/stream?timeout=10s` - find records for given `Request` and stream it. Terminate stream on `timeout` inactivity.\n\n### Storage\n\nDKLL server uses mongo db to save and access records. It is possible and almost trivial to replace mongo with different \nbackend by implementing 2 interfaces ([Publisher](https://github.com/umputun/dkll/blob/master/app/server/forwarder.go#L22) and \n[DataService](https://github.com/umputun/dkll/blob/master/app/server/rest_server.go#L28)) with just 3 functions:\n\n- `Publish(records []core.LogEntry) (err error)`\n- `LastPublished() (entry core.LogEntry, err error)`\n- `Find(req core.Request) ([]core.LogEntry, error)`\n\n### Security and auth\n\nBoth syslog and http don't restrict access. To allow some basic auth the simplest way is to run dkll server\nbehind [nginx-le](https://github.com/umputun/nginx-le) proxy with basic auth\n[configured on nginx level](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/). \nTo limit access to syslog port (514) firewall (internal or external) cab be used.\n \nTODO: example\n \n\n## Agent\n\nAgent container runs on each host and collects logs from all containers on the host. The logs sent to remote dkll server and/or \nstored and rotated locally. The agent can intercept logs from containers configured with a logging driver that works with docker\n logs (journald and json-file).  \n\nTo deploy agent use provided `compose-agent.yml` \n\n```\ndkll [OPTIONS] agent [agent-OPTIONS]\n\nApplication Options:\n      --dbg                show debug info [$DEBUG]\n\nHelp Options:\n  -h, --help               Show this help message\n\n[agent command options]\n      -d, --docker=        docker host (default: unix:///var/run/docker.sock) [$DOCKER_HOST]\n          --syslog         enable logging to syslog [$LOG_SYSLOG]\n          --syslog-host=   syslog host (default: 127.0.0.1:514) [$SYSLOG_HOST]\n          --syslog-prefix= syslog prefix (default: docker/) [$SYSLOG_PREFIX]\n          --files          enable logging to files [$LOG_FILES]\n          --max-size=      size of log triggering rotation (MB) (default: 10) [$MAX_SIZE]\n          --max-files=     number of rotated files to retain (default: 5) [$MAX_FILES]\n          --max-age=       maximum number of days to retain (default: 30) [$MAX_AGE]\n          --mix-err        send error to std output log file [$MIX_ERR]\n          --loc=           log files locations (default: logs) [$LOG_FILES_LOC]\n      -x, --exclude=       excluded container names [$EXCLUDE]\n      -i, --include=       included container names [$INCLUDE]\n      -j, --json           wrap message with JSON envelope [$JSON]\n          --demo           demo mode, generates simulated log entries [$DEMO]\n          --demo-every=    demo interval (default: 3s) [$DEMO_EVERY]\n      \n```\n\n- at least one of destinations (`files` or `syslog`) should be allowed\n- location of log files can be mapped to host via `volume`, ex: `- ./logs:/srv/logs` (see `compose-agent.yml`)\n- both `--exclude` and `--include` flags are optional and mutually exclusive, i.e. if `--exclude` defined `--include` not allowed, and vise versa.\n\nIf you use the provided docker image, by default docker agent will run with `UID=1001`. Make sure that the access for docker socket granted for that user. Another way is specifing `APP_UID` environment variable for the agent container with either UID with docker privileges or `0` for running with root privileges.\n\n#### Demo mode\n\nDefining `--demo` or `$DEMO` switches agent to demo mode emitting fake log messages from fake containers. Everything preconfigured in `compose-demo.yml` and can be activated with `MONGO_PASSWD=\u003cany random password\u003e docker-compose -f compose-demo.yml up`.  \n\n## Client\n\nCommand line client accessing dkll server and printing the content.\n\n### Usage\n\nDKLL client should be used directly as a compiled binary. You can get precompiled \n[release](https://github.com/umputun/dkll/releases) or build it from the source (`make deploy`). \n\n```\ndkll [OPTIONS] client [client-OPTIONS]\n\n\nApplication Options:\n      --dbg       show debug info [$DEBUG]\n\nHelp Options:\n  -h, --help      Show this help message\n\n[client command options]\n      -a, --api=  API endpoint (client) [$DKLL_API]\n      -c=         show container(s) only\n      -h=         show host(s) only\n      -x=         exclude container(s)\n      -m          show syslog timestamp\n      -p          show pid\n      -s          show syslog messages\n      -f          follow mode\n      -t          tail mode\n      -n=         show N records\n      -g=         grep on entire record\n      -G=         un-grep on entire record\n          --tail= number of initial records (default: 10)\n          --tz=   time zone (default: Local)\n```\n\n* containers (-c), hosts (-h) and exclusions (-x) can be repeated multiple times. \n* both containers and hosts support regex inside \"/\", i.e. `/^something/`\n\n## Development \n\n- go v1.11 and above required\n- repository uses modules and should be cloned outside of GOPATH\n- `dkll server` tests need mongo up and running, i.e. `docker run -d --name=mongo -p 27017:27017 mongo`\n \n","funding_links":["https://github.com/sponsors/umputun"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumputun%2Fdkll","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fumputun%2Fdkll","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumputun%2Fdkll/lists"}