{"id":27185080,"url":"https://github.com/threefoldtech/shim-logs","last_synced_at":"2025-04-09T17:10:30.013Z","repository":{"id":52907426,"uuid":"259742795","full_name":"threefoldtech/shim-logs","owner":"threefoldtech","description":"External binary used with shim to handle containers logs","archived":false,"fork":false,"pushed_at":"2023-10-12T07:59:49.000Z","size":34,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":12,"default_branch":"master","last_synced_at":"2023-10-13T01:07:13.486Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/threefoldtech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-28T20:20:31.000Z","updated_at":"2023-10-12T07:59:54.000Z","dependencies_parsed_at":"2022-08-23T10:50:17.132Z","dependency_job_id":null,"html_url":"https://github.com/threefoldtech/shim-logs","commit_stats":null,"previous_names":[],"tags_count":4,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fshim-logs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fshim-logs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fshim-logs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fshim-logs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/threefoldtech","download_url":"https://codeload.github.com/threefoldtech/shim-logs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248074976,"owners_count":21043490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T17:10:29.264Z","updated_at":"2025-04-09T17:10:30.008Z","avatar_url":"https://github.com/threefoldtech.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zero-OS custom shim-logs\n\nIn order to provide enduser a way to get their container logs, we expose\nto the user some tools to send logs to a remote location.\n\nIn a first time, this was implemented in `contd` but if `contd` were restarted for\nupdate or so, logs were not attached anymore. Logs handling was managed by `containerd-shim` which\nis made to handle containers and let the daemon restart without loosing connection to container.\n\nSince containerd support [external binary to handle logs](https://gitlab.dev.cncf.ci/containerd/containerd/commit/e6ae9cc64f61fc5f65bdb5a8efeeca23ac1d28ea)\nwe use this method to fetch logs next-to the container and sending them to user-defined endpoint.\n\n# Implementation\n\nFirst implementation was made in Go like the example `containerd` provided, but since this\nprocess will be running for each containers which needs logs handling, this can become a lot.\n\nResource wise, this implementation uses ~160K of memory for each process, which is a lower footprint\nthan others implementation we tried.\n\n# How it works\n\nIt follow the workflow that `containerd` provides:\n- Log binary is started with 3 more file descriptors (3, 4 and 5), which are\nrespectively stdout, stderr and ready notifier.\n- As soon as logs are ready, 5 is closed\n- 3 and 4 are read async and forwarded to specified endpoint\n\n# Debugging\n\nYou can also use this tool to simulate containerd behaviors to write your own shim\nmodule. On the `tools` directory, you have a small tool used to simulate a `shim` process\nstarted like it would be on containerd with samed `fd` attached etc.\n\n# Testing\n\nYou can test out-of-box the `shim-logs redis` endpoint by compiling `tools` and creating\nsmall local environment:\n```\nmkdir -p /var/cache/modules/contd/config/maxux\nmkdir -p /var/cache/modules/contd/logs/maxux\nchown $USER /var/cache/modules/contd/logs/maxux\n```\n\nAnd set the `/var/cache/modules/contd/config/maxux/debug-logs.json` file:\n```json\n[\n    {\n        \"type\": \"redis\",\n        \"data\": {\n            \"stdout\": \"redis://:foobared@127.0.0.2/stdout\",\n            \"stderr\": \"redis://127.0.0.1/stderr\"\n        }\n    }\n]\n```\n\nYou can then start `./shim-spawn` on the `tools` directory to forward generated text to redis.\nExample below use one connection password protected for stdout.\n\n# Cleanup Testing\n\nJust delete debug directory: `rm -rf /var/cache/modules/contd`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreefoldtech%2Fshim-logs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthreefoldtech%2Fshim-logs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreefoldtech%2Fshim-logs/lists"}