{"id":21964529,"url":"https://github.com/intob/logd","last_synced_at":"2025-03-22T21:21:56.737Z","repository":{"id":228698417,"uuid":"739290742","full_name":"intob/logd","owner":"intob","description":"Tail \u0026 query real-time logs of many apps.","archived":false,"fork":false,"pushed_at":"2024-04-11T12:38:43.000Z","size":356,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-15T08:04:09.238Z","etag":null,"topics":["logging","monitoring","ops"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/intob.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-01-05T08:03:28.000Z","updated_at":"2024-04-11T10:31:37.000Z","dependencies_parsed_at":"2024-04-11T14:20:23.195Z","dependency_job_id":null,"html_url":"https://github.com/intob/logd","commit_stats":null,"previous_names":["swissinfo-ch/logd","inneslabs/logd","intob/logd"],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intob%2Flogd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intob%2Flogd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intob%2Flogd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intob%2Flogd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/intob","download_url":"https://codeload.github.com/intob/logd/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245021301,"owners_count":20548318,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["logging","monitoring","ops"],"created_at":"2024-11-29T12:24:06.195Z","updated_at":"2025-03-22T21:21:56.709Z","avatar_url":"https://github.com/intob.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# v0.16.0:\nv0.15 through to v0.16 has been about solving replays. Since v0, the message consists of the following:\n[_TIME_BYTES]\n```go\n// pkg/pkg.go\ntype Pkg struct {\n\tSum, TimeBytes, Payload []byte\n}\n\nfunc Unpack(data []byte, pkg *Pkg) error {\n\tif len(data) \u003c 32+15 {\n\t\treturn errors.New(\"data too short\")\n\t}\n\tpkg.Sum = data[:32]              /*32B sha256*/\n\tpkg.TimeBytes = data[32 : 32+15] /*15B time*/\n\tpkg.Payload = data[32+15:]       /* payload */\n\treturn nil\n}\n\nfunc Sign(secret, payload []byte) []byte {\n\ttimeBytes, _ := time.Now().MarshalBinary() // 15B\n\tdata := make([]byte, 0, 32 /* sha256 */ +15 /* time */ +len(payload))\n\tdata = append(data, secret...)\n\tdata = append(data, timeBytes...)\n\tdata = append(data, payload...)\n\th := sha256.Sum256(data)\n\tdata = append(data[:0], h[:32]...)\n\tdata = append(data, timeBytes...)\n\treturn append(data, payload...)\n}\n```\n## First iteration was a ring buffer of the sha256 sums, named `history`. This did not scale well, because the more packets received, the larger the history must be, but then the more expensive it is to verify each packet is not in the history.\n\n\n## Cuckoo filter\n- Simpler\n- Faster (\u003e200K/s on my machine)\n\n\n# Tail \u0026 query real-time logs of many apps.\nA simple program for streaming log data, built on Protobuf, SHA256, and UDP.\n\nThis program does not yet write log data to a file, although this is clearly an important feature to come.\n```bash\ngo run .\n```\n\n# Config\nThe program will search up to the root for a file named `logdrc.yml`.\n```yaml\n# logdrc.yml\nudp:\n  laddr_port: \":6102\"\n  guard:\n    history_size: 10000\n    sum_ttl: 100ms\napp:\n  laddr_port: \":6101\"\nstore:\n  ring_sizes:\n    /prod/my/app/http: 1000000\n    /prod/my/app/udp: 1000000\n    /debug: 10000\n  fallback_size: 1000000\n```\nYou may set your secrets in here, or as env vars.\n```bash\nexport LOGD_READ_SECRET = \"123456\"\nexport LOGD_WRITE_SECRET = \"123456\"\n```\n\n# Auth\nLogd authenticates clients for either reading or writing using SHA256 hash-based message authentication.\n\nI chose to use hash-based ephemeral message authentication with a very short TTL (100ms)\nbecause it's computationally cheap, and simple, and it's cheap to guard against replays over a short timespan.\n\nWriting is over UDP only. This will *probably* not change for sake of simplicity, although sometimes I do wish for it.\n\n# Logger\nThe simplest way to write logs is using the `logger` package.\n```go\nlog, err := logger.NewLogger(context.TODO(), \u0026logger.LoggerCfg{\n\t\tHost:   \"some.host\",\n\t\tPort:   6102,\n\t\tSecret: \"your-writer-secret\",\n\t\tMsgKey: \"/ops/joey/my-app\", // allows us to filter log data\n\t\tStdout: true,               // also write to stdout\n\t})\nlog.Info(\"🌱 this is how we write logs, baby: %s\", err)\n```\n\n## Custom integration\nLogs are written by connecting to a UDP socket.\nSee the following example. Error checks skipped for brevity.\n```go\n// dial udp\naddr, _ := conn.GetAddr(\"your.host\")\nsocket, _ := conn.Dial(addr)\n\n// serialise message using protobuf\npayload, err := proto.Marshal(\u0026cmd.Cmd{\n\t\tName: cmd.Name_WRITE,\n\t\tMsg: \u0026cmd.Msg{\n\t\t\tT:          timestamppb.Now(),\n\t\t\tKey:        \"/your/app\",\n\t\t\tLvl:        cmd.Lvl_INFO,\n\t\t\tTxt:        \"some log message\",\n\t\t},\n\t})\n\n// sign packet\nsignedMsg, _ := auth.Sign([]byte(\"your-secret\"), payload, time.Now())\n\n// write to socket\nsocket.Write(signedMsg)\n```\n\n# Protobuf\nIf you modify the protobuf spec in `cmd.proto`, you must re-generate the code.\n```bash\n# install protobuf \u0026 gen-go\nbrew install protobuf\nbrew install protoc-gen-go\n\n# generate protobuf source files\nprotoc --go_out=. cmd.proto\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintob%2Flogd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintob%2Flogd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintob%2Flogd/lists"}