{"id":19273632,"url":"https://github.com/zikwall/grower_","last_synced_at":"2025-10-08T14:28:58.613Z","repository":{"id":46796378,"uuid":"404672187","full_name":"zikwall/grower_","owner":"zikwall","description":"Grover is my experience of creating a system for collecting events, logs, etc. similar to Kafka (producers, consumers with group, topics, partitions)","archived":false,"fork":false,"pushed_at":"2021-09-25T08:13:52.000Z","size":66,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-05T14:24:39.284Z","etag":null,"topics":["go","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zikwall.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-09T10:04:21.000Z","updated_at":"2022-07-29T13:07:03.000Z","dependencies_parsed_at":"2022-08-31T01:00:21.571Z","dependency_job_id":null,"html_url":"https://github.com/zikwall/grower_","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fgrower_","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fgrower_/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fgrower_/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fgrower_/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zikwall","download_url":"https://codeload.github.com/zikwall/grower_/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240378870,"owners_count":19792039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang"],"created_at":"2024-11-09T20:43:41.595Z","updated_at":"2025-10-08T14:28:58.536Z","avatar_url":"https://github.com/zikwall.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Grower\n\n### Задумки\n\n- Партиции хранить в директориях топиков\n- Партиции разделить на сегменты (по весу или по количеству сообщений), каждый сегмент хранить в директории партиции\n- Подумать и реализовать политику хранения сегментов, например, удалять все сегменты старше 5 дней или когда сумма сегментов превышает 100k или размер превышает 1ГБ удалять первые пришедшие (FIFO)\n- Сегменты нумеровать так 0-10 сообщений 00000000000.growerlog и 10-20 сообщений 00000000010.growerlog\n- Добавить индексные файлы к сегментам с офсетами и позициями байтов, формат сегмента:\n\n```log\noffset | position\n1      | 0\n2      | 45\n```\n\n- Для быстрого поиска и чтения использовать индексный файл по аналогии с логами (00000000000.growerindex, 00000000010.growerindex) с бинарным поиском, затем делать побайтовое смещение типа Seek\n- При записи нужно будет сохранять смещение каждой записи (начало смещения)\n- Подумать над gRPC для клиентов\n- zero copy disk -\u003e socket\n\n### Общая схема работы\n\n```shell\n// create two topics with 3 and 2 partitions\n$ create topics -\u003e rainbow(3), sanbox(2)\n\n// producer 18 messages write to rainbow topic\n$ producer -\u003e (rainbow, 'Hello, Kitty 0!')\n$ producer -\u003e (rainbow, 'Hello, Kitty 1!')\n$ producer -\u003e (rainbow, 'Hello, Kitty 2!')\n$ ...\n$ producer -\u003e (rainbow, 'Hello, Kitty 10!')\n\n// describe topic in this moment, partition size max is 3 message\n$ desc topic rainbow\n-- /rainbow\n  -- /0\n    -- 00.growerlog   ['Hello, Kitty 0!', 'Hello, Kitty 3!', 'Hello, Kitty 6!']\n    -- 00.growerindex \n    -- 03.growerlog   ['Hello, Kitty 9!', 'Hello, Kitty 12!', 'Hello, Kitty 15!']\n    -- 03.growerindex\n  -- /1\n    -- 00.growerlog   ['Hello, Kitty 1!', 'Hello, Kitty 4!', 'Hello, Kitty 7!']\n    -- 00.growerindex\n    -- 03.growerlog   ['Hello, Kitty 10!', 'Hello, Kitty 13!', 'Hello, Kitty 16!']\n    -- 03.growerindex\n  -- /2\n    -- 00.growerlog   ['Hello, Kitty 2!', 'Hello, Kitty 5!', 'Hello, Kitty 8!']\n    -- 00.growerindex\n    -- 03.growerlog   ['Hello, Kitty 11!', 'Hello, Kitty 14!', 'Hello, Kitty 17!']\n    -- 03.growerindex\n```\n\n```shell\n$ S1 = subscribe -\u003e rainbow\n$ S2 = subscribe -\u003e rainbow\n\n// S1 linked partitions [0, 1]\n// S2 linked partitions [2]\n\n// batch size 3\n$ S1 -\u003e read [ all from 0 and 1 partitions ]\n$ S2 -\u003e read [ all from last partition ]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzikwall%2Fgrower_","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzikwall%2Fgrower_","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzikwall%2Fgrower_/lists"}