{"id":19054775,"url":"https://github.com/fabioluz/proto-aggregator","last_synced_at":"2026-05-18T03:03:20.987Z","repository":{"id":228381501,"uuid":"773824363","full_name":"fabioluz/proto-aggregator","owner":"fabioluz","description":"Aggregates kafka messages serializing data with protocol buffers.","archived":false,"fork":false,"pushed_at":"2024-03-18T13:33:56.000Z","size":69,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-02T11:11:47.371Z","etag":null,"topics":["gcs","go","kafka","postgresql","protobuf"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fabioluz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-18T13:17:03.000Z","updated_at":"2024-03-19T20:04:48.000Z","dependencies_parsed_at":"2024-03-18T14:50:08.736Z","dependency_job_id":null,"html_url":"https://github.com/fabioluz/proto-aggregator","commit_stats":null,"previous_names":["fabioluz/proto-aggregator"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioluz%2Fproto-aggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioluz%2Fproto-aggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioluz%2Fproto-aggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabioluz%2Fproto-aggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fabioluz","download_url":"https://codeload.github.com/fabioluz/proto-aggregator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240110358,"owners_count":19749288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gcs","go","kafka","postgresql","protobuf"],"created_at":"2024-11-08T23:39:45.062Z","updated_at":"2025-10-04T00:52:23.670Z","avatar_url":"https://github.com/fabioluz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Proto Aggregator\n\nProto Aggregator is a tool that solves the problem of saving a copy of all Kafka messages in long-term storage. It achieves this by compiling the messages using Protocol Buffers (protobuf) and uploading them to an upstream service in batches.\n\n## Running the project\n\nYou can run this project by running the docker compose file.\n\n```\ndocker compose up\n```\n\nThis will spin up Kafka, PostgreSQL, and a Mocked Google Cloud Storage server.\n\nTo see the application in action, you can produce kafka messages by running:\n\n```\ncat examples/messages.jsonl | docker exec -i kafka kafka-console-producer.sh --producer.config /opt/bitnami/kafka/config/producer.properties --bootstrap-server 127.0.0.1:9094 --topic logs\n```\n\nBy inspecting the logs, you should see the application aggregating the messages into just one file.\nYou can download the file at:\n```\ncurl \"https://localhost:4443/storage/v1/b/logs/o/log.pb?alt=media\" -o \"all-logs.pb\"\n```\n\n## How it Works\n\nProto Aggregator efficiently handles Kafka messages by utilizing Go routines. It continuously reads messages from Kafka and accumulates them into a batch. When the batch is full or no messages are received for a specified period, it sends the batch to a channel. Another Go routine processes this channel, removing potential duplicates, serializing the messages using protobuf, and appending the compiled data to a file in Google Cloud Storage.\n\n### Removing Duplicates\n\nWhen streaming from Kafka, it is possible that you read the same message multiple times. To solve that problem Proto Aggregator uses a PostgreSQL transactions to make sure that the same message can only be saved once, even if you are using multiple aggregator instances. In the project's case, the messages are `logs` generated by the system, which should be saved in a long-term storage for future analysis.\n\nIt has a table `processed_logs` with one `log_id` column.\n\n```\nCREATE TABLE IF NOT EXISTS processed_logs (log_id UUID);\nCREATE INDEX IF NOT EXISTS processed_logs_log_id_idx ON processed_logs USING HASH (log_id);\n```\n\nWhen a batch is received by the `processBatches` function, an SQL Transaction is opened.\n\nThe system takes an `EXCLUSIVE LOCK` to `processed_logs` table, making sure that other instances will have to wait for this transaction before saving their data.\n\nNow, we need to check if the new batch does not contain a `log` that already exists in `processed_logs` table. A simple query such as\n\n```\nSELECT log_id FROM processed_logs WHERE log_in IN (\u003cHUGE LIST OF IDS\u003e)\n```\n\nwould be enough to give a list of duplicated logs, but that is very inefficient. If the list inside the `IN` clause is too big, the query would be too slow. The bigger the batch, slower the query.\n\nAs a better approach, we create a temporary table called `temp_processed_logs`.\n\n```\nCREATE TEMP TABLE temp_logs (log_id UUID) ON COMMIT DROP;\nCREATE INDEX temp_logs_log_id_idx ON temp_logs USING HASH (log_id);\n```\n\nNow, we can use Postgre `COPY` command to efficiently insert all the `log_id`s from the batch into the temporary table.\nThen, we just need to join both tables to extract duplicated `log_id`s.\n\n```\nSELECT t.log_id\nFROM temp_logs t\nINNER JOIN processed_logs p ON t.log_id = p.log_id\n```\n\nThis will give us all existing logs, which should be removed from the batch if there is any.\nWe can now insert all the new logs into the final `processed_logs` table using `COPY` command. You can see this logic in action by inspecting the `deduplicateBatch` function.\n\n### Serializing messages with Protobuf\n\nProtobuf is great for serializing strucutred objects into a binary format. However, it doesn't come with delimiters, which means that if you add many objects in the same file, you don't know when the objects start and end. This makes the deserializing process much harder, if not impossible. \n\nIn order to solve that, we can use custom delimiters to tell us when each object ends. This is a common technique used in many projects out there and it is perfect for this use case. All we need to do is to add the size of the message (in bytes) before each message, so our final compiled file will look like this:\n\n```\nsize | message | size | message | size | message ...\n```\n\nYou can see how that is done in the code by checking the implementation of `parseBatch` and `prependLittleEndianSize`.\n\n### Append messages to Google Cloud Storage\n\nNow that we have our messages compiled, we want to append that into our main in file GCS.\nAll we need to do is:\n- Upload a temporary file with the new data using the [Upload API](https://cloud.google.com/storage/docs/uploading-objects)\n- Compose the temporary file with the main file using the [Compose API](https://cloud.google.com/storage/docs/composing-objects)\n- Deleting the temporary file using [Delete API](https://cloud.google.com/storage/docs/deleting-objects)\n\n## Deserializing the logs\n\nTo deserilize the logs from the final file, you can download the file as describe above in this document, then use the following script read the file:\n\n```\nfunc main() {\n    // Open the file for reading\n    file, err := os.Open(\"your-downloaded-file.pb\")\n    if err != nil {\n        log.Fatalf(\"failed to open file: %v\", err)\n    }\n    defer file.Close()\n\n    // Read messages until EOF\n    for {\n        // Read the next message from the file\n        message, err := readMessage(file)\n        if err == io.EOF {\n            // No more messages to read, exit the loop\n            break\n        } else if err != nil {\n            log.Fatalf(\"Failed to read message: %v\", err)\n        }\n\n        // Process the message (e.g., print it)\n        fmt.Printf(\"Read message: %+v\\n\", message)\n    }\n}\n\nfunc readMessage(file *os.File) (*ProtoLog, error) {\n\t// Read the length prefix of the next message\n\tvar length int32\n\terr := binary.Read(file, binary.LittleEndian, \u0026length)\n\tif err == io.EOF {\n\t\treturn nil, io.EOF\n\t} else if err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to read message length: %v\", err)\n\t}\n\n\t// Read the message data\n\tdata := make([]byte, length)\n\t_, err = io.ReadFull(file, data)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to read message data: %v\", err)\n\t}\n\n\t// Unmarshal the message data into a MyMessage struct\n\tmessage := \u0026ProtoLog{}\n\terr = proto.Unmarshal(data, message)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to unmarshal message: %v\", err)\n\t}\n\n\treturn message, nil\n}\n\n```\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabioluz%2Fproto-aggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabioluz%2Fproto-aggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabioluz%2Fproto-aggregator/lists"}