{"id":15789609,"url":"https://github.com/jeffdecola/data-crunch-engine","last_synced_at":"2026-04-29T17:36:27.951Z","repository":{"id":55877384,"uuid":"180748626","full_name":"JeffDeCola/data-crunch-engine","owner":"JeffDeCola","description":"A lightweight asynchronous microservice that will receive data, crunch data and return the results. You provide the data crunching algorithm.","archived":false,"fork":false,"pushed_at":"2023-07-10T07:35:13.000Z","size":888,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-10-05T22:02:31.149Z","etag":null,"topics":["data-cruncher","go","golang","message-queue","microservice","microservices-architecture","protobuf","service"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JeffDeCola.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-11T08:29:26.000Z","updated_at":"2023-01-07T19:48:00.000Z","dependencies_parsed_at":"2024-10-26T03:26:26.501Z","dependency_job_id":"0140a8ee-42cd-40e3-a50c-0d17fd3da6b6","html_url":"https://github.com/JeffDeCola/data-crunch-engine","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JeffDeCola%2Fdata-crunch-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JeffDeCola%2Fdata-crunch-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JeffDeCola%2Fdata-crunch-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JeffDeCola%2Fdata-crunch-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JeffDeCola","download_url":"https://codeload.github.com/JeffDeCola/data-crunch-engine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246516681,"owners_count":20790281,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-cruncher","go","golang","message-queue","microservice","microservices-architecture","protobuf","service"],"created_at":"2024-10-04T22:02:33.146Z","updated_at":"2026-04-29T17:36:27.946Z","avatar_url":"https://github.com/JeffDeCola.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DATA CRUNCH ENGINE\n\n[![Tag Latest](https://img.shields.io/github/v/tag/jeffdecola/data-crunch-engine)](https://github.com/JeffDeCola/data-crunch-engine/tags)\n[![jeffdecola.com](https://img.shields.io/badge/website-jeffdecola.com-blue)](https://jeffdecola.com)\n[![MIT License](https://img.shields.io/:license-mit-blue.svg)](https://jeffdecola.mit-license.org)\n[![Go Reference](https://pkg.go.dev/badge/github.com/JeffDeCola/data-crunch-engine.svg)](https://pkg.go.dev/github.com/JeffDeCola/data-crunch-engine)\n[![Go Report Card](https://goreportcard.com/badge/github.com/JeffDeCola/data-crunch-engine)](https://goreportcard.com/report/github.com/JeffDeCola/data-crunch-engine)\n[![Docker Pulls](https://badgen.net/docker/pulls/jeffdecola/data-crunch-engine?icon=docker\u0026label=pulls)](https://hub.docker.com/r/jeffdecola/data-crunch-engine/)\n\n```text\n*** THIS CHEAT SHEET IS UNDER CONSTRUCTION - CHECK BACK SOON ***\n```\n\n_A lightweight asynchronous microservice that will receive data,\ncrunch data and return the results. You provide the data crunching algorithm._\n\nTable of Contents\n\n* [OVERVIEW](https://github.com/JeffDeCola/data-crunch-engine#overview)\n* [PREREQUISITES](https://github.com/JeffDeCola/data-crunch-engine#prerequisites)\n* [SOFTWARE STACK](https://github.com/JeffDeCola/data-crunch-engine#software-stack)\n* [PROTOCOL COMPILE FOR GO](https://github.com/JeffDeCola/data-crunch-engine#protocol-compile-for-go)\n* [RUN](https://github.com/JeffDeCola/data-crunch-engine#run)\n* [CREATE BINARY](https://github.com/JeffDeCola/data-crunch-engine#create-binary)\n* [STEP 1 - TEST](https://github.com/JeffDeCola/data-crunch-engine#step-1---test)\n* [STEP 2 - BUILD (DOCKER IMAGE VIA DOCKERFILE)](https://github.com/JeffDeCola/data-crunch-engine#step-2---build-docker-image-via-dockerfile)\n* [STEP 3 - PUSH (TO DOCKERHUB)](https://github.com/JeffDeCola/data-crunch-engine#step-3---push-to-dockerhub)\n* [STEP 4 - DEPLOY (TO DOCKER)](https://github.com/JeffDeCola/data-crunch-engine#step-4---deploy-to-docker)\n* [CONTINUOUS INTEGRATION \u0026 DEPLOYMENT](https://github.com/JeffDeCola/data-crunch-engine#continuous-integration--deployment)\n\nDocumentation and References\n\n* [data-crunch-engine docker image](https://hub.docker.com/r/jeffdecola/data-crunch-engine)\n  on DockerHub\n* This repos\n  [github webpage](https://jeffdecola.github.io/data-crunch-engine/)\n  _built with\n  [concourse](https://github.com/JeffDeCola/data-crunch-engine/blob/master/ci-README.md)_\n\n## OVERVIEW\n\nThis `data-crunch-engine` is,\n\n* Written in go\n* Utilizes goroutines (concurrency)\n* Uses protobuf over NATS for messaging\n* Built to a lightweight Docker Image\n\nThis illustration shows a high level view,\n\n![IMAGE - data-crunch-engine-high-level-view - IMAGE](docs/pics/data-crunch-engine-high-level-view.svg)\n\nNotice that you may have multiple `data-crunch-engine`s running.\n\nAnd a more detailed view of the data-crunch engine,\n\n![IMAGE - data-crunch-engine - IMAGE](docs/pics/data-crunch-engine.svg)\n\n## PREREQUISITES\n\nYou will need the following go packages,\n\n```bash\ngo get -u -v github.com/sirupsen/logrus\ngo get -u -v github.com/cweill/gotests/...\n```\n\n## SOFTWARE STACK\n\n* DEVELOPMENT\n  * [go](https://github.com/JeffDeCola/my-cheat-sheets/tree/master/software/development/languages/go-cheat-sheet)\n  * gotests\n* OPERATIONS\n  * [concourse/fly](https://github.com/JeffDeCola/my-cheat-sheets/tree/master/software/operations/continuous-integration-continuous-deployment/concourse-cheat-sheet)\n    (optional)\n  * [docker](https://github.com/JeffDeCola/my-cheat-sheets/tree/master/software/operations/orchestration/builds-deployment-containers/docker-cheat-sheet)\n* SERVICES\n  * [dockerhub](https://hub.docker.com/)\n  * [github](https://github.com/)\n\nWhere,\n\n* **GUI**\n  _golang net/http package and ReactJS_\n* **Routing \u0026 REST API framework**\n  _golang gorilla/mux package_\n* **Backend**\n  _golang_\n* **Database**\n  _N/A_\n\n## PROTOCOL COMPILE FOR GO\n\nThe protocol buffer human readable file is located\n[here](https://github.com/JeffDeCola/data-crunch-engine/blob/master/proto/messages.proto)\n\nThe two interfaces have been defined as,\n\n```go\n// Check your error\nfunc checkErr(err error) {\n    if err != nil {\n        log.Fatal(\"ERROR:\", err)\n    }\n}\n```\n\n```proto\nmessage MyData {\n    int64 ID = 1;\n    int64 Data = 2;\n    string Meta = 3;\n}\n```\n\n```proto\nmessage MyResult {\n    int64 ID = 1;\n    int64 Data = 2;\n    string Meta = 3;\n    google.protobuf.Timestamp DTimeStamp = 4;\n    int64 RData = 5;\n    google.protobuf.Timestamp ProcessTime = 6;\n}\n```\n\nThis file has already been compiled, but you may recompile it using the shell script.\n\n## RUN\n\nFirst, start your NATS server,\n\n```bash\nnats-server -DV -p 4222 -a 127.0.0.1\n```\n\nIn separate terminals start the `data-engine`, the `data-crunch-engine`\nand the `results-engine` respectively,\n\n```bash\ngo run data-engine.go messages.pb.go\ngo run data-crunch-engine.go messages.pb.go\ngo run results-engine.go messages.pb.go\n```\n\nCurrently, I have a placeholder as follows,\n\nTo\n[run.sh](https://github.com/JeffDeCola/data-crunch-engine/blob/master/data-crunch-engine-code/run.sh),\n\n```bash\ncd data-crunch-engine-code\ngo run main.go\n```\n\nAs a placeholder, every 2 seconds it will print,\n\n```txt\n    INFO[0000] Let's Start this!\n    Hello everyone, count is: 1\n    Hello everyone, count is: 2\n    Hello everyone, count is: 3\n    etc...\n```\n\n## CREATE BINARY\n\nTo\n[create-binary.sh](https://github.com/JeffDeCola/data-crunch-engine/blob/master/data-crunch-engine-code/bin/create-binary.sh),\n\n```bash\ncd data-crunch-engine-code/bin\ngo build -o data-crunch-engine ../main.go\n./data-crunch-engine\n```\n\nThis binary will not be used during a docker build\nsince it creates it's own.\n\n## STEP 1 - TEST\n\nTo create unit `_test` files,\n\n```bash\ncd data-crunch-engine-code\ngotests -w -all main.go\n```\n\nTo run\n[unit-tests.sh](https://github.com/JeffDeCola/data-crunch-engine/tree/master/data-crunch-engine-code/test/unit-tests.sh),\n\n```bash\ngo test -cover ./... | tee test/test_coverage.txt\ncat test/test_coverage.txt\n```\n\n## STEP 2 - BUILD (DOCKER IMAGE VIA DOCKERFILE)\n\nTo\n[build.sh](https://github.com/JeffDeCola/data-crunch-engine/blob/master/data-crunch-engine-code/build/build.sh)\nwith a\n[Dockerfile](https://github.com/JeffDeCola/data-crunch-engine/blob/master/data-crunch-engine-code/build/Dockerfile),\n\n```bash\ncd data-crunch-engine-code\ndocker build -f build/Dockerfile -t jeffdecola/data-crunch-engine .\n```\n\nYou can check and test this docker image,\n\n```bash\ndocker images jeffdecola/data-crunch-engine:latest\ndocker run --name data-crunch-engine -dit jeffdecola/data-crunch-engine\ndocker exec -i -t data-crunch-engine /bin/bash\ndocker logs data-crunch-engine\ndocker rm -f data-crunch-engine\n```\n\nIn **stage 1**, rather than copy a binary into a docker image (because\nthat can cause issues), the Dockerfile will build the binary in the\ndocker image,\n\n```bash\nFROM golang:alpine AS builder\nRUN go get -d -v\nRUN go build -o /go/bin/data-crunch-engine main.go\n```\n\nIn **stage 2**, the Dockerfile will copy the binary created in\nstage 1 and place into a smaller docker base image based\non `alpine`, which is around 13MB.\n\n## STEP 3 - PUSH (TO DOCKERHUB)\n\nYou must be logged in to DockerHub,\n\n```bash\ndocker login\n```\n\nTo\n[push.sh](https://github.com/JeffDeCola/data-crunch-engine/blob/master/data-crunch-engine-code/push/push.sh),\n\n```bash\ndocker push jeffdecola/data-crunch-engine\n```\n\nCheck the\n[data-crunch-engine docker image](https://hub.docker.com/r/jeffdecola/data-crunch-engine)\nat DockerHub.\n\n## STEP 4 - DEPLOY (TO DOCKER)\n\nTo\n[deploy.sh](https://github.com/JeffDeCola/data-crunch-engine/blob/master/data-crunch-engine-code/deploy/deploy.sh),\n\n```bash\ncd data-crunch-engine-code\ndocker run --name data-crunch-engine -dit jeffdecola/data-crunch-engine\ndocker exec -i -t data-crunch-engine /bin/bash\ndocker logs data-crunch-engine\ndocker rm -f data-crunch-engine\n```\n\n## CONTINUOUS INTEGRATION \u0026 DEPLOYMENT\n\nRefer to\n[ci-README.md](https://github.com/JeffDeCola/data-crunch-engine/blob/master/ci-README.md)\non how I automated the above steps.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffdecola%2Fdata-crunch-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjeffdecola%2Fdata-crunch-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffdecola%2Fdata-crunch-engine/lists"}