{"id":20826745,"url":"https://github.com/cedrickchee/neocargo","last_synced_at":"2025-06-14T15:35:08.309Z","repository":{"id":65298849,"uuid":"282352549","full_name":"cedrickchee/neocargo","owner":"cedrickchee","description":"neoCargo microservices in Go with PostgreSQL, MongoDB, Terraform, Google Kubernetes Engine, and CircleCI","archived":false,"fork":false,"pushed_at":"2020-07-25T03:00:12.000Z","size":152,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-04-19T19:28:14.601Z","etag":null,"topics":["containerization","continuous-delivery","continuous-integration","go-micro","golang-application","google-cloud","grpc-service","kubernetes-cluster","microservices","monorepo","terraform-project"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cedrickchee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-25T02:07:42.000Z","updated_at":"2023-02-04T06:08:48.000Z","dependencies_parsed_at":"2023-01-16T15:15:23.591Z","dependency_job_id":null,"html_url":"https://github.com/cedrickchee/neocargo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedrickchee%2Fneocargo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedrickchee%2Fneocargo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedrickchee%2Fneocargo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedrickchee%2Fneocargo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cedrickchee","download_url":"https://codeload.github.com/cedrickchee/neocargo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243174064,"owners_count":20248228,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containerization","continuous-delivery","continuous-integration","go-micro","golang-application","google-cloud","grpc-service","kubernetes-cluster","microservices","monorepo","terraform-project"],"created_at":"2024-11-17T23:09:55.001Z","updated_at":"2025-03-12T07:23:47.788Z","avatar_url":"https://github.com/cedrickchee.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚢 neoCargo Microservices in Go\n\nneoCargo is a shipping container management platform.\n\nThis project is an example of multiple microservices implementation in Go in a [**monorepo**](https://en.wikipedia.org/wiki/Monorepo).\n\n(_Demo ([asciinema](https://asciinema.org/)) coming soon._)\n\nThe project main focus is tech proof of concept (POC). So, for that purpose, we try to keep the code simple.\n\nThe neoCargo backend consists of 3 microservices:\n\n- [Shipments](./neocargo-service-shipment)\n- [Vessels](./neocargo-service-vessel)\n- [Users + Authentication](./neocargo-service-user)\n\n## Docs\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\u003cb\u003eGetting Started\u003c/b\u003e\u003c/summary\u003e\n\n- [Tech Stack](#tech-stack)\n- [System Architecture](#system-architecture)\n- [Prerequisite](#prerequisite)\n- [Usage](#usage)\n    + [Installation](#installation)\n    + [Local Development](#local-development)\n    + [Testing](#testing)\n- [Deployment](#deployment)\n    + [Terraform a Cluster on Google Cloud](#terraform-a-cluster-on-google-cloud)\n    + [Continuous Integration (CI) Pipeline](#continuous-integration-ci-pipeline)\n\u003c/details\u003e\n\n---\n\n## Tech Stack\n\n- [Protocol Buffers](https://developers.google.com/protocol-buffers) and [gRPC](https://grpc.io/) as transport protocol\n- [go-micro](https://micro.mu/)\n- Docker container ([Alpine Linux](https://alpinelinux.org/about/) as base image) - smaller image with multi-stage build\n- [Docker Compose](https://docs.docker.com/compose/)\n- PostgreSQL or MongoDB database\n- User authentication with [JWT](https://jwt.io/)\n- [Google Cloud](https://cloud.google.com/)\n- [Kubernetes](https://kubernetes.io/)\n- [NATS](https://nats.io/)\n- [CircleCI](https://circleci.com/)\n- [Terraform](https://www.terraform.io/)\n\n## System Architecture\n\n(_Note: incomplete._)\n\nA good architecture is when services are decoupled. We will achieve that with\nevent driven architecture or pubsub.\n\nWe need to integrate the NATS broker plug-in into our services.\n\n## Prerequisite\n\n- This project assumes you're using Go 1.13 and upwards and have `export GO111MODULE=on` set.\n- Install Go\n- Install the protoc compiler\n- Install gRPC / protobuf\n- Install Go libraries:\n    - [protoc-gen-go](https://pkg.go.dev/github.com/golang/protobuf/protoc-gen-go): The protoc-gen-go binary is a protoc plugin to generate a Go protocol buffer package.\n\n## Usage\n\n### Installation\n\n- Rename file `config.env.example` in project root to `config.env`\n- Replace the variables (e.g.: `POSTGRES_PASSWORD`) in `config.env`\n\n### Local Development\n\n**Build and run Docker Compose stack**\n\nI created a simple Makefile to build, run, test, and teardown Docker Compose\nstack in local development machine.\n\nJust run in project root:\n\n```sh\n# Build stack\n$ make build\n\n# Run stack\n$ make run\n\ndocker-compose up\nCreating network \"neocargo_default\" with the default driver\nCreating database          ... done\nCreating datastore ... done\nCreating neocargo_vessel_1 ... done\nCreating neocargo_user_1     ... done\nCreating neocargo_shipment_1 ... done\nCreating neocargo_user-cli_1 ... done\nCreating neocargo_cli_1      ... done\nAttaching to datastore, database, neocargo_vessel_1, neocargo_user_1, neocargo_shipment_1, neocargo_user-cli_1, neocargo_cli_1\ndatabase     | The files belonging to this database system will be owned by user \"postgres\".\ndatabase     | This user must also own the server process.\ndatabase     |\ndatabase     | The database cluster will be initialized with locale \"en_US.utf8\".\ndatabase     | The default database encoding has accordingly been set to \"UTF8\".\ndatabase     | The default text search configuration will be set to \"english\".\ndatabase     |\ndatabase     | Data page checksums are disabled.\ndatabase     |\ndatabase     | fixing permissions on existing directory /var/lib/postgresql/data ... ok\ndatabase     | creating subdirectories ... ok\ndatabase     | selecting dynamic shared memory implementation ... posix\ndatabase     | selecting default max_connections ... 100\ndatabase     | selecting default shared_buffers ... 128MB\ndatabase     | selecting default time zone ... UTC\ndatabase     | creating configuration files ... ok\ndatabase     | running bootstrap script ... ok\ndatabase     | performing post-bootstrap initialization ... sh: locale: not found\ndatabase     | 2020-07-24 14:32:04.281 UTC [29] WARNING:  no usable system locales were found\ndatabase     | ok\ndatabase     | syncing data to disk ... initdb: warning: enabling \"trust\" authentication for local connections\ndatabase     | You can change this by editing pg_hba.conf or using the option -A, or\ndatabase     | --auth-local and --auth-host, the next time you run initdb.\ndatabase     | ok\ndatabase     |\ndatabase     |\ndatabase     | Success. You can now start the database server using:\ndatabase     |\ndatabase     |     pg_ctl -D /var/lib/postgresql/data -l logfile start\ndatabase     |\ndatabase     | waiting for server to start....2020-07-24 14:32:05.542 UTC [34] LOG:  starting PostgreSQL 12.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit\ndatabase     | 2020-07-24 14:32:05.546 UTC [34] LOG:  listening on Unix socket \"/var/run/postgresql/.s.PGSQL.5432\"\ndatabase     | 2020-07-24 14:32:05.588 UTC [35] LOG:  database system was shut down at 2020-07-24 14:32:05 UTC\ndatabase     | 2020-07-24 14:32:05.592 UTC [34] LOG:  database system is ready to accept connections\ndatabase     |  done\ndatabase     | server started\ndatabase     | CREATE DATABASE\ndatabase     |\ndatabase     |\ndatabase     | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*\ndatabase     |\ndatabase     | 2020-07-24 14:32:05.748 UTC [34] LOG:  received fast shutdown request\ndatabase     | waiting for server to shut down....2020-07-24 14:32:05.750 UTC [34] LOG:  aborting any active transactions\ndatabase     | 2020-07-24 14:32:05.751 UTC [34] LOG:  background worker \"logical replication launcher\" (PID 41) exited with exit code 1\ndatabase     | 2020-07-24 14:32:05.751 UTC [36] LOG:  shutting down\ndatabase     | 2020-07-24 14:32:05.768 UTC [34] LOG:  database system is shut down\ndatabase     |  done\ndatabase     | server stopped\ndatabase     |\ndatabase     | PostgreSQL init process complete; ready for start up.\ndatabase     |\ndatabase     | 2020-07-24 14:32:05.862 UTC [1] LOG:  starting PostgreSQL 12.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit\ndatabase     | 2020-07-24 14:32:05.862 UTC [1] LOG:  listening on IPv4 address \"0.0.0.0\", port 5432\ndatabase     | 2020-07-24 14:32:05.862 UTC [1] LOG:  listening on IPv6 address \"::\", port 5432\ndatabase     | 2020-07-24 14:32:05.866 UTC [1] LOG:  listening on Unix socket \"/var/run/postgresql/.s.PGSQL.5432\"\ndatabase     | 2020-07-24 14:32:05.910 UTC [45] LOG:  database system was shut down at 2020-07-24 14:32:05 UTC\ndatabase     | 2020-07-24 14:32:05.919 UTC [1] LOG:  database system is ready to accept connections\ncli_1        | 2020-07-24 14:32:06.198383 I | [./neocargo-cli-shipment]\ncli_1        | 2020-07-24 14:32:06.199659 I | Not enough arguments, expecting file and token\nuser-cli_1   | 2020-07-24 14:32:05.553226 I | Error creating user:  {\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"service neocargo.service.user: not found\",\"status\":\"Internal Server Error\"}\nuser-cli_1   | {\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"service neocargo.service.user: not found\",\"status\":\"Internal Server Error\"}\nshipment_1   | 2020-07-24 14:32:05  file=v2@v2.9.1/service.go:200 level=info Starting [service] neocargo.service.shipment\nshipment_1   | 2020-07-24 14:32:05  file=grpc/grpc.go:864 level=info Server [grpc] Listening on [::]:50051\nshipment_1   | 2020-07-24 14:32:05  file=grpc/grpc.go:697 level=info Registry [mdns] Registering node: neocargo.service.shipment-1741495b-a5d2-41e6-9847-560329cc5101\nvessel_1     | 2020-07-24 14:32:05  file=v2@v2.9.1/service.go:200 level=info Starting [service] neocargo.service.vessel\nvessel_1     | 2020-07-24 14:32:05  file=grpc/grpc.go:864 level=info Server [grpc] Listening on [::]:50051\nvessel_1     | 2020-07-24 14:32:05  file=grpc/grpc.go:697 level=info Registry [mdns] Registering node: neocargo.service.vessel-080f792b-9106-46de-ae27-e61c8c8e1a95\nuser_1       | --------------------------------------------------------\nuser_1       |  docker-compose-wait 2.7.3\nuser_1       | ---------------------------\nuser_1       | Starting with configuration:\nuser_1       |  - Hosts to be waiting for: [database:5432]\nuser_1       |  - Timeout before failure: 30 seconds\nuser_1       |  - TCP connection timeout before retry: 5 seconds\nuser_1       |  - Sleeping time before checking for hosts availability: 0 seconds\nuser_1       |  - Sleeping time once all hosts are available: 0 seconds\nuser_1       |  - Sleeping time between retries: 1 seconds\nuser_1       | --------------------------------------------------------\nuser_1       | Checking availability of database:5432\nuser_1       | Host database:5432 not yet available...\nuser_1       | Host database:5432 not yet available...\nuser_1       | Host database:5432 is now available!\nuser_1       | --------------------------------------------------------\nuser_1       | docker-compose-wait - Everything's fine, the application can now start!\nuser_1       | --------------------------------------------------------\nuser_1       | 2020-07-24 14:32:06.051395 I | conn: host=database user=postgres dbname=neocargo password=password sslmode=disable\nuser_1       | 2020-07-24 14:32:06.065657 I | sqlx connect done\nuser_1       | 2020-07-24 14:32:06.065737 I | DB connection OK: \u0026{0xc000242900 postgres false 0xc00030b260}\nuser_1       | 2020-07-24 14:32:06  file=v2@v2.9.1/service.go:200 level=info Starting [service] neocargo.service.user\nuser_1       | 2020-07-24 14:32:06  file=grpc/grpc.go:864 level=info Server [grpc] Listening on [::]:50051\nuser_1       | 2020-07-24 14:32:06  file=grpc/grpc.go:697 level=info Registry [mdns] Registering node: neocargo.service.user-6d44822a-0fd1-4389-915e-e7ea8378b68b\nneocargo_user-cli_1 exited with code 1\nneocargo_cli_1 exited with code 1\n```\n\nYou can stop all of your current containers by running:\n\n```sh\n# Teardown stack\n$ make stop\n```\n\n### Testing\n\nTest it all by running our CLI tool.\n\nTo run it through docker-compose, run:\n\n- Create a shipment\n\n```sh\n$ make run-cli\n\ndocker-compose run cli \\\n        ./neocargo-cli-shipment \\\n        shipment.json \\\n        eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.TJVA95OrM7E2cBab30RMHrHDcEfxjoYZgeFONFh7HgQ\nStarting datastore ... done\nStarting neocargo_vessel_1 ... done\nStarting neocargo_shipment_1 ... done\n2020-07-24 14:39:38.524923 I | [./neocargo-cli-shipment shipment.json eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.TJVA95OrM7E2cBab30RMHrHDcEfxjoYZgeFONFh7HgQ]\n2020-07-24 14:39:39.063664 I | Could not create a shipment: {\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"error fetching vessel, returned nil\",\"status\":\"Internal Server Error\"}\nmake: *** [Makefile:35: run-cli] Error 1\n```\n\n- Create a user\n\n```sh\n$ make run-user-cli\n\ndocker-compose run user-cli \\\n        -- \\\n        --name=\"John Doe\" \\\n        --email=\"john@foo.bar\" \\\n        --password=\"test#test123\" \\\n        --company=\"SpaceY\"\nStarting database ... done\nStarting neocargo_user_1 ... done\nCreated user. Response:\n2020-07-24 14:42:21.887597 I | Could not list users:  {\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"must pass a pointer, not a value, to StructScan destination\",\"status\":\"Internal Server Error\"}\n{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"must pass a pointer, not a value, to StructScan destination\",\"status\":\"Internal Server Error\"}\nmake: *** [Makefile:41: run-user-cli] Error 1\n```\n\n## Deployment\n\n### Terraform a Cluster on Google Cloud\n\nWe will create a cloud environment to host our services. We will be using\nTerraform to provision and manage our cloud cluster on Google Cloud.\n\nSteps:\n\n- Create your [Google Cloud](http://console.cloud.google.com/) project.\n- Go to \"IAM \u0026 Admin\" tab in Google Cloud console and [create a new service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). Make sure you select \"JSON\" for \"Key type\".\n- Modify the configurations in our [Infra-as-Code project](./infra).\n- Move the service key you created earlier into the project root and name it `gcloud-service-key.json`.\n\nNext, create a new cluster:\n\n```sh\n$ terraform init\n\n# View deployment plan\n$ terraform plan\n\n# Apply the changes\n$ terraform apply\n```\n\nOnce it's done, see your new cluster. Go to Google Cloud console and look for Kubernetes Engine (GKE).\nNext, deploy our containers to the cluster.\n\n#### Google Kubernetes Engine (GKE)\n\nSet-up and deploy containers into cluster using GKE.\n\nSteps:\n\n- Ensure you have the [kubectl cli installed locally](https://cloud.google.com/kubernetes-engine/docs/quickstart#local-shell):\n\n```\n$ gcloud components install kubectl\n```\n\nUsually, you'd deploy a PostgreSQL/MongoDB instance, or database instance along\nwith every service, for complete separation.\n\nThen we deploy our services, shipment-service, vessel-service, and user-service.\n\n**MongoDB containers**\n\nI created three Kubernetes deployment files for MongoDB:\n- [storage](./deployment/mongodb-deployment.yml)\n- [stateful set](./deployment/mongodb-deployment.yml)\n- [our service](./deployment/service.yml)\n\n```sh\n$ kubectl create -f ./deployment/mongodb-ssd.yml\n$ kubectl create -f ./deployment/mongodb-deployment.yml\n$ kubectl create -f ./deployment/mongodb-service.yml\n```\n\nThe result of this will be a replicated set of MongoDB containers, with stateful storage and a service exposing the datastore across our other pods.\n\n**Vessel service**\n\nI have created a [deployment file for vessel service](./deployment/vessel-service-deployment.yml).\n\nWe're pushing and pulling our Docker image from a private [Container Registry](https://cloud.google.com/container-registry/docs/pushing-and-pulling).\n\n```sh\n$ docker build -t asia.gcr.io/neocargo/vessel-service:latest .\n$ gcloud docker -- push asia.gcr.io/neocargo/vessel-service:latest\n```\n\nThen, deploy vessel-service to our cluster.\n\n```sh\n$ kubectl create -f ./deployments/vessel-service-deployment.yml\n$ kubectl create -f ./deployments/vessel-service.yml\n```\n\nDo the same for our other services.\n\n**Deploy Micro**\n\n[Deployment](./deployment/micro-deployment.yml) file.\n\nIn our service here, we expose an external load balancer, with an IP address out to the public.\n\nRun `$ kubectl get services` to get a public IP address.\n\nAfter all that's deployed, make a service call to micro container:\n\n```sh\n$ curl localhost/rpc -XPOST -d '{\n    \"request\": {\n        \"name\": \"test\",\n        \"capacity\": 100,\n        \"max_weight\": 200000,\n        \"available\": true\n    },\n    \"method\": \"VesselService.Create\",\n    \"service\": \"vessel\"\n}' -H 'Content-Type: application/json'\n```\n\nHere, our gRPC services, being proxied and converted to a web friendly format, using a sharded, MongoDB instance.\n\n### Continuous Integration (CI) Pipeline\n\nWe will set up all of this into a CI process to manage our deployments.\n\n**Set up CircleCI with our services**\n\nFirst, sign-up and create a new project in CircleCI.\n\nNext, change your build configuration (let's use our shipment-service for this).\n\nOpen [shipment-service build configuration](./neocargo-service-shipment/.circleci/config.yml)\nin your editor. In order to make this work, we need Google Cloud service account\nkey, such as the one we created in [\"Deployment\" section](#Deployment) of this\nREADME, and we need to encode this into base64 and store it as an environment\nvariable within our CircleCI build project settings. You can read more about\nenvironment variables and how to use them on [CircleCI 2.0 here](https://circleci.com/docs/2.0/env-vars/).\n\nThat's it. We have CI for one of our services. For a production-grade service,\nI recommend you run your tests first, before your deploy step.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcedrickchee%2Fneocargo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcedrickchee%2Fneocargo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcedrickchee%2Fneocargo/lists"}