{"id":18103462,"url":"https://github.com/marco-souza/spreadsheet-writer","last_synced_at":"2026-04-20T03:02:12.189Z","repository":{"id":169847302,"uuid":"645877343","full_name":"marco-souza/spreadsheet-writer","owner":"marco-souza","description":"This projects offers you an API for writing is some Google Spreadsheet using Nest.JS + Kafka","archived":false,"fork":false,"pushed_at":"2023-05-27T18:27:58.000Z","size":142,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T05:45:54.882Z","etag":null,"topics":["docker","docker-compose","kafka","nestjs","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marco-souza.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-26T16:43:51.000Z","updated_at":"2023-05-26T16:44:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"007b1b33-50e7-4a26-a070-cb7503b3b2e9","html_url":"https://github.com/marco-souza/spreadsheet-writer","commit_stats":null,"previous_names":["marco-souza/spreadsheet-writer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/marco-souza/spreadsheet-writer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marco-souza%2Fspreadsheet-writer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marco-souza%2Fspreadsheet-writer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marco-souza%2Fspreadsheet-writer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marco-souza%2Fspreadsheet-writer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marco-souza","download_url":"https://codeload.github.com/marco-souza/spreadsheet-writer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marco-souza%2Fspreadsheet-writer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32031070,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T00:18:06.643Z","status":"online","status_checked_at":"2026-04-20T02:00:06.527Z","response_time":94,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","nestjs","typescript"],"created_at":"2024-10-31T22:12:15.561Z","updated_at":"2026-04-20T03:02:12.164Z","avatar_url":"https://github.com/marco-souza.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"http://nestjs.com/\" target=\"blank\"\u003e\u003cimg src=\"https://nestjs.com/img/logo-small.svg\" width=\"200\" alt=\"Nest Logo\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Google Spreadsheet Writer\n\nThis project implements a **Google Spreadsheet** writer using [Nest](https://github.com/nestjs/nest) framework with **Kafka** as our transport layer and **Docker** +  **docker-compose** for provisioning our services.\n\nYou can see the *output spreadsheet* [here](https://docs.google.com/spreadsheets/d/1EPsPGMzc_lInQ8CdePUh4rgCLMWOfxFoHcLlfc_Cjl8/edit?usp=sharing).\n\nhttps://github.com/marco-souza/spreadsheet-writer/assets/4452113/d8e7f6c3-d438-432c-9b52-befb763c08a0\n\n### Services diagram\n\n```mermaid\n%%{init: {'theme':'drak'}}%%\ngraph LR;\n  A((api-gateway\\nPOST '/'))--emit--\u003eB[kafka-service];\n  B--receive msg--\u003eC((writer-service));\n  C--api call--\u003eD(Google\\nSpreadsheet\\nAPI);\n  style A fill:#654fFF;\n  style C fill:#654fFF;\n  style B fill:#FF458f;\n  style D fill:#FF458f;\n  style 3P-services fill:#FF458f;\n  style internal-services fill:#654fFF;\n```\n\nThe diagram above reflects the architecture of this project.\n\nIt has an *api-gateway* which *emits* events to *Kafka*, our transport layer.\n\nWe also have a *writer-service* listening kafka messages on the same topic that *api-gateway* is emitting events. The writer service will parse messages and then publish it to a given *Google Spreadsheet*, which can be specified inside the `.env`.\n\nAs we'll have a few services, I choose to create a **[Nest.js Monorepo](https://docs.nestjs.com/cli/monorepo)** to separate each service implementation.\n\n## Dependencies\n\nTo run this project you'll need to have the following items to your computer:\n\n- `nodejs`\n- `pnpm` - package manager used to develop this project\n- `docker` and `docker-compose` - to provisioning the development environment\n- `ansible-vault` (optional) - used to decrypt the `.env` credentials\n\n## Credentials\n\nTo run this project you need to have a `.env` file with the following credentials:\n\n```shell\n# your .env file\nGOOGLE_PRIVATE_KEY=\"****\"\nGOOGLE_SERVICE_ACCOUNT_EMAIL=\"****\"\nGOOGLE_PRIVATE_SPREADSHEET_ID=\"****\"\n```\n\n\u003e **Note**:\n\u003e\n\u003e You also need to invite the application's email to your spreadsheet.\n\u003e Check more details [here](https://theoephraim.github.io/node-google-spreadsheet/#/getting-started/authentication?id=service-account).\n\nIf you have the **credentials password**, just decrypt the `.env.encrypted` file to generate my development credentials by running `pnpm decrypt` and inputting the password.\n\n## Usage\n\nThis project offers a few handy commands that will help you run and test it.\n\n\u003e **Warning**:\n\u003e\n\u003e This project is not meant to run **outside docker** as the network hostname are manually defined.\n\nThe following commands are available:\n\n```bash\n# Docker commands\npnpm d:up     # alias to docker-compose up\npnpm d:down   # alias to docker-compose down\npnpm d:logs   # alias to docker-compose logs\npnpm d:exec   # alias to docker-compose exec (debug)\n\n# Useful commands\npnpm fmt      # format files\npnpm lint     # lint projects\npnpm decrypt  # decrypt .env\n\n# Running tests\npnpm tests api-gateway\npnpm tests writer\n```\n\nAfter starting the project using `pnpm d:up`, you can make a API Call for adding data to the Spreadsheet:\n\n```sh\ncurl -X POST http://localhost:3000 \\\n   -H 'Content-Type: application/json' \\\n   -d '{\n    \"date\": \"2012-04-23T18:25:43.511Z\",\n    \"message\": \"hello\",\n    \"payload\": {\n        \"light\": 1,\n        \"color\": \"#FFEE00\"\n    },\n    \"internet\": true\n  }'\n```\n\n## Implementation Details\n\nI started the project implementing a [standalone](https://docs.nestjs.com/standalone-applications) Nest.js API that receives a payload, validates it and parse into a CSV.\n\nAfter finishing this first step, I decided to turn the project into a [monorepo](https://docs.nestjs.com/cli/monorepo), splitting the code into *api-gateway* and *writer* service. These services were communicating initially via TCP.\n\nWhen I got it working, the next step was to configure kafka, so it was time to containerize the project!\n\nBefore even think of integrating kafka, I struggled a little bit with the \"network wiring\". Nest.js services usually communicates using the hostname `localhost`, but, as I was running multiple containers apps using `docker-compose`, each service has a different hostname (e.g. `api-gateway`, `writer`, `kafka`, so on). I fixed it adjusting the initial TCP setup to use the new hostname.\n\nGreat! The app is containerized, now it's time to configure kafka. I used the `bitnami/kafka` image, which also requires a `zookeeper` instance. Kafka uses it to store and manage metadata information about Kafka clusters, and also manage and organize kafka brokers (servers).\n\nAfter configuring zookeeper and kafka, I needed to configure the Nest.js transport layer to use kafka and refactor the code for connecting on it.\n\nThe only part missing now was the Google Spreadsheet integration. I did some research on that and found an interesting library called [`google-spreadsheet`](https://www.npmjs.com/package/google-spreadsheet). I had to create a factory for injecting it as dependency of the `SpreadsheetService`, responsible for adding a row on the specified spreadsheet.\n\nThe `GOOGLE_PRIVATE_SPREADSHEET_ID` environment variable can be defined in `.env`. This is the same ID used on the Spreadsheet URL. You also need to invite the `GOOGLE_SERVICE_ACCOUNT_EMAIL` the created spreadsheet - [how to configure google service credentials](https://theoephraim.github.io/node-google-spreadsheet/#/getting-started/authentication?id=service-account).\n\n## Next Steps\n\nThe project is working, but we can consider a few follow-upo tasks to improve the code base.\n\n- **move hostname configs to env vars**: To avoid breaking the app when running outside docker we need to move the hard-coded network hostnames to some configurations using the environment variable. Therefore we can customize it using docker-compose.\n- **prepare for production**: The current configuration is for development only. For a production build we need to consider improving our `Dockerfile` to generate a cleaner image, build the app with `NODE_ENV=production` and move the generate image to some Docker Register.\n- **split services dependencies**: This project is too simple to have a complex monorepo, but if we are considering pushing it to production, it would be nice to have one `package.json` by services to split the dependencies. For example, the `google-spreadsheet` module is not a direct dependency of the `api-gateway`, but it'll still be included as a dependencies as it shares the same `package.json` the the `writer` service.\n- **CI/CD automation**: It would be nice to have some workflows as our quality-gateway, enforcing linting rules and working tests. We can also consider having deployment workflows.\n- **configure docker for running e2e test**: This was not configured to run integration tests inside docker, but this would be also a nice to have feature.\n- **improve application logging**: improve the logs and consider adding some log storage service.\n\n## Stay in touch\n\n- Author - [Marco Antônio](https://www.linkedin.com/in/masouzajunior/)\n- Website - \u003chttps://marco.tremtec.com\u003e\n\n## References\n\n- Nest.js docs - \u003chttps://docs.nestjs.com/\u003e\n- Nest.js Monorepo - \u003chttps://docs.nestjs.com/cli/monorepo\u003e\n- Nest.js Kafka - \u003chttps://docs.nestjs.com/microservices/kafka\u003e\n- Nest.js Standalone Apps - \u003chttps://docs.nestjs.com/standalone-applications\u003e\n- Kafka image - \u003chttps://hub.docker.com/r/bitnami/kafka/\u003e\n- ZooKeeper image - \u003chttps://hub.docker.com/r/bitnami/zookeeper/\u003e\n- Google Spreadsheet - \u003chttps://www.npmjs.com/package/google-spreadsheet\u003e\n- Google credentials setup - \u003chttps://theoephraim.github.io/node-google-spreadsheet/#/getting-started/authentication?id=service-account\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarco-souza%2Fspreadsheet-writer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarco-souza%2Fspreadsheet-writer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarco-souza%2Fspreadsheet-writer/lists"}