{"id":15137238,"url":"https://github.com/gmhafiz/go8","last_synced_at":"2025-04-08T08:11:52.689Z","repository":{"id":38739172,"uuid":"282658795","full_name":"gmhafiz/go8","owner":"gmhafiz","description":"Go + Postgres + Chi Router + sqlx + ent + authentication + testing + opentelemetry Starter Kit for API Development","archived":false,"fork":false,"pushed_at":"2025-02-15T10:47:59.000Z","size":4158,"stargazers_count":462,"open_issues_count":2,"forks_count":48,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-01T07:40:26.830Z","etag":null,"topics":["api","authentication","e2e-testing","ent","go","golang","integration-testing","opentelemetry","starter-kit","unit-test"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gmhafiz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-26T13:46:10.000Z","updated_at":"2025-03-23T15:25:57.000Z","dependencies_parsed_at":"2023-12-30T03:30:39.722Z","dependency_job_id":"58638b84-b3b4-4184-ba76-418e84d775e1","html_url":"https://github.com/gmhafiz/go8","commit_stats":{"total_commits":245,"total_committers":3,"mean_commits":81.66666666666667,"dds":0.09795918367346934,"last_synced_commit":"7c488d77439beabdf3ec129dc52e2b5d3e9213a3"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmhafiz%2Fgo8","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmhafiz%2Fgo8/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmhafiz%2Fgo8/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmhafiz%2Fgo8/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gmhafiz","download_url":"https://codeload.github.com/gmhafiz/go8/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247801170,"owners_count":20998339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","authentication","e2e-testing","ent","go","golang","integration-testing","opentelemetry","starter-kit","unit-test"],"created_at":"2024-09-26T07:00:33.390Z","updated_at":"2025-04-08T08:11:52.667Z","avatar_url":"https://github.com/gmhafiz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n            .,*/(#####(/*,.                               .,*((###(/*.\n        .*(%%%%%%%%%%%%%%#/.                           .*#%%%%####%%%%#/.\n      ./#%%%%#(/,,...,,***.           .......          *#%%%#*.   ,(%%%#/.\n     .(#%%%#/.                    .*(#%%%%%%%##/,.     ,(%%%#*    ,(%%%#*.\n    .*#%%%#/.    ..........     .*#%%%%#(/((#%%%%(,     ,/#%%%#(/#%%%#(,\n    ./#%%%(*    ,#%%%%%%%%(*   .*#%%%#*     .*#%%%#,      *(%%%%%%%#(,.\n    ./#%%%#*    ,(((##%%%%(*   ,/%%%%/.      .(%%%#/   .*#%%%#(*/(#%%%#/,\n     ,#%%%#(.        ,#%%%(*   ,/%%%%/.      .(%%%#/  ,/%%%#/.    .*#%%%(,\n      *#%%%%(*.      ,#%%%(*   .*#%%%#*     ./#%%%#,  ,(%%%#*      .(%%%#*\n       ,(#%%%%%##(((##%%%%(*    .*#%%%%#(((##%%%%(,   .*#%%%##(///(#%%%#/.\n         .*/###%%%%%%%###(/,      .,/##%%%%%##(/,.      .*(##%%%%%%##(*,\n              .........                ......                .......\nA starter kit for Go API development. Inspired by [How I write HTTP services after eight years](https://pace.dev/blog/2018/05/09/how-I-write-http-services-after-eight-years.html).\n\nHowever, I wanted to use [chi router](https://github.com/go-chi/chi) which is more common in the community, [sqlx](https://github.com/jmoiron/sqlx) for database operations and design towards layered architecture (handler -\u003e business logic -\u003e database).\n\nIn short, this kit is a Go + Postgres + Chi Router + sqlx + ent + authentication + testing starter kit for API development.\n\n# Motivation\n\nOn the topic of API development, there are two opposing camps between using framework (like [echo](https://github.com/labstack/echo), [gin](https://github.com/gin-gonic/gin), [buffalo](http://gobuffalo.io/)) and starting small and only adding features you need through various libraries. \n\nHowever, the second option isn't that straightforward. you will want to structure your project in such a way that there are clear separation of functionalities for your controller, business logic and database operations. Dependencies need to be injected from outside to inside. Being modular, swapping a library like a router or database library to a different one becomes much easier.\n\n# Features\n\nThis kit is composed of standard Go library together with some well-known libraries to manage things like router, database query and migration support.\n\n  - [x] Framework-less and net/http compatible handlers\n  - [x] Router/Mux with [Chi Router](https://github.com/go-chi/chi)\n  - [x] Database Operations with [sqlx](https://github.com/jmoiron/sqlx)\n  - [x] Database Operations with [ent](https://entgo.io/docs/getting-started)\n  - [x] Database migration with [goose](https://github.com/pressly/goose)\n  - [x] Input [validation](https://github.com/go-playground/validator) that returns multiple error strings\n  - [x] Read all configurations using a single `.env` file or environment variable\n  - [x] Clear directory structure, so you know where to find the middleware, domain, server struct, handle, business logic, store, configuration files, migrations etc. \n  - [x] (optional) Request log that logs each user uniquely based on host address\n  - [x] CORS\n  - [x] Scans and auto-generate [Swagger](https://github.com/swaggo/swag) docs using a declarative comments format \n  - [x] Custom model JSON output\n  - [x] Filters (input DTO), Resource (output DTO) for pagination parsing and custom response respectively.\n  - [x] Cache layer\n  - [x] Authentication using cookie-based session\n  - [x] Uses [Task](https://taskfile.dev) to simplify various tasks like mocking, linting, test coverage, hot reload etc\n  - [x] Unit testing of repository, use case, and handler using mocks and [dockertest](https://github.com/ory/dockertest)\n  - [x] Integration testing\n  - [x] End-to-end test using ephemeral docker containers\n  - [x] OpenTelemetry with Grafana, otel-collector, Prometheus, Loki, and Jaeger\n\n# Quick Start\n\nIt is advisable to use the latest supported [Go version](https://go.dev/dl/go1.24.0.linux-amd64.tar.gz) (\u003e= v1.23). Optionally `docker` and `docker-compose` for easier start up. There is a quick guide for Debian in the [appendix](#appendix).\n\nGet it\n\n```shell\ngit clone https://github.com/gmhafiz/go8\ncd go8\n```\n\nThe application depends on a database. Ideally applications load configurations from [environment variable](https://12factor.net/config) (Method A) or from a vault. \n\n(Method A)\n\nSet values by exporting them into environment variables\n\n```shell\nexport DB_DRIVER=postgres\nexport DB_HOST=localhost\nexport DB_PORT=5432\nexport DB_USER=postgres\nexport DB_PASS=password\nexport DB_NAME=go8_db\n```\n\n(Method B)\n\nIt is also possible to set them in an `.env` file. Just make sure this file is ignored in `.gitignore` because it should never be checked into source control.\n\n Fill in your database credentials in `.env` by making a copy of `env.example` first.\n```shell\ncp env.example .env\nvim .env\n```\n\n## Database\n\nHave a database ready either by installing them yourself or use the following command. The `docker-compose-infra.yml` will use database credentials set in either `.env` file or environment variables which is initialized in the previous step. In addition, creating database this way will also create an integration database as well.\n\n```sh\ndocker-compose -f docker-compose-infra.yml up -d postgres\n```\n\nOnce the database is up, tables and seed data needs to be created using a migration system with the following command. The seed data is needed for authentication later.\n\n```shell\ngo run cmd/migrate/main.go\ngo run cmd/seed/main.go\n```\n\nYou will see a bunch of dependencies download for a first time run followed by the sql migration files\n\n```\n2023/09/16 19:06:56 connecting to database... \n2023/09/16 19:06:56 database connected\n2023/09/16 19:06:56 OK   20221213140051_create_books.sql (28.26ms)\n2023/09/16 19:06:56 OK   20221213140144_create_authors.sql (19.26ms)\n2023/09/16 19:06:56 OK   20221213140219_create_book_authors.sql (13.34ms)\n2023/09/16 19:06:56 OK   20230409004013_create_users_table.sql (14.13ms)\n2023/09/16 19:06:56 OK   20230409004420_create_sessions_table.sql (9.67ms)\n2023/09/16 19:06:56 goose: successfully migrated database to version: 20230409004420\n```\n\nRun the API with the following command.\n\n```shell\ngo run cmd/go8/main.go\n```\n\nYou will see the address of the API is running at.\n\n```shell\n2024/08/20 12:52:43 Starting API version: v0.1.0\n{\"time\":\"2024-08-20T12:52:43.809953984+10:00\",\"level\":\"INFO\",\"msg\":\"connecting to database... \"}\n{\"time\":\"2024-08-20T12:52:43.820974939+10:00\",\"level\":\"INFO\",\"msg\":\"database connected\"}\n        .,*/(#####(/*,.                               .,*((###(/*.\n    .*(%%%%%%%%%%%%%%#/.                           .*#%%%%####%%%%#/.\n  ./#%%%%#(/,,...,,***.           .......          *#%%%#*.   ,(%%%#/.\n .(#%%%#/.                    .*(#%%%%%%%##/,.     ,(%%%#*    ,(%%%#*.\n.*#%%%#/.    ..........     .*#%%%%#(/((#%%%%(,     ,/#%%%#(/#%%%#(,\n./#%%%(*    ,#%%%%%%%%(*   .*#%%%#*     .*#%%%#,      *(%%%%%%%#(,.\n./#%%%#*    ,(((##%%%%(*   ,/%%%%/.      .(%%%#/   .*#%%%#(*/(#%%%#/,\n ,#%%%#(.        ,#%%%(*   ,/%%%%/.      .(%%%#/  ,/%%%#/.    .*#%%%(,\n  *#%%%%(*.      ,#%%%(*   .*#%%%#*     ./#%%%#,  ,(%%%#*      .(%%%#*\n   ,(#%%%%%##(((##%%%%(*    .*#%%%%#(((##%%%%(,   .*#%%%##(///(#%%%#/.\n     .*/###%%%%%%%###(/,      .,/##%%%%%##(/,.      .*(##%%%%%%##(*,\n          .........                ......                .......\n{\"time\":\"2024-08-20T12:52:43.821753831+10:00\",\"level\":\"INFO\",\"msg\":\"Serving at 0.0.0.0:3080\"}\n```\n\nTo use, open a new terminal and follow examples in the `examples/` folder\n\nCreate a book:\n\n```shell\ncurl -v --request POST 'http://localhost:3080/api/v1/book' \\\n --header 'Content-Type: application/json' \\\n --data-raw '{\n    \"title\": \"Test title\",\n    \"image_url\": \"https://example.com\",\n    \"published_date\": \"2020-07-31T15:04:05.123499999Z\",\n    \"description\": \n    \"test description\"\n  }'\n```\n\nRetrieve all books:\n\n```shell\ncurl --request GET 'http://localhost:3080/api/v1/book'\n```\n\nTo see all available routes, run\n\n```shell\ngo run cmd/route/main.go\n```\n\n![go run cmd/routes/main.go](assets/routes.png)\n\nTo run both unit and integration tests,\n\n```sh\ngo test ./...\n```\n\nFor end-to-end tests, run the following docker-compose command. It will run a database, a server, and its table migrations. Then it runs end-to-end program (located in `e2e/main.go`) against it \u0026mdash; all in a self-contained container environment. \n\n```sh\ndocker-compose -f e2e/docker-compose.yml up --build\n```\n\nYou will see a bunch of containers being built before the tests are run. The e2e test ends with the following message\n\n```\ngo8_e2e_test | 2023/12/30 03:08:04 api is up\ngo8_e2e_test | 2023/12/30 03:08:04 testEmptyBook passes\ngo8_e2e_test | 2023/12/30 03:08:04 testAddOneBook passes\ngo8_e2e_test | 2023/12/30 03:08:04 testGetBook passes\ngo8_e2e_test | 2023/12/30 03:08:04 testUpdateBook passes\ngo8_e2e_test | 2023/12/30 03:08:04 testDeleteOneBook passes\ngo8_e2e_test | 2023/12/30 03:08:04 all tests have passed.\ngo8_e2e_test exited with code 0\n```\n\nPress `Ctrl+C` to quit the e2e test and stop all e2e containers. \n\nTo remove the containers,\n\n```sh\ndocker-compose -f e2e/docker-compose.yml down\n```\n\n# Table of Contents\n\n- [Introduction](#introduction)\n- [Motivation](#motivation)\n- [Features](#features)\n- [Quick Start](#quick-start)\n- [OpenTelemetry](#opentelemetry)\n- [Tooling](#tooling)\n   * [Tools](#tools)\n      + [Install](#install)\n   * [Tasks](#tasks)\n      + [List Routes](#list-routes)\n      + [Format Code](#format-code)\n      + [Sync Dependencies](#sync-dependencies)\n      + [Compile Check](#compile-check)\n      + [Unit tests](#unit-tests)\n      + [golangci Linter](#golangci-linter)\n      + [Check](#check)\n      + [Hot reload](#hot-reload)\n      + [Generate Swagger Documentation](#generate-swagger-documentation)\n      + [Go generate](#go-generate)\n      + [Test Coverage](#test-coverage)\n      + [Build](#build)\n      + [Clean](#clean)\n- [Structure](#structure)\n   * [Starting Point](#starting-point)\n   * [Configurations](#configurations)\n      - [Environment Variables](#environment-variables)\n      - [.env files](#env-files)\n   * [Database](#database)\n   * [Router](#router)\n   * [Domain](#domain)\n      + [Repository](#repository)\n      + [Use Case](#use-case)\n      + [Handler](#handler)\n      + [Initialize Domain](#initialize-domain)\n   * [Middleware](#middleware)\n      + [Middleware External Dependency](#middleware-external-dependency)\n   * [Dependency Injection](#dependency-injection)\n   * [Libraries](#libraries)\n- [Migration](#migration)\n    * [Using Task](#using-task)\n        + [Create Migration](#create-migration)\n        + [Migrate up](#migrate-up)\n        + [Rollback](#rollback)\n    * [Without Task](#without-task)\n        + [Create Migration](#create-migration-1)\n        + [Migrate Up](#migrate-up)\n        + [Rollback](#rollback-1)\n- [Run](#run)\n    * [Local](#local)\n    * [Docker](#docker)\n        + [docker-compose](#docker-compose)\n- [Build](#build-1)\n    * [With Task](#with-task)\n    * [Without Task](#without-task-1)\n- [Authentication](#authentication)\n    * [How It Works](#how-it-works)\n    * [Expiry](#expiry)\n    * [Logging Out](#logging-out)\n    * [Security](#security-consideration)\n    * [Performance](#performance)\n    * [Integration testing](#integration-testing)\n- [Cache](#cache)\n    * [LRU](#lru)\n    * [Redis](#redis)\n- [Swagger docs](#swagger-docs)\n- [Utility](#utility)\n- [Testing](#testing)\n   * [Unit Testing](#unit-testing)\n      - [Handler](#handler-1)\n      - [Use Case](#use-case-1)\n      - [Repository](#repository-1)\n   * [Integration Testing](#integration-testing-1)\n   * [End-to-End Test](#end-to-end-test)\n- [TODO](#todo)\n- [Acknowledgements](#acknowledgements)\n- [Appendix](#appendix)\n   * [Dev Environment Installation](#dev-environment-installation)\n\n# OpenTelemetry\n\nDuring development, we follow a process and have many tools including functional tests (unit, integration, end-to-end), static analyzer, performance regression tests, and others before we ship to production. In spite of developers' best effort, bugs do occur in production. Typical tools we have during development may not be applicable to a production setting. What we need is some visibility on how our program's internal state and how it behaves in production for example, error logs can be useful to give us this information. An endpoint which is slow will need to be looked at, and it will be great if we can pinpoint exactly where the offending part of a codebase is. \n\nOpenTelemetry is a collection of APIs, SDKs, and tools to instrument, generate, collect, and export telemetry data including metrics, logs, and traces. Using docker-compose, all infrastructure needed to bring up OpenTelemetry integration can be started automatically. This includes configurations and the dashboard.\n\n```sh\ndocker-compose -f docker-compose-infra.yml up -d\n```\n\nOnce everything is up, the dashboard is accessed through Grafana at http://localhost:3300. Initial login credential is admin/admin. Then you will be prompted to set a new password.\n\nBefore moving forward with the dashboard, OpenTelemetry needs to be enabled in the api server. Switch on the feature in environment variable either by exporting it or by editing the value for `OTEL_ENABLE` from `false` to `true` in `.env`.\n\n```sh\nsed -i \"s/^OTEL_ENABLE=.*/OTEL_ENABLE=true/\" \".env\"\n# or\nexport OTEL_ENABLE=true\n```\n\nAs everything is started as containers, finding the services are now done with its container name instead of `localhost`. Using docker-compose provides the convenience of setting the correct host necessary for api server to find the services. Stop the current api server and start it again with the provided docker-compose file.\n\n```sh\nctrl+C\ndocker-compose up -d\n```\n\nCircling back to Grafana, the dashboard named 'Observe' is empty until there are some data being created. Synthetic load can be generated by using tools like [k6](https://grafana.com/docs/k6/latest/get-started/installation/) or [locust](https://docs.locust.io/en/stable/installation.html).\n\n```sh\nk6 run scripts/k6.js\n```\n\n# Tooling\n\nThe above quick start is sufficient to start the API. However, we can take advantage of a tool to make task management easier. While you may run migration with `go run cmd/migrate/main.go`,  it is a lot easier to remember to type `task migrate` instead. Think of it as a simplified `Makefile`.\n\nYou may also choose to run sql scripts directly from `database/migrations` folder instead.\n\nThis project uses [Task](https://github.com/go-task/task) to handle various tasks such as migration, generation of swagger docs, build and run the app. It is essentially a [sh interpreter](https://github.com/mvdan/sh).\n\nInstall task runner binary bash script:\n\n```sh\nsudo ./scripts/install-task.sh\n```\n\nThis installs `task` to `/usr/local/bin/task` so `sudo` is needed.\n\n`Task` tasks are defined inside `Taskfile.yml` file. A list of tasks available can be viewed with:\n\n```sh\ntask -l\n```\nor\n\n```sh\ntask list\n```\n\n## Tools\n\nVarious tooling can be installed automatically by running which includes\n\n * [golang-ci](https://golangci-lint.run)\n    * An opinionated code linter from https://golangci-lint.run/\n * [swag](https://github.com/swaggo/swag)\n    * Generates swagger documentation\n * [goose](https://github.com/pressly/goose)\n    * Migration tool\n * [ent](https://entgo.io/docs/getting-started)\n    * Database ORM tool\n * [mirip](https://github.com/gmhafiz/mirip)\n    * Generate mocks from interface \n * [air](https://github.com/air-verse/air)\n    * Hot reload app \n\n### Install\n\nInstall the tools above with:\n\n```sh\ntask install:tools\n```\n\n## Tasks\n\nVarious tooling are included within the `Task` runner. Configurations are done inside `Taskfile.yml` file.\n\n### List Routes\n\nList all registered routes, typically done by `register.go` files by\n\n```sh\ngo run cmd/route/main.go\n```\n\nor\n\n```sh\ntask routes\n```\n\n### Go generate\n\n```sh\ntask generate\n```\n\nRuns `go generate ./...`. It looks for `//go:generate` tags found in .go files. Useful for recreating mock file for unit tests.\n\n### Generate Swagger Documentation\n\nEnsure your environment variable is `API_RUN_SWAGGER=true`, then re-run the api. Generates swagger files with the following command. \n```sh\ntask swagger\n```\n\nThis reads annotations from controller and model file to create a swagger documentation file. The UI can be accessed from [http://localhost:3080/swagger/](http://localhost:3080/swagger/)\n\n### Format Code\n\n```sh\ntask fmt\n```\n\nRuns `go fmt ./...` to lint Go code\n\n`go fmt` is part of official Go toolchain that formats your code into an opinionated format.\n\n### Compile Check\n\nRuns Go's tooling `go vet ./...`.\n\n```sh\ntask vet\n```\n\nQuickly catches compile error.\n\n### golangci Linter\n\n```sh\ntask lint\n```\n\nRuns [https://golangci-lint.run](https://golangci-lint.run/) linter. Includes [gosec](https://github.com/securego/gosec) via `golangci.yaml`.\n\n### Unit tests\n\nRuns unit tests using `go test ./...`. Other commands are `task test:verbose` for verbose output and `task test:slow` to find top slow-running tests.\n\n```sh\ntask test\n```\n\n### Vulnerability Check\n\nFind software supply chain vulnerabilities using https://vuln.go.dev.\n\n```sh\ntask vuln\n```\n\nThis runs `govulncheck ./...`\n\n### Check\n\n```sh\ntask check\n```\n\nRuns all the above tasks (Format Code until Security Checks)\n\n### Sync Dependencies\n\n```sh\ntask tidy\n```\n\nRuns `go mod tidy` to sync dependencies.\n\n### Hot reload\n\n```sh\ntask dev\n```\n\nRuns `air` which watches for file changes and rebuilds binary. Configure in `.air.toml` file.\n\n### Test Coverage\n\n```sh\ntask test:coverage\n```\n\nRuns unit test coverage with `go test -cover ./...`\n\n### Build\n\n```sh\ntask build\n```\n\nCreate a statically linked executable for linux.\n\n### Clean\n\n```sh\ntask clean\n```\n\nClears all files inside `bin` directory as well as cached unit test files.\n\n\n# Structure\n\nThis project follows a layered architecture mainly consisting of three layers:\n\n1. Handler\n2. Use Case\n3. Repository\n\n![layered-architecture](assets/layered-architecture.png)\n\nThe handler is responsible to receiving requests, validating them hand over to business logic.\nValues returned from use case layer is then formatted and to be returned to the client.\n\nBusiness logic (use case) is the meat of operations, and it calls a repository if necessary.\n\nDatabase calls lives in this repository layer where data is retrieved from a store.\n\nAll of these layers are encapsulated in a domain, and an API can contain many domain.\n\nEach layer communicates through an interface which means the layer depends on\nabstraction instead of concrete implementation. This achieves loose-coupling and\nmakes unit testing easier.\n\n## Starting Point\n\nStarting point of project is at `cmd/go8/main.go`\n\n![main](assets/main.png)\n\n\nThe `Server` struct in `internal/server/server.go` is where all important dependencies are\nregistered and to give a quick glance on what your server needs.\n\n![server](assets/server.png)\n\n`s.Init()` in `internal/server/server.go` simply initializes server configuration, database, input validator, router, global middleware, domains, and swagger. Any new dependency added to the `Server` struct can be initialized here too.\n\n![init](assets/init.png)\n\n\n## Configurations\n\nThe api can be configured by taking values from environment variables or an `.env` file.\n\nThese variables are read into specific `Configs` struct initialized in `configs/configs.go`. Each of the embedded struct are defined in its own file of the same package where its fields are read from either environment variable or `.env` file.\n\n![configs](assets/configs.png)\n\nThis approach allows code completion when accessing your configurations.\n\n![config code completion](assets/config-code-completion.png)\n\n#### Environment Variables\n\nSimply set the environment variables by exporting them like so:\n\n```sh\nexport DB_DRIVER=postgres\nexport DB_HOST=localhost\nexport DB_PORT=5432\netc\n```\n\n#### .env files\n\nAlternatively, the `.env` file can be used. Just make sure this file is not committed in source control. Make a copy from example env file:\n\n```sh\ncp env.example .env\n```\n\n#### Add new Config Struct\n\nTo add a new type of configuration, for example for Elasticsearch\n\n1. Create a new go file in `./configs`\n\n```shell\ntouch configs/elasticsearch.go\n```\n\n2. Create a new struct for your type\n\n```go\ntype Elasticsearch struct {\n  Address  string\n  User     string\n  Password string\n}\n```\n\nVarious validation and default values among others can also be configured into this struct. Visit https://github.com/kelseyhightower/envconfig for more details.\n\n3. Add a constructor for it\n\n```go\nfunc ElasticSearch() Elasticsearch {\n   var elasticsearch Elasticsearch\n   envconfig.MustProcess(\"ELASTICSEARCH\", \u0026elasticsearch)\n\n   return elasticsearch\n}\n``` \n\n4. Register to main Config struct\n \n```go\ntype Config struct {\n\tApi\n\tCors\n\t...\n\tElasticsearch // add the constructor\n}\n\nfunc New() *Config {\n\t...\n\treturn \u0026Config{\n\t\t...\n\t\tElasticsearch: ElasticSearch(), // Add here\n    }\n}\n```\n\n5. Add to `.env` of the new environment variables\n\nA namespace is defined as `ELASTICSEARCH`.\n\n```shell\nELASTICSEARCH_ADDRESS=http://localhost:9200\nELASTICSEARCH_USER=user\nELASTICSEARCH_PASS=password\n```\n\n#### Others\n\nLimiting the number of connection pool avoids ['time-slicing' of the CPU](https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing). Use the following formula to determine a suitable number\n\n    number of connections = ((core_count * 2) + effective_spindle_count)    \n\n## Database\n\nMigrations files are stored in `database/migrations` folder. [goose](https://github.com/pressly/goose) library is used to perform migration using `task` commands.\n\n## Router\n\nRouter multiplexer or mux is created for use by `Domain`. While [chi](https://github.com/go-chi/chi) library is being used here, you can swap out the router to an alternative one when assigning `s.router` field. However, you will need to adjust how you register your handlers in each domain.\n\n## Domain\n\nLet us look at how this project attempts at layered architecture. A domain consists of:\n\n1. Handler (Controllers)\n2. Use case (Business Logic)\n3. Repository (Database)\n\nLet us start by looking at how `repository` is implemented.\n\n### Repository\n\nStarting with `Database`. This is where all database operations are handled. Inside the `internal/domain/health` folder:\n\n![book-domain](assets/domain-health.png)\n\nInterfaces for both use case and repository are on its own file under the `health` package while its implementation are in `usecase` and `repository` package respectively.\n\nThe `health` repository has a single method\n\n`internal/domain/health/repository.go`\n\n```go\n type Repository interface {\n     Readiness() error\n }\n````    \n\nAnd it is implemented in a package called `postgres` in `internal/domain/health/repository/postgres/postgres.go`\n\n```go\nfunc (r *repository) Readiness() error {\n  return r.db.Ping()\n}\n```\n\n### Use Case\n\nThis is where all business logic lives. By having repository layer underneath in a separate layer, those functions are reusable in other use case layers.\n\n### Handler\n\nThis layer is responsible in handling request from outside world and into the `use case` layer. It does the following:\n\n1. Parse request into a 'request' struct\n2. Sanitize and validates said struct\n3. Pass into `use case` layer\n4. Process results from coming from `use case` layer and decide how the payload is going to be formatted to the outside world.\n\nRoute API are defined in `RegisterHTTPEndPoints` in their respective `register.go` file.\n\n####  1. Parse request into 'request' struct\n\nA struct must be defined with fields along with its type. For example in `book.CreateRequest`, there are four possible inputs and all of them are required, denoted by struct tag. `ImageURL` needs to be a URL, which is a handy utility provided by the [validation library](https://github.com/go-playground/validator), among others.\n\n```go\ntype CreateRequest struct {\n\tTitle         string `json:\"title\" validate:\"required\"`\n\tPublishedDate string `json:\"published_date\" validate:\"required\"`\n\tImageURL      string `json:\"image_url\" validate:\"url\"`\n\tDescription   string `json:\"description\" validate:\"required\"`\n}\n```\n`validate` struct tag is parsed by the library to know which one is required. `json` struct tag helps with customising fields' letter case.\n\nTo parse the request, simply use the `json` package.\n\n```go\nvar bookRequest book.CreateRequest\nerr := json.NewDecoder(r.Body).Decode(\u0026bookRequest)\n```\n\nTake care to supply the address of `bookRequest` variable by prepending ampersand `\u0026` to it.\n\nIf there are no errors, the `bookRequest` can be passed into the usecase or repository layer.\n\nUsing the `json` package to parse is although simple, works well. It doesn't cover all cases in which you may want to refer to this [article](https://www.alexedwards.net/blog/how-to-properly-parse-a-json-request-body)\n\n### Initialize Domain\n\nFinally, a domain is initialized by wiring up all dependencies in server/initDomains.go. Here, any dependencies can be injected such as a custom logger.\n\n```go\nfunc (s *Server) initBook() {\n   newBookRepo := bookRepo.New(s.GetDB())\n   newBookUseCase := bookUseCase.New(newBookRepo)\n    s.Domain.Book = bookHandler.RegisterHTTPEndPoints(s.router, newBookUseCase)\n}\n```\n\n## Middleware\n\nA middleware is just a handler that returns a handler as can be seen in the `internal/middleware/cors.go`\n\n```go\nfunc Cors(next http.Handler) http.Handler {\n    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n    \n        // do something before going into Handler\n        \n        next.ServerHTTP(w, r)\n        \n        // do something after handler has been served\n    }\n}\n```\n\nThen you may choose to have this middleware to affect all routes by registering it in`setGlobalMiddleware()` or only a specific domain at `RegisterHTTPEndPoints()` function in its `register.go` file.\n\n\n### Middleware External Dependency\n\nSometimes you need to add an external dependency to the middleware which is often the case for\nauthorization be that a config or a database. To do that, we wrap our middleware with a\n`func(http.Handler) http.Handler`. Any dependencies can now be passed in into `Auth()`.\n\n```go\nfunc Auth(cfg configs.Configs) func(http.Handler) http.Handler {\n    return func(next http.Handler) http.Handler {\n        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n            claims, err := getClaims(r, cfg.Jwt.SecretKey)\n            if err != nil {\n                w.WriteHeader(http.StatusUnauthorized)\n                return\n            }\n    \n            next.ServeHTTP(w, r)\n        })\n    }\n}\n```\n\n## Dependency Injection\n\nDependency injection in Go is simple. We can simply pass in whatever we need\ninto the function or method signature. There is no need to make a dependency injection\ncontainer like other object-oriented programming language.\n\nHow does dependency injection happens? It starts with `InitDomains()` method. We\ninitialize the dependency we want early in the start-up of the program and then\npass it down the layers.\n\n```go\nnewBookRepo := bookRepo.New(s.DB())\nnewBookUseCase := bookUseCase.New(newBookRepo)\ns.Domain.Book = bookHandler.RegisterHTTPEndPoints(\n\ts.router,\n\ts.validator,\n\tnewBookUseCase,\n)\n```\n\nThe repository gets access to a pointer to `sql.DB` from the `s.DB()`\ninitialisation to perform database operations. This layer also knows nothing of\nlayers above it. `NewBookUseCase` depends on that repository and finally the\nhandler depends on the use case.\n\n## Libraries\n\nInitialization of external libraries are located in `third_party/`\n\nSince `sqlx` is a third party library, it is initialized in `/third_party/database/sqlx.go`\n\n# Migration\n\nMigration is a good step towards having a versioned database and makes publishing to a production server a safe process.\n\nAll migration files are stored in `database/migrations` folder.\n\n## Using Task\n\n### Create Migration\n\nUsing `Task`, creating a migration file is done by the following command. Name the file after `NAME=`.\n\n```sh\ntask migrate:create NAME=create_a_tablename\n```\n\nWrite your schema in pure sql in the 'up' section and any reversal in the 'down' section of the file.\n \n### Migrate up\n\nAfter you are satisfied with your `.sql` files, run the following command to migrate your database.\n\n```sh\ntask migrate\n```\n\nTo migrate one step\n\n```sh\ntask migrate:step\n```\n      \n### Rollback\n    \nTo roll back migration by one step\n\n```sh\ntask migrate:rollback\n```\n\nFurther `goose` commands are available in its [page](https://github.com/pressly/goose)\n\n\n## Without Task\n\n### Create Migration\n\nOnce `goose` tool is [installed](https://github.com/pressly/goose), create a migration with\n\n```sh\nmigrate create -ext sql -dir database/migrations -format unix \"{{.NAME}}\"\n```\n\n### Migrate Up\n\nYou will need to create a data source name string beforehand. e.g.:\n\n    postgres://postgres_user:$password@$localhost:5432/db?sslmode=false\n\nNote: You can save the above string into an environment variable for reuse e.g.\n\n```sh\nexport DSN=postgres://postgres_user:$password@$localhost:5432/db?sslmode=false\n```\n\nThen migrate with the following command, specifying the path to migration files, data source name and action.\n\n```sh\nmigrate -path database/migrations -database $DSN up\n```\n\nTo migrate 1 step,\n\n```sh\nmigrate -path database/migrations -database $DSN up-by-one\n```\n\n### Rollback\n\nRollback migration by using `down` action and the number of steps\n\n```sh\nmigrate -path database/migrations -database $DSN down\n```\n\n# Run\n\n## Local\n\nConventionally, all apps are placed inside the `cmd` folder.\n\nIf you have `Task` installed, the server can be run with:\n\n\n```sh\ntask run\n```\n\nor without `Task`, just like in quick start section:\n\n```sh\ngo run cmd/go8/main.go\n```\n\n## Docker\n\nYou can build a docker image with the app with its config files. Docker needs to be installed beforehand.\n\n```sh\ntask docker:build\n```\n\nThis task also makes a copy of `.env`. Since Docker doesn't copy hidden file, we make a copy of it on our `src` stage before transferring it to our final `scratch` stage. It also inserts formats git tag and git hash as the API version which runs at compile time.\n\nNote that this is a multistage Dockerfile. Since we statically compile this API, we can use a minimal images like a `distroless` that includes both timezones and CA certificates.\n\nRun the following command to build a container from this image. `--net=host` tells the container to use local's network so that it can access host database.\n\n```sh\ndocker-compose up -d postgres # If you haven't run this from quick start\n```\n\nOr using Task\n```sh\ntask docker:run\n```\n\n### docker-compose\n\nIf you prefer to use docker-compose instead, both server and the database can be run with:\n\n```sh\ntask docker-compose:start\n```\n\n# Build\n\n## With Task\n\nIf you have task installed, simply run\n\n```sh\ntask build\n```\n\nIt does a task check prior to the build and puts both the binary and `.env` files into `./bin` folder\n\n## Without Task\n\n```sh\ngo mod download\nCGO_ENABLED=0 GOOS=linux\ngo build -ldflags=\"-X main.Version=$(git describe --abbrev=0 --tags)-$(git rev-list -1 HEAD) -s\" -o ./server ./cmd/go8/main.go;\n```\n\n# Authentication\n\nAuthentication is proving who or identity of a person as opposed to Authorization where it tells what this person can do. The most popular way is by using JWT which is a stateless solution by using signed token generated from the server and then stored client-side. This token contains an algorithm, date and time it expires, and may store arbitrary data. This token is attached to each subsequent requests by client-side where the server verifies its validity by checking if the token has not been tampered and then see if it is not past expiry date.\n\nThis repository on the other hand demonstrates an age-old method of authentication \u0026mdash; sessions. It generates a session containing random characters and set in a cookie. This session is also stored in a database, and this makes it a stateful solution. Arbitrary data can be saved inside the session by saving it in the database but client-side only receives the random token.\n\nAdvantage of this approach is invalidating or logging out a user can be done by simply deleting the associated session record in the database. With `HttpOnly` flag set to true, this ensures the cookie can only be read by the browser, not client-side javascript. Thus, this prevents cross-site scripting (XSS) attack. `SameSite=lax` and setting the `Domain` minimises the risk of cross-site request forgery (CSRF). However, this approach incurs additional cost of database querying in every request.\n\nWhy not JWT in cookie? This prevents XSS with correct cookie flags. However, session invalidation is not possible without storing the sessions in a persistent store.\n\n## How It Works\n\nSession management is done using [alexedwards/scs](https://github.com/alexedwards/scs) library. All routes go through a custom middleware based on `LoadAndSave` middleware. On top of storing data in a database and setting the cookie in a response header before going into any handler, it also saves user ID as a foreign key to `users` table.\n\nAt first time login, no cookie is present. Upon successful login, we call `RenewToken()` method that creates a new random string token, and after it exits the handler, it re-enters the `LoadAndSave` middleware to save the token into the database. If any action to add data were done, it will be saved in the database, not in the cookie. The response will contain the cookie in the header.\n\n## Usage\n\nIt is recommended to both migrate (if you have not) and seed `users` table with a super admin account. Run with\n\n```sh\ngo run cmd/migrate/main.go\ngo run cmd/seed/main.go\n```\n\nThis generates a super admin user with a randomly generated password. This password is automatically saved in `.env` file.\n\nLet us register a new user:\n\n```sh\ncurl -vX POST  -H 'content-type: application/json' 'http://localhost:3080/api/v1/register' -d '{\n  \"first_name\": \"Hafiz\",\n  \"last_name\": \"Shafruddin\",\n  \"email\": \"email@example.com\",\n  \"password\": \"highEntropyPassword\"\n}'\n```\n\nYou should get a 201 Created HTTP status response. Now you can login\n\n```sh\ncurl -vX POST -H 'content-type: application/json' \\ \n'http://localhost:3080/api/v1/login' \\\n-d '{\n  \"email\": \"email@example.com\",\n  \"password\": \"highEntropyPassword\"\n}\n'\n```\n\nYou will get a 200 OK HTTP status response along with a cookie in the header. By default, our token is stored from `session` key in the `Set-Cookie` header. In this example, it is `gedFYqAUXejpgmBhnCkKLip7dOjecbBC1HzSHCX7KGI`.\n\n```\n\u003c HTTP/1.1 200 OK\n\u003c Cache-Control: no-cache=\"Set-Cookie\"\n\u003c Content-Type: application/json\n\u003c Set-Cookie: session=gedFYqAUXejpgmBhnCkKLip7dOjecbBC1HzSHCX7KGI; Path=/; Expires=Wed, 03 May 2023 13:45:59 GMT; Max-Age=86365; SameSite=Lax\n\u003c Vary: Origin\n\u003c Vary: Cookie\n\u003c Date: Tue, 02 May 2023 13:46:34 GMT\n\u003c Content-Length: 147\n```\n\nAt this point, you will also see that a new record is stored into `sessions` table.\n\n```sql\nSELECT * FROM sessions;\n```\n\nYou can see the token is stored along with our user ID. `data` column stores arbitrary payload in the session as supported by [alexedwards/scs](https://github.com/alexedwards/scs) library.\n\n| token                                       | user\\_id | data                        | expiry                            |\n|:--------------------------------------------|:---------|:----------------------------|:----------------------------------|\n| gedFYqAUXejpgmBhnCkKLip7dOjecbBC1HzSHCX7KGI | 2        | 0x26FF810301...truncated... | 2023-05-11 13:17:38.062590 +00:00 |\n\n\n\nFor making subsequent requests on protected routes, we need to attach the cookie in the header. If you are using browser's native `fetch` function, simply use `credentials: include` or if using axios, switch on the `withCredentials` setting:\n\nfetch\n```js\nlogin() {\n  $fetch(`${this.baseURL}/login`, {\n    headers: { 'Access-Control-Allow-Origin': true },\n    credentials: 'include',\n    method: 'POST',\n    body: {\n      email: 'email@example.com',\n      password: 'password',\n    },\n  })\n```\n\naxios\n```\naxios.defaults.withCredentials = true\n```\n\nFor now, let us make a curl request.\n\n```sh\ncurl -v 'http://localhost:3080/api/v1/restricted/me' --cookie \"session=gedFYqAUXejpgmBhnCkKLip7dOjecbBC1HzSHCX7KGI\"\n```\n\nAnd you will get a 200 HTTP response along with user ID payload `{\"user_id\":2}`.\n\n## Expiry\n\nBy default, the session stays for 24 hours but this can be changed by editing `SESSION_DURATION` key. A background job regularly checks the table for expired sessions and remove them from the table.\n\n## Logging Out\n\nA logged in user can simply call `/v1/logout` to log out. The cookie need to be present for the api to know which token to delete.\n\nA super admin can log out any user provided that the super admin's `users` ID is equal to `1`. Obviously a proper authorization is needed in real-world application.\n\n## Security Consideration\n\nThese are the important cookie flags that needs to be reviewed.\n\n1. Secure: Ensures cookie are only used in secure TLS (https)\n2. HttpOnly: Ensure no client-side javascript can read the values in cookie, only browser can. This prevents XSS\n3. Domain: Where cookie is expected to work\n4. SameSite: Using  either `Lax` or `Strict` ensures cookies only work on your domain\n\nThanks to `HttpOnly` flag, no client-side can access this token thus preventing XSS attack. Both setting a domain\nand `SameSite` flag value set to at least `Lax` helps with preventing CSRF attack \u0026mdash; although it does not prevent CSRF entirely. For example `SameSite` attribute was not supported in [old browsers](https://caniuse.com/?search=samesite). For this, we can request a new token called CSRF Token. For every modifying requests, we attach this new token alongside and check its existence in the database.\n\nAlso set allowed domains if possible in either `.env` or environment variable.\n\n```sh\nexport SESSION_DOMAIN=https://mySite.com\n```\n\n## Performance\n\nSince database is called for every protected endpoints, both throughput and latency can be an issue. However, token column is indexed which makes record retrieval near instant \u0026mdash; typically sub-millisecond.\n\nThe authentication [library](https://github.com/alexedwards/scs) also includes a redis implementation but a custom implementation is needed to also store user ID against a session token when committing a token to Redis.\n\n\n## Integration Testing\n\nThe `authentication` domain is tested using integration tests (in `integration_test.go`) which covers the handler and the database. We use a table-driven driven tests where a set of scenarios are stored as a slice (array) containing what we supply in the parameters, what result we expect from those parameters, and we gave a name to identify each test. The slice is then looped and calls appropriate functions. The tests are tested against real database by spinning them up as docker containers. With testing, implementations can be changed and improved without affecting correctness and desired outcome.\n\nNote: This repository also details regarding unit tests which are explained in the [Testing](#testing) section.\n\nTo run only integration test,\n\n```sh\ngo test -run Integration ./...\n```\n\n# Cache\n\nThe three most significant bottlenecks are\n\n1. Input output (I/O) like disk access including database.\n2. Network calls - like calling another API.\n3. Serialization - like serializing or deserializing JSON\n\nWe demonstrate how caching results can speed up API response:\n\n## LRU\n\nTo make this work, we introduce another layer that sits between use case and database layer.\n\n`internal/author/repository/cache/lru.go` shows an example of using an LRU cache to tackle the biggest bottleneck. Once we get a result for the first time, we store it by using the requesting URL as its key. Subsequent requests of the same URL will return the result from the cache instead of from the database.\n\nThe request url only exists in the handler layer by accessing it from `*http.Request`.\nTo pass the request url to our cache layer, we can either pass it down the layers as\na method parameter, or we can use `context` to save this url. In general, `context`\nis not the way to store variables but since this url is scoped to this one request,\nit is okay.\n\n```go\nctx := context.WithValue(r.Context(), author.CacheURL, r.URL.String())\n```\n\nThen in the cache layer, we retrieve it using the same key (`author.CacheURL`)\nand we must assert the type to `string`.\n\n```go\nurl := ctx.Value(author.CacheURL).(string)\n```\n\nThe code above can panic if the key does not exist. To be safer, we can check if\na value is retrieved successfully.\n\n```go\nurl, ok := ctx.Value(author.CacheURL).(string)\nif !ok {\n\t// handle if not ok, or call repository layer.\n}\n```\n\nUsing the `url`, we try and retrieve a value from the cache,\n```go\nval, ok := c.lru.Get(url)\n```\n\nIf it doesn't exist, retrieve from database layer and add it to our cache for future use.\n```go\nc.lru.Add(url, res)\n```\n\nAvoiding I/O bottleneck results in an amazing speed, **11x** more requests/second (328 bytes response size) compared to an already blazing fast endpoint as shown by `wrk` benchmark:\n\nCPU: AMD 3600 3.6Ghz\nStorage: SSD\n\n```shell\nwrk -t2 -d60 -c200  'http://localhost:3080/api/v1/author?page=1\u0026size=3'\nRunning 1m test @ http://localhost:3080/api/v1/author?page=1\u0026size=3\n  2 threads and 200 connections\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\n    Latency     4.23ms    5.07ms  71.75ms   83.36%\n    Req/Sec    40.64k     3.55k   52.91k    68.45%\n  4847965 requests in 1.00m, 1.48GB read\nRequests/sec:  80775.66\nTransfer/sec:     25.27MB\n```\n\nCompared to calling database layer:\n```shell\nwrk -t2 -d60 -c200  'http://localhost:3080/api/v1/author?page=1\u0026size=3'\nRunning 1m test @ http://localhost:3080/api/v1/author?page=1\u0026size=3\n  2 threads and 200 connections\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\n    Latency    70.66ms  116.57ms   1.24s    88.09%\n    Req/Sec     3.66k   276.15     4.53k    70.50%\n  437285 requests in 1.00m, 136.79MB read\nRequests/sec:   7280.82\nTransfer/sec:      2.28MB\n```\n\nSince a cache stays in the store if it is frequently accessed, invalidating the cache must be done if there are any changes to the stored value in the event of update and deletion. Thus, we need to delete the cache that starts with the base URL of this domain endpoint.\n\nFor example:\n```go\nfunc (c *AuthorLRU) Update(ctx context.Context, toAuthor *models.Author) (*models.Author, error) {\n\tc.invalidate(ctx)\n\n\treturn c.service.Update(ctx, toAuthor)\n}\n\nfunc (c *AuthorLRU) invalidate(ctx context.Context) {\n\turl := ctx.Value(author.CacheURL)\n\tsplit := strings.Split(url.(string), \"/\")\n\tbaseURL := strings.Join(split[:4], \"/\")\n\n\tkeys := c.lru.Keys()\n\tfor _, key := range keys {\n\t\tif strings.HasPrefix(key.(string), baseURL) {\n\t\t\tc.lru.Remove(key)\n\t\t}\n\t}\n}\n```\n## Redis\n\nBy using Redis as a cache, you can potentially take advantage of a cluster architecture for more RAM instead of relying on the RAM on current server your API is hosted. Also, the cache won't be cleared like in-memory `LRU` when a new API is deployed.\n\nSimilar to LRU implementation above, this Redis layer sits in between use case and database layer.\n\nThis Redis library requires payload in a binary format. You may choose the builtin `encoding/json` package or `msgpack` for smaller payload and **7x** higher speed than without a cache. Using `msgpack` over `json` tackles serialization bottleneck.\n\n```go\n// marshal \ncacheEntry, err := msgpack.Marshal(res)\n// unmarshal\nerr = msgpack.Unmarshal([]byte(val), \u0026res)\n```\n\n```shell\nwrk -t2 -d60 -c200  'http://localhost:3080/api/v1/author?page=1\u0026size=3'\nRunning 1m test @ http://localhost:3080/api/v1/author?page=1\u0026size=3\n  2 threads and 200 connections\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\n    Latency     4.05ms    2.56ms  37.48ms   73.63%\n    Req/Sec    25.48k     1.45k   30.73k    71.29%\n  3039522 requests in 1.00m, 0.93GB read\nRequests/sec:  50638.73\nTransfer/sec:     15.84MB\n```\n\n\n# Swagger docs\n\nSwagger UI allows you to play with the API from a browser\n\n![swagger UI](assets/swagger.png)\n     \nEdit `cmd/go8/go8.go` `main()` function host and BasePath  \n\n    // @host localhost:3080\n    // @BasePath /api/v1\n\n   \nGenerate with\n\n    task swagger # runs: swag init \n    \nAccess at\n\n    http://localhost:3080\n\nThe command `swag init` scans the whole directory and looks for [swagger's declarative comments](https://github.com/swaggo/swag#declarative-comments-format) format.\n\nCustom theme is obtained from [https://github.com/ostranme/swagger-ui-themes](https://github.com/ostranme/swagger-ui-themes)\n\n# Utility\n\nCommon tasks like retrieving query parameters or `filters` are done inside `utility` folder. It serves as one place abstract functionalities used across packages.\n\nNote that further packages are in the subdirectories of `utilty`. No files\nar under `utilty` package because it is [unclear from the name](https://go.dev/doc/effective_go#package-names) on what it does. \n\n# Testing\n\nA testable code is a sign that you have a good code structure. However, it can be hard to write not only of a function, but also how different functions work together. \nGoing on a tangent, following [SOLID principle](https://en.wikipedia.org/wiki/SOLID)  is a good way to design our code and make it testable. But this repository isn't nearly complex enough to show good examples of each principle. In any case, we shall start with unit tests.\n\n## Unit Testing\n\nUnit testing can be run with\n\n```sh\ntask test:unit\n```\n    \nWhich runs `go test -v -short ./...`\n\nA quick note, in Go, a unit test file is handled by appending `_test` to a file's name. For example, to test `/internal/domain/book/handler/http/handler.go`, we add unit test file in the same directory by creating `/internal/domain/book/handler/http/handler_test.go`. No more hunting for test file nested deep in a `tests` directory like in other language!\n\nTests that run fast means you run them [more often](https://medium.com/pragmatic-programmers/unit-tests-are-first-fast-isolated-repeatable-self-verifying-and-timely), which leads to writing more tests. You can find top ten slow tests by running the following [command](https://leighmcculloch.com/posts/go-find-slow-tests/)\n\n_Only for Unix-based operating system_\n\nShell command:\n```sh\n# Cleans test cache\ngo clean -testcache\n\n# Test output is json format for easy parsing by jq tool.\ngo test -v -json ./... | jq -r 'select(.Action == \"pass\" and .Test != null) | .Test + \",\" + (.Elapsed | tostring)'  | sort -r -k2 -n -t, | head\n```\n\nOr using `Task`:\n```sh\ntask test:slow\n```\n\nTo perform a unit test we take advantage of go's interface. The layers between\nhandler, use case and database are loosely-coupled. This is achieved by accepting\nan interface and return a struct. This way, you can swap that struct implementation\nwith something else, say, a mock. So when you run the unit test, it will run\nthat mock implementation instead of your concrete implementation. This is all\nthanks to Go's interface which is implicit.\n\nThis repository shows table-driven unit testing strategy  in all three layers.\nBoth handler and usecase layers swaps the implementation of underneath layer \nwith mocks while in repository layer, we use _real_ database in docker to test\nagainst, using `dockertest` library.\n\n### Handler\n\nWe explore on how to perform unit testing on creating an Author. There are several things that need to happen, namely:\n\n1. Create a request.\n2. Validate request.\n3. Call business logic layer underneath it and handle various error that may come up.\n    - We are not going to actually call our business logic layer. We use mocks instead.\n4. Perform data transformation for user consumption.\n5. Compare errors and values.\n\nIn general, all unit tests will have `args` and `want` struct. `args` struct is\nwhat we need to supply to the unit test while `want` struct is where we define\nwhat we expect the result is going to be.\n\nFirstly, we create `handler_test.go` file in the same directory. Create a unit\ntest function called `TestHandler_Create()`.\n\n```go\nfunc TestHandler_Create(t *testing.T) {\n\t\n}\n```\n\nIn there, we add `CreateRequest` to `args` struct.\n\n```go\ntype args struct {\n    *author.CreateRequest\n}\n```\n\nIn `want` struct, we expect the usecase to return three things, the author an \nerror, and http status code. We add an HTTP status response code because our \nhandler can return different code depending on the result.\n\nWhat is strange here is we included a `usecase` struct. This is the struct that\ncontains values that our mock will return.\n\n```go\ntype want struct {\n\tusecase struct {\n        *author.Schema\n        error\n    }\n    response *author.GetResponse\n    status int\n    error\n}\n```\n\nThe final struct embeds both structs, and we give a name to it. Naming the test\nmakes it easier to run individual tests.\n\n```go\ntype test struct {\n    name string\n    args\n    want\n}\n```\n\nNow that we have all necessary structs, we can begin with our table-driven tests.\nIt is just a matter of filling `test` struct with our values.\n\n```go\ntests := []test{\n\t{\n        name: \"simple\",\n        args: args {\n            CreateRequest: \u0026author.CreateRequest{\n                FirstName:  \"First\",\t\t\t\n                MiddleName: \"Middle\",\t\t\t\n                LastName:   \"Last\",\n\t            Books:      nil,\n            }   \t\t\n        },\n        want: want{\n\t\t\t\tusecase: struct {\n\t\t\t\t\t*author.Schema\n\t\t\t\t\terror\n\t\t\t\t}{\n\t\t\t\t\t\u0026author.Schema{\n\t\t\t\t\t\tID:         1,\n\t\t\t\t\t\tFirstName:  \"First\",\n\t\t\t\t\t\tMiddleName: \"Middle\",\n\t\t\t\t\t\tLastName:   \"Last\",\n\t\t\t\t\t\tCreatedAt:  time.Now(),\n\t\t\t\t\t\tUpdatedAt:  time.Now(),\n\t\t\t\t\t\tDeletedAt:  nil,\n\t\t\t\t\t\tBooks:      make([]*book.Schema, 0),\n\t\t\t\t\t},\n\t\t\t\t\tnil,\n\t\t\t\t},\n\t\t\t\tresponse: \u0026author.GetResponse{\n\t\t\t\t\tID:         1,\n\t\t\t\t\tFirstName:  \"First\",\n\t\t\t\t\tMiddleName: \"Middle\",\n\t\t\t\t\tLastName:   \"Last\",\n\t\t\t\t\tBooks:      make([]*book.Schema, 0),\n\t\t\t\t},\n\t\t\t\terr:    nil,\n\t\t\t\tstatus: http.StatusCreated,\n\t\t\t},\n}\n```\n\nAs you can see, the _author_ record gets replicated three times. But there is a\nsubtle difference between the input and output. In the `args` struct, we only\nsupply three things, the names, but left the books as empty. In the `response`\nstruct, it has the names, but also a zero-length array of books! This is an\nintended behaviour - I purposely wanted to return an array no matter what.\n\nSo within this `tests` slice, we can add as many unit tests as we want. We could\ntry giving invalid input, input that fails validation, etc.  \n\nTo run the tests, we loop over this slice of tests:\n\n```go\nfor _, test := range tests {\n    t.Run(test.name, func(t *testing.T) {\n        \n    }\n}\n```\n\nWe use `httptest` package to call our tests by creating a `writer`(request) to \ncall the handler, and a `recorder` to receive response from the handler.\n\n```go\nrr := httptest.NewRequest(http.MethodPost, \"/api/v1/author\", \u003cbody\u003e)\nww := httptest.NewRecorder()\n```\n\nThe request points to the URL of the endpoint, and we make a `POST` request to it.\nSince we are sending a JSON payload, we send it in the third argument. It accepts\nan `io.Reader` so we need to encode our JSON payload into `bytes.Buffer`:\n\n```go\nvar buf bytes.Buffer\nerr = json.NewEncoder(\u0026buf).Encode(test.args.CreateRequest)\n\nrr := httptest.NewRequest(http.MethodPost, \"/api/v1/author\", \u0026buf)\n```\n\nThis is a good place to assert that no error has happened.\n\n```go\nerr = json.NewEncoder(\u0026buf).Encode(test.args.CreateRequest)\nassert.Nil(t, err)\n```\n\nTo call our handler, we need to instantiate it. It is created from `RegisterHTTPEndPoints()`.\n\n```go\nh := RegisterHTTPEndPoints(router, val, uc)\n```\n\nThis function requires three dependencies. The `router` and `validator` are easy:\n\n```go\nrouter := chi.NewRouter()\nval := validator.New()\n```\n\nThe final dependency requires a bit of work. The handler depends on the usecase\ninterface, and it in turn calls the appropriate concrete implementation. For our\nunit test, we swap out the implementation with a mock. And this mock returns\nvalue from our `want.usecase` struct. Now our unit test can work in isolation, \nand do not depend on any layer underneath it!\n\nCreate a new file called `usecase_mock.go`. Declare a new mock struct and within it,\ncontains a field that matches our usecase signature by looking at the usecase \ninterface.\n\n`usecase.go`\n```go\ntype UseCase interface {\n    Create(ctx context.Context, a *author.CreateRequest) (*author.Schema, error)\n}\n```\n\n`usecase_mock.go`\n```go\ntype AuthorUseCaseMock struct {\n    CreateFunc func(ctx context.Context, a *author.CreateRequest) (*author.Schema, error)\n}\n```\nNotice that we append the `Create()` method with `Func` field. Now that we have\nthe struct defined, we add a concrete implementation from it.\n\n`usecase_mock.go`\n```go\nfunc (m *AuthorMock) Create(ctx context.Context, a *author.CreateRequest) (*author.Schema, error) {\n\treturn a.CreateFunc(ctx, req)\n}\n```\n\nNow that we have a usecase mock, we can now give the `uc` variable for `RegisterHTTPEndPoints()` function. Using `AuthorUseCaseMock` struct from `mock` package, we initialize `CreateFunc` field from it. Then, it is just a matter of returning the values to what we have defined in our `want` struct.\n\n`handler_test.go`\n```go\nuc := \u0026mock.AuthorUseCaseMock{\n    CreateFunc: func(ctx context.Context, a *author.CreateRequest) (*author.Schema, error) {\n        return test.want.usecase.Schema, test.want.usecase.error\n    },\n}\n```\n\nWe finally have all of our dependencies initialized. Now we can call `Create()`\nmethod. We pass in the writer(`ww`) and a recorder `rr` into it - which matches our\nhandler signature (`Create(w http.ResponseWriter, r *http.Request)`)\n\n```go\nh := RegisterHTTPEndPoints(router, val, uc)\nh.Create(ww, rr)\n```\n\nResponse is recorded into `ww` variable. To receive the response, we decode from\n`ww.Body` into `author.Schema` struct:\n\n```go\nvar got author.GetResponse\nif err = json.NewDecoder(ww.Body).Decode(\u0026got); err != nil {\n    t.Fatal(err)\n}\n```\n\nFinally, we can do some assertions to check if the returned response matches with\nwhat we expect.\n\n```go\nassert.Equal(t, ww.Code, test.status)\nassert.Equal(t, \u0026got, test.want.response)\n```\n\nWhile `go test ./...` runs all tests, we can choose to run only this specific test. We `cd` into the directory and use `-run` to specify the \u003cfunction name/test name\u003e. `-run` can also accept regex\n\n```shell\ncd internal/domain/author/handler\ngo test -run=\"TestHandler_Create/simple\"\n\nPASS\nok      github.com/gmhafiz/go8/internal/domain/author/handler   0.010s\n```\n\nThere are a lot of things going on in this unit test. It is very verbose, but it\nis clear on what happens here. A table-test allows us to quickly construct arguments,\nwants and what we expect. Constructing a mock file can be tedious, so a tool like\n[mirip](https://github.com/gmhafiz/mirip) can be used to generate a mock from your\ninterface.\n\n### Use Case\n\nThe idea is the same as unit testing a handler. We have a set of arguments, what\nis expected from it, and a slice of `test` struct that we iterate to test against.\n\nThis time, we do not have to worry about write and recorder. We only need to \ninstantiate usecase along with its dependencies. To make this simple, we will\nonly mock database(repository struct).\n\nThe `Create()` method expects a `context` and `*author.CreateRequest` and returns\n`*author.Schema` and an `error`.\n```go\ntype args struct {\n    *author.CreateRequest\n}\ntype want struct {\n    *author.Schema\n    error\n}\n```\n\nOur `test` struct becomes\n\n```go\ntype test struct {\n    name string\n    args\n    want\n}\n```\n\nLike handler unit tests, we fill in the `test` slice with our data\n\n```go\ntests := []test{\n    {\n        name: \"simple\",\n        args: args{\n            CreateRequest: \u0026author.CreateRequest{\n                FirstName:  \"First\",\n                MiddleName: \"Middle\",\n                LastName:   \"Last\",\n                Books:      nil,\n            },\n        },\n        want: want{\n            Author: \u0026author.Schema{\n                ID:         1,\n                FirstName:  \"First\",\n                MiddleName: \"Middle\",\n                LastName:   \"Last\",\n                CreatedAt:  time.Time{},\n                UpdatedAt:  time.Time{},\n                DeletedAt: nil,\n                Books:     nil,\n            },\n            err: nil,\n        },\n    },\n}\n```\n\nTo instantiate a usecase, we call the `New()` function.\n\n```go\nuc := New(nil, repoAuthor, nil, nil, nil, nil)\n```\n\nWe only care about CRUD at this stage, so we only need to mock out the repository\nlayer. We start by creating a mock file and create a struct containing methods that \nmatches the signature defined in repository interface.\n\n`postgres.go`\n```go\ntype Repository interface {\n    Create(ctx context.Context, a *author.CreateRequest) (*author.Schema, error)\n}\n```\n`postgres_mock.go`\n```go\npackage database\n\ntype AuthorMock struct {\n\tCreateFunc func(ctx context.Context, a *author.CreateRequest) (*author.Schema, error)\n}\n```\n\nThen we implement `CreateFunc` method.\n\n```go\npackage database\n\ntype AuthorMock struct {\n    CreateFunc func(ctx context.Context, a *author.CreateRequest) (*author.Schema, error)\n}\n\nfunc (m *AuthorMock) Create(ctx context.Context, a *author.CreateRequest) (*author.Schema, error) {\n\treturn m.CreateFunc(ctx, a)\n}\n```\n\nAs mentioned before, we can use [mirip](https://github.com/gmhafiz/mirip) to\nautomatically generate this mock file.\n\n```go\n//go:generate mirip -rm -out postgres_mock.go . Repository\ntype Repository interface {...\n```\n\nNow that we have repository mock, we can start looping through `tests` variable.\n```go\nfor _, test := range tests {\n    t.Run(test.name, func(t *testing.T) {\n\n    }\n}\n```\n\nInside, we declare `\u0026repository.AuthorMock` for repository mock. It returns\nthe author and error that we want as declared in the table-test.\n\n```go\nrepoAuthor := \u0026repository.AuthorMock{\n    CreateFunc: func(ctx context.Context, r *author.CreateRequest) (*author.Schema, error) {\n        return test.want.Author, test.want.err\n    },\n}\n\nuc := New(nil, repoAuthor, nil, nil, nil)\n```\n\nWith the usecase declared, we can call its `Create()` method.\n\n```go\ngot, err := uc.Create(context.Background(), test.args.CreateRequest)\n```\n\nFinally, we perform a couple of assertions to check for error and response\n\n```go\nassert.Equal(t, test.want.error, err)\nassert.Equal(t, test.want.Author, got)\n```\n\nRun the test with\n\n```shell\ncd internal/domain/author/usecase\ngo test -run=\"TestAuthorUseCase_Create/simple\"\n\nPASS\nok      github.com/gmhafiz/go8/internal/domain/author/usecase   0.004s\n```\n\n### Repository\n\nUnit testing repository layer is different from above in the way that we test\nthem against real database using Docker, instead of using mocks. It is a lot\nmore complex to set up because now we need to do at least two things:\n\n1. Instantiate a new database in Docker\n2. Perform migration to create the tables\n3. Seed, if necessary\n\nA question arise wondering - if we are testing against a real database, even if\nit is a temporary one, is it really a unit test? Should it not be called an \nintegration test? Honestly it does not matter what it is called now. The fact\nthat we are still in one layer, the repository layer, not touching the usecase,\nor handler, I'd consider that we are still doing unit tests. Moreover, I find\nthat doing mocks for database is flaky and doesn't capture many edge cases, so\npotentially rendering them useless.\n\nMoving on, to set up, we use `TestMain`. It will run before all unit tests in this package. The code is basically a copy-paste from https://github.com/ory/dockertest. We \ncan customize the image (`postgres`) and version using tag (`15`). The username,\npassword and database name is not important, and they can be anything. These \ndatabases will be automatically shut down. In spite of spinning a database for \nthese tests, running these unit tests are still quick. For example, running all\n15 CRUD tests in this `database` package takes only **2** seconds.\n\n```go\nfunc TestMain(m *testing.M) {\n\t// 1. the place where we create our database container that starts really fast\n\t// 2. Run migration\n}\n\n```\nOnce the database is up, we need to create initial tables. Thus, we call\n`migrate(\"up\")`. Full code is in `postgres_test.go`.\n\nNow that we have database set up, we write our first repository unit test on\n`Create()`.\n\n```go\nfunc TestAuthorRepository_Create(t *testing.T) {\n\n}\n```\n\nLike other unit tests, we supply `args`, `want`, and `test` struct. Look at\n`type Repository interface` to know the function's signature to infer what are\nneeded and what it returns.\n\n```go\ntype args struct {\n    author *author.CreateRequest\n}\n\ntype want struct {\n    author *author.Schema\n    err    error\n}\n\ntype test struct {\n    name string\n    args\n    want\n}\n```\n\nIn the `test` slice, we supply the values. In this particular example, a simple \none. You may browse the codebase for more test scenarios.\n\n```go\nstartTime := time.Now()\n\ntests := []test{\n    {\n        name: \"normal\",\n        args: args{\n            author: \u0026author.CreateRequest{\n                FirstName:  \"First\",\n                MiddleName: \"Middle\",\n                LastName:   \"Last\",\n                Books:      nil,\n            },\n        },\n        want: want{\n            author: \u0026author.Schema{\n                    ID:         1,\n                    FirstName:  \"First\",\n                    MiddleName: \"Middle\",\n                    LastName:   \"Last\",\n                    CreatedAt:  time.Now(),\n                    UpdatedAt:  time.Now(),\n                    DeletedAt:  nil,\n                },\n            err: nil,\n        },\n    },\n}\n```\n\nEventually we need to call `Create()` method from this repository. For that we \nneed to instantiate a repository, and this repository requires a database client.\n\n```go\nvar (\n    dockerDB *db\n)\n\ntype dockerDB struct {\n    Conn *sql.DB\n    Ent  *gen.Client\n}\n\nfunc dbClient() *gen.Client {\n    sqlxDB := sqlx.NewDb(DockerDB.Conn, \"postgres\")\n    drv := entsql.OpenDB(dialect.Postgres, sqlxDB.DB)\n    client := gen.NewClient(gen.Driver(drv))\n    DockerDB.Ent = client\n    \n    return client\n}\n\nclient := dbClient()\n```\nThis database client is created from `sqlx`, and since we are using `ent` ORM, we create a client from it too. Then we store the `client` in a variable local to `database` package. This way, the database client is accessible to all other unit tests.\n\nWith a database client, we can now create a repository\n\n```go\nrepo := New(client)\n```\n\nTo run our table-test, like before, we iterate the `test` slice\n\n```go\nfor _, test := range tests {\n    t.Run(test.name, func (t *testing.T) {\n        ctx := context.Background()\n        \n        created, err := repo.Create(ctx, test.args.author)\n    })\n}\n```\nIt returns two values, `created` and `err`. So we assert them. Since we used real database, it returns values that we cannot know in advance like `created_at` field. If you scroll up a bit, you will notice that we declared a `startTime`. Using this trick I learned in the [#gopher](https://gophers.slack.com) Slack channel, you simply assert that both `created_at`, and `updated_at` timestamp values must be between that `startTime` value and the same as the generated timestamps.\n\n```go\nassert.Equal(t, err, test.want.err)\nassert.Equal(t, created.ID, test.want.author.ID)\nassert.Equal(t, created.FirstName, test.want.author.FirstName)\nassert.Equal(t, created.MiddleName, test.want.author.MiddleName)\nassert.Equal(t, created.LastName, test.want.author.LastName)\n\nassert.True(t, startTime.Before(created.CreatedAt) || startTime.Equal(created.CreatedAt))\nassert.True(t, startTime.Before(created.UpdatedAt) || startTime.Equal(created.UpdatedAt))\nassert.Equal(t, test.want.author.DeletedAt, created.DeletedAt)\n```\n\nTo run\n\n```shell\ncd internal/domain/author/repository/database\ngo test -run=\"TestAuthorRepository_Create/normal\"\n\nPASS\nok      github.com/gmhafiz/go8/internal/domain/author/repository/database       2.320s\n\n```\n\nTo get more test coverage, more tests need to be added to the table-test. For \nexample, inserting empty payload, test for errors, and inserting author with\nattached books. \n\nWhen doing multiple inserts, the ID will be increased. Remember that we are \ntesting against real database. So your expected ID should follow the increment.\n\nThe way tests are laid out, there is a single database client used by all\nunit tests. Alternatively, you can choose to have one database client for each\nof `Create()`, `Read()`, `Update()` and `Delete()` which would have mean 4 \nseparate databases in its own Docker container.\n\nNote that in doing a `Read()` unit test, if you choose to seed by doing an \ninsert, instead of using SQL script before reading, you are already doing an \nintegration test. An integration test is simply several unit tests that work \ntogether.\n\nIn conclusion, unit testing repository layer is more verbose as it needed a\nthird party library. However, the structure is still similar, with the addition\nof setting up the database. Another advantage is you can inspect the values\ninside the database by looking at the `databaseUrl` variable inside `TestMain()`.\n\n## Integration Testing\n\nAn integration testing tests if different modules at different layers work with each other. For example, we can start testing from handler layer by supplying simple inputs, and compare if the result is what is expected. This integration test go through use case and repository layer thus making sure all layers play well with each other.\n\nIntegration testing has been detailed in the Authentication's [integration testing](#integration-testing) section, but it can be run with\n\n```sh\ngo test -run Integration ./...\n```\n\n## End-to-End Test\n\nEnd-to-end tests is a way to make sure that the api behaves as intended for most commonly performed user interactions. It tries to simulate real-world scenario and thus our setup must also be as close as possible to a production setup. This kind of test is typically done after both unit and integration tests are completed. \n\nIn this approach, and you only need to write Go code for the tests. We run both api and database as Docker containers, then run database migrations on the api. Once that is done, we run another container that calls each HTTP endpoint. All of these operations are cobbled together with a docker-compose file and can be run in a single command. Since this approach is self-contained, it is quick to run, no port forwarding is needed, and can be shut down just as quick. This approach sounds similar to integration testing but the difference is we hit HTTP endpoints instead of initialising any modules in any layers. Note that there are [alternative approaches](#other-approaches) to doing this but the one demonstrated here is I believe the simplest to set up because it takes advantage of containers for isolation and it takes a single command to run.\n\nThe one thing that differs with production is all containers use distinct environment variables as defined in the `e2e` directory. Critical differences are the hostname for both api and database ad they use a name defined in the service section in the `e2e/docker-compose.yml` file. The reason is that Docker manages the hostnames within the network, so we cannot point to a container using its IP address. For example in our e2e docker compose file:\n\n```yaml\nversion: '3.8'\nservices:\n  postgres: # database container is named postgres \n    image: \"postgres:15.4\"\n    container_name: \"go8_postgres_e2e\"\n    restart: \"no\"\n    etc...\n\n  server: # our api is named server\n    image: \"go8/server_test\"\n    etc...\n```\n\nThus, we change environment variable from pointing to an IP addresses to hostnames: \n\n| Main .env         | e2e/.env         |\n|:------------------|:-----------------|\n| API_HOST=0.0.0.0  | API_HOST=server  |\n| DB_HOST=localhost | DB_HOST=postgres |\n\nThis means after api container is run, our e2e test is not calling `http://localhost:3080`, but instead, it calls the address with `http://server:3090` where `server` is the api address as defined in the docker-compose.yml file and the port number as defined in `e2e/.env` file.\n\nOur end-to-end program also needs to wait until the api server is ready before doing any HTTP calls to it. For that reason, we try to call the `/api/health/readiness` endpoint. If the api is not ready, we retry using exponential backoff. \n\n### Run \n\nTo run e2e test,\n\n```sh\ndocker-compose -f e2e/docker-compose.yml up --build\n```\n\nor using `task`.\n\n```sh\ntask test:e2e\n```\n\nThis (re)builds all containers (database, api, migration, e2e test) in correct order and run them. We purposely not run it daemon mode with `-d` option so that we can see log output. \n\n![end to end test](assets/run-end-to-end-tests.png)\n\nOnce all tests have been run, press Ctrl+C to end. This will stop the containers but not remove them. To remove the containers, run\n\n```sh\ndocker-compose -f e2e/docker-compose.yml down\n```\n\nor using `task`,\n\n```sh\ntask test:e2e:down\n```\n\n### Limitations\n\nSince a single database is used, each test can modify underlying data and interfere with expected output of other e2e tests. E2e tests tend to not change frequently as opposed to unit tests. \n\n### Improvements\n\nNo data seeding is demonstrated here. To make this e2e as real as possible, you can use real-ish production data using redacted or changed Personal Information Identifier (PII) as needed. Seeding can simply be part of migration or you can use a seeder module instead. \n\n### Other approaches\n\n1. Run main api as a separate goroutine, then run e2e test against it (run with flag). This results in fewer binaries, but data might need to be separate.\n2. Run both api and database as containers, expose the ports. Then use a third party tool or framework that helps in defining and running the tests. Since we are only testing against the api's port, this can be done in another language. \n3. Use a Go framework to simplify calling api endpoints and comparing values being returned.\n4. Instead of writing Go code, write the input and expected output in a [DSL](https://en.wikipedia.org/wiki/Domain-specific_language) like yaml. \n5. Some would like to do both e2e and UI testing in one go.  \n\n# TODO\n\n - [x] Fix end-to-end test\n - [x] Complete HTTP integration test\n - [x] Better return response\n - [x] LRU cache\n - [X] Redis Cache\n - [x] Opentelemetry\n   - [x] Tracing\n   - [x] Metric\n   - [x] Logging\n\n# Acknowledgements\n\n * https://quii.gitbook.io/learn-go-with-tests/questions-and-answers/http-handlers-revisited\n * https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html\n * https://github.com/moemoe89/integration-test-golang\n * https://github.com/george-e-shaw-iv/integration-tests-example\n * https://gist.github.com/Oxyrus/b63f51929d687c1e20cda3447f834147\n * https://github.com/sno6/gosane\n * https://github.com/AleksK1NG/Go-Clean-Architecture-REST-API\n * https://medium.com/@benbjohnson/standard-package-layout-7cdbc8391fc1\n * https://github.com/arielizuardi/golang-backend-blog\n \n# Appendix\n\n## Dev Environment Installation\n\nFor Debian:\n\n```shell\nsudo apt update \u0026\u0026 sudo apt install git curl build-essential docker docker-compose wget vim jq\nwget https://go.dev/dl/go1.24.0.linux-amd64.tar.gz\nsudo tar -C /usr/local -xzf go1.24.0.linux-amd64.tar.gz\nexport PATH=$PATH:/usr/local/go/bin\necho 'PATH=$PATH:/usr/local/go/bin' \u003e\u003e ~/.bash_aliases\necho 'PATH=$PATH:$HOME/go/bin' \u003e\u003e ~/.bash_aliases\nsource ~/.bashrc\ngo install golang.org/x/tools/...@latest\n\nsudo usermod -aG docker ${USER}\nnewgrp docker\nsudo systemctl restart docker\nsu - ${USER} # or logout and login\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgmhafiz%2Fgo8","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgmhafiz%2Fgo8","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgmhafiz%2Fgo8/lists"}