{"id":23173299,"url":"https://github.com/knoblauchpilze/backend-toolkit","last_synced_at":"2025-04-05T00:24:33.614Z","repository":{"id":264486531,"uuid":"893508336","full_name":"Knoblauchpilze/backend-toolkit","owner":"Knoblauchpilze","description":"Utilities to build a backend service with Go","archived":false,"fork":false,"pushed_at":"2025-03-24T20:07:53.000Z","size":86,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-24T21:22:28.019Z","etag":null,"topics":["backend","go","postgres","rest-api"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Knoblauchpilze.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-24T16:17:44.000Z","updated_at":"2025-03-24T20:07:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"561fd91b-eb4f-4ce7-8ba5-a225ccc920b6","html_url":"https://github.com/Knoblauchpilze/backend-toolkit","commit_stats":null,"previous_names":["knoblauchpilze/backend-toolkit"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Knoblauchpilze%2Fbackend-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Knoblauchpilze%2Fbackend-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Knoblauchpilze%2Fbackend-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Knoblauchpilze%2Fbackend-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Knoblauchpilze","download_url":"https://codeload.github.com/Knoblauchpilze/backend-toolkit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247268062,"owners_count":20911069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","go","postgres","rest-api"],"created_at":"2024-12-18T05:11:38.321Z","updated_at":"2025-04-05T00:24:33.608Z","avatar_url":"https://github.com/Knoblauchpilze.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# backend-toolkit\n\n# Overview\n\nThis project uses the following technologies:\n\n- [postgres](https://www.postgresql.org/) for the databases.\n- [go](https://go.dev/) as the server backend language.\n\n# Badges\n\n[![Build and test packages](https://github.com/Knoblauchpilze/backend-toolkit/actions/workflows/build-and-test-packages.yml/badge.svg)](https://github.com/Knoblauchpilze/backend-toolkit/actions/workflows/build-and-test-packages.yml)\n\n[![codecov](https://codecov.io/gh/Knoblauchpilze/backend-toolkit/graph/badge.svg?token=GDVROJ3V4Q)](https://codecov.io/gh/Knoblauchpilze/backend-toolkit)\n\n# Why this project\n\nThis project was born from creating multiple go projects for backend services and realizing that we mostly need the same starter-pack in regards to finding a web framework, building a CI, interacting with a database and so on.\n\nMost of these components are very similar from one project to the next and benefit greatly from being shared. This way all bugs can be easily ported to all projects through a versioning system and we can also very easily import it in a new project.\n\nAdditionally we don't need to worry so much about testing those common packages as it can be done directly in the base module.\n\n# Sample code\n\nA bit of code is usually really helpful to see what's possible with a certain module. So here's what's possible with this project:\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"net/http\"\n\t\"os\"\n\n\t\"github.com/Knoblauchpilze/backend-toolkit/pkg/db\"\n\t\"github.com/Knoblauchpilze/backend-toolkit/pkg/db/postgresql\"\n\t\"github.com/Knoblauchpilze/backend-toolkit/pkg/logger\"\n\t\"github.com/Knoblauchpilze/backend-toolkit/pkg/rest\"\n\t\"github.com/Knoblauchpilze/backend-toolkit/pkg/server\"\n\t\"github.com/labstack/echo/v4\"\n)\n\nfunc main() {\n\t// Create a logger printing to standard output\n\tlog := logger.New(logger.NewPrettyWriter(os.Stdout))\n\n\t// Create the connection to access the database\n\tdbConfig := postgresql.NewConfigForLocalhost(\"my-database\", \"my-user\", \"my-password\")\n\n\tconn, err := db.New(context.Background(), dbConfig)\n\tif err != nil {\n\t\tlog.Errorf(\"Failed to create db connection: %v\", err)\n\t\tos.Exit(1)\n\t}\n\tdefer conn.Close(context.Background())\n\n\t// Create the server\n\tserverConfig := server.Config{\n\t\tPort: 1234,\n\t}\n\ts := server.NewWithLogger(serverConfig, log)\n\n\t// Add a route with some handler\n\troute := rest.NewRoute(http.MethodGet, \"/info\", infoHandlerGenerator(conn))\n\tif err := s.AddRoute(route); err != nil {\n\t\tlog.Errorf(\"Failed to register route %v: %v\", route.Path(), err)\n\t\tos.Exit(1)\n\t}\n\n\t// Start the server\n\terr = s.Start(context.Background())\n\tif err != nil {\n\t\tlog.Errorf(\"Error while serving: %v\", err)\n\t\tos.Exit(1)\n\t}\n}\n\nfunc infoHandlerGenerator(conn db.Connection) echo.HandlerFunc {\n\treturn func(c echo.Context) error {\n\t\tsqlQuery := \"SELECT count FROM my-table\"\n\n\t\t// Use the connection to query the database and unmarshal the result\n\t\t// easily in an integer or a struct or anything you want\n\t\tvalue, err := db.QueryOne[int](c.Request().Context(), conn, sqlQuery)\n\t\tif err != nil {\n\t\t\treturn c.JSON(http.StatusInternalServerError, \"Failed to query database\")\n\t\t}\n\n\t\treturn c.String(http.StatusOK, value)\n\t}\n}\n```\n\nBy using only packges provided in this repository we are able to setup a server handling requests and fetching information from a database very easily.\n\nThe key features of the project are:\n\n- a simple way to configure a connection to a `postgres` database using [pgx](https://github.com/jackc/pgx).\n- an easy to use server using [echo](https://echo.labstack.com/) as a base.\n- a powerful logging system that leverages [zerolog](https://github.com/rs/zerolog) and integrates it with `echo`.\n\nWe define multiple tags and versions in this repository to make it easy to pinpoint a specific behavior and upgrade when needed.\n\n# Key features\n\n## Errors\n\nA fundamental aspect of Go is that error should be integrated as part of the normal flow of a program. To this end, we usually create quite a lot of distinct errors and can need to wrap errors when they are interpreted by a higher layer of the programs we write.\n\nIn this project we define the concept of an [error with code](https://github.com/Knoblauchpilze/backend-toolkit/blob/master/pkg/errors/error.go#L15) as follows:\n\n```go\ntype ErrorWithCode interface {\n\tCode() ErrorCode\n}\n```\n\nThe idea is that we can have an error with a code attached to it so that customers/developers can communicate specific problems that make it easy to identify afterwards what went wrong.\n\nThe package provides multiple convenience methods to create such an error either from scratch (meaning without previous error) or by wrapping an existing error (typically when a dependent third party library fails with an unrecoverable error that can't be handled at the current level).\n\nThe errors are also defining a message that can explain the code. An error can also be marshalled to JSON so that it's easy to interpret in HTTP calls. This looks like so:\n\n```json\n{\n  \"Code\": 1,\n  \"Message\": \"foo\",\n  \"Cause\": {\n    \"Code\": 1,\n    \"Message\": \"bar\"\n  }\n}\n```\n\nIt is encouraged to create custom error codes specialized to our business logic.\n\n## Logging\n\nAs a transverse concern, logging is usually quite important in a backend service. The main attributes we want to guarantee with a common package is:\n\n- easy to read.\n- easily customizable to allow display of headers and prefixes (typically modules or services).\n- ability to correlate logs for a request with one another.\n\nTo this end we used some capabilities provided by `echo` and `zerolog` and tried to make them work in combination.\n\n### Echo context\n\nBy default a handler using `echo` has the following prototype:\n\n```go\ntype HandlerFunc func(c echo.Context) error\n```\n\nThe `echo.Context` contains a logger which is attached by default to each request. It stems from the general logger configured when instantiating the `echo.Echo` object.\n\n### Binding zerolog to echo logger\n\nThe `zerolog` package and the `echo` package have slightly different interfaces to allow logging. In the future we might want to use a different logging backend. Therefore it does not seem very secure to expose the internals of the logging system we use outside of the package.\n\nIn the [logger](pkg/logger) package we defined a general log interface (`logger.Logger`) and provide some adapters to convert to other types. This is a typical way to bind different logging systems (see e.g. [pgx's adapter](https://github.com/jackc/pgx-zerolog) for `zerolog`).\n\nFor this project we have the following function:\n\n```go\nfunc Wrap(log Logger) echo.Logger {\n\t/* ... */\n}\n```\n\nThis allows to create a logger as usual and pass it over to the `echo` object easily.\n\n## Database interaction\n\nAn important part of a backend service is usually to interact with some database where the information is stored. For most of the projects we had to work with in the past this meant spinning up a `postgres` database and interact with it.\n\nWe've been using the [pgx](https://github.com/jackc/pgx) for a long time and found it quite versatile. The `db` package is using it under the hood but hiding some of the internals in an attempt to allow easily upgrading to newer version and hide some of the complexity of managing the connection to the database.\n\n### The connection\n\nThe main type brought by the `db` package is the [db.Connection](pkg/db/connection.go). It allows to start a transaction and execute some SQL code.\n\n### Querying\n\n`pgx` defines two main concepts: `Exec` and `Query`. The difference is explained in [this StackOverflow](https://stackoverflow.com/questions/60180651/what-are-the-differences-between-queryrow-and-exec-in-golang-sql-package) post and boils down (roughly) to whether we use `SELECT` or some other statement.\n\n`pgx` defines some very convenient methods to query data and automatically scan it into the fields of a struct or return it as a specific type. This eliminates a lot of the boilerplate we had to do in the past with the `rows.Scan(...)`.\n\nIn order to leverage generics, we would like to offer something like:\n\n```go\ntype myStruct struct {\n\tA int\n\tB string\n}\n\nfunc foo(conn db.Connection) error {\n\ts, err := conn.Query[myStruct](conn, \"SELECT A, B FROM my_table\")\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tfmt.Printf(\"My struct: %v\\n\", s)\n\n\treturn nil\n}\n```\n\nThe problem with this syntax is that the generic type to assign to the `Query` method would have to be known when creating the connection. Additionally we can't change the generic type of the `Query` method when a connection is created so it's not so flexible.\n\nTo this end, the `db` package defines a free function like below:\n\n```go\nfunc Query[T any](conn db.Connection, sqlQuery string, arguments ...any) (T, error) {\n\t/* ... */\n}\n```\n\n## The rest server\n\nAnother common aspect of offering a backend service is to have an HTTP server. In the past we usually used the [echo](https://echo.labstack.com/) framework. Although it's already providing some good abstraction, we noticed that some operations were quite common:\n\n- configuring the server (base path, port)\n- start and stop gracefully\n- register routes\n\nThis is the purpose of the [rest](pkg/rest) and [server](pkg/server) packages: they define utilities that can be used to easily register routes and attach them to a server. This server can in turn started and stopped easily.\n\n## Middleware\n\nNo matter the project and what HTTP handlers are actually doing, it's common that we expect some processing to happen for all of them. Typical examples are:\n\n- error recovery\n- timing\n- observability\n\nIn Go (and in most HTTP framework) those concerns are usually handled through middlewares. A middleware is a piece of code that 'decorates' an existing handler to enhance its capabilities. A typical example is a rate-limiting middleware which keeps track of how often an endpoint was called and by whom and denies some requests in case too many are received.\n\n### Request tracing\n\nAn important aspect of microservices is tracing. This allows to effectively follow the path of a request across services boundaries and is usually accomplished by adding a _correlation id_ to a request.\n\nIn this toolkit we act on this premise with two middlewares, described below.\n\n### The response envelope middleware\n\nIn order to provide consistent response format across an APIs, the [response_envelope](pkg/middleware/response_envelope.go) middleware captures the output of any incoming request and wraps it into something that looks like the following:\n\n```json\n{\n  \"requestId\": \"b8e9de68-3d49-4d40-a9a6-f8f3d3eab8f1\",\n  \"status\": \"SUCCESS\",\n  \"details\": {\n    \"value\": 12\n  }\n}\n```\n\nThis clearly indicates:\n\n- the request identifier\n- whether it succeeded or not\n- potential details about the success or failure\n\nHaving something like this in place for an API allows consumers to easily know whether a request was successful or not. Additionally we can rely on HTTP status codes to provided information about what went wrong.\n\nThe `ResponseEnvelope` middleware is added by default to the `Server`.\n\n### A note on generating the request identifier\n\nIn order to keep track of the journey of a request in the microservice architecture, the `ResponseEnvelope` middleware tries to retrieve an existing identifier from the headers of the request: if this exists, it uses it as a request id. If not, it generates a new one.\n\nThis allows to make sure that if the request is forwarded to another service (and as long as **the code is attaching the request id to the new HTTP request**) we will also see traces for the other service with this request's id.\n\n### Logging for a request\n\nAny backend service usually produces logs: in case a request fails or misbehave, it might be interesting to inspect traces for this specific request.\n\nWe already mentioned in the [logging](#logging) section that this framework makes it easy to configure a global logger than can be shared across the controllers and services.\n\nAdditionally with the [RequestTracer](pkg/middleware/request_tracer.go) middleware we can attach a custom logger with a prefix matching the request identifier.\n\nThis looks like the following (for an example with the [user-service](https://github.com/Knoblauchpilze/user-service)):\n\n```\n2024-12-15 13:55:26 INF [95ad22c8-8854-466c-83f7-2630fac365ba] GET localhost:60001/v1/users processed in 14.187412ms -\u003e 200\n```\n\nWe clearly see which request it is and can correlate the request across multiple services.\n\n# Installation\n\nThe tools described below are directly used by the project. It is mandatory to install them in order to build the project locally.\n\nSee the following links:\n\n- [golang](https://go.dev/doc/install): this project was developed using go `1.23.2`.\n- [golang migrate](https://github.com/golang-migrate/migrate/blob/master/cmd/migrate/README.md): following the instructions there should be enough.\n- [postgresql](https://www.postgresql.org/) which can be taken from the packages with `sudo apt-get install postgresql-14` for example.\n\nWe also assume that this repository is cloned locally and available to use. To achieve this, just use the following command:\n\n```bash\ngit clone git@github.com:Knoblauchpilze/backend-toolkit.git\n```\n\n**Note:** the migrate tool and postgres are only used for the test database and for tests. If you don't install them it means you'll not be able to run the tests locally.\n\n# How to extend this project\n\nAs this project is meant to contain common tools to kick-start a backend project in Go, additions are welcomed. As a general rule, anything we add here should be:\n\n- generic enough so that it can be used in multiple projects.\n- well tested in order to guarantee a good level of quality for the base code.\n- convenient to use either with structures from the standard library or from this project.\n\nIf you think what you want to propose fits those criteria, you can open a [new PR](https://github.com/Knoblauchpilze/backend-toolkit/pulls) in this project or [an issue](https://github.com/Knoblauchpilze/backend-toolkit/issues).\n\n# Create a new release\n\nA convenience script is provided in the [scripts](scripts) folder to create a new release: [create_release.sh](scripts/create_release.sh).\n\nYou can run this script as follows:\n\n```bash\n./create_release.sh v1.2.3\n```\n\nThe version is optional: in case it's not provided, the script will try to determine the latest one and add one to it. Typically if the latest published version available in the repository (**locally**) is `v1.2.3`, running the script without arguments will result in a version `v1.2.3.1` being created.\n\nThe new version will automatically be published to the public repository. Therefore use this script with care.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknoblauchpilze%2Fbackend-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fknoblauchpilze%2Fbackend-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknoblauchpilze%2Fbackend-toolkit/lists"}