{"id":21161322,"url":"https://github.com/apinf/emqttd-nix","last_synced_at":"2025-08-29T13:08:06.891Z","repository":{"id":87590773,"uuid":"110216760","full_name":"apinf/emqttd-nix","owner":"apinf","description":"Nix expressions to build emqttd, for use in apinf.","archived":false,"fork":false,"pushed_at":"2017-12-19T14:58:37.000Z","size":28,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-21T09:28:00.510Z","etag":null,"topics":["emqttd","erlang-developement","nix","nixpkgs","reproducible-builds"],"latest_commit_sha":null,"homepage":null,"language":"Nix","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"eupl-1.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apinf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-10T07:30:01.000Z","updated_at":"2019-03-14T22:39:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"24df26eb-f107-4f58-8f41-0a2e068408be","html_url":"https://github.com/apinf/emqttd-nix","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apinf%2Femqttd-nix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apinf%2Femqttd-nix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apinf%2Femqttd-nix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apinf%2Femqttd-nix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apinf","download_url":"https://codeload.github.com/apinf/emqttd-nix/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243606954,"owners_count":20318314,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["emqttd","erlang-developement","nix","nixpkgs","reproducible-builds"],"created_at":"2024-11-20T13:12:54.475Z","updated_at":"2025-03-14T16:11:01.275Z","avatar_url":"https://github.com/apinf.png","language":"Nix","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nix expressions for emqttd\n\nThis repo contains nix-expressions to build emqttd and a docker image for it in a deterministic manner.\n\n# Usage\n\nEnsure [nix][] is installed, and run\n\n`nix-build` to build emqttd. This version is expected to be run in a container.\n\n## Docker image\n\n`nix-build docker.nix` will build a docker image. Note that this does not require docker to be installed on the machine.\n\nA path that looks similar to\n`/nix/store/inxac5930nz66gsx64wvx8hh78bfaayv-docker-image-apinf-emqttd.tar.gz`\nwill be printed at the end, if the build was succesful.  This is an archive in\nthe `docker save` format, and can be transferred to the production machines or\ndistributed using container registries.\n\nTo load the image into a local docker daemon, run `docker load -i \u003cpath-to-image\u003e`.\n\nThe image expects hostnames `elasticsearch` and `postgres` to resolve to the respective services.\n\n[nix]: https://nixos.org/nix/\n\n## Docker compose\n\nThe version of emqttd bundled here has elasticsearch logger and auth_pgsql modules enabled. To run everything together,\na compose file is provided for convenience. With `docker-compose` installed, run\n\n```\ncd docker-compose\ndocker-compose up\n```\n\nThis will also run elasticsearch and postgresql populated with default values.\nNote that there may be start order issues at times, we are working on improving\nappropriate waiting and retry strategies.\n\nTo prevent startup order issues, you can run\n\n```\ndocker-compose up postgres\ndocker-compose up elasticsearch\ndocker-compose up emqttd\n```\n\n# Testing\n\nThere is no automated end to end test suite (yet). To verify if everything\nworks as expected, install an mqtt client and publish and/or subscribe to\nemqttd.\n\nVerify the following:\n\n - Access control rules set in postgres database must be honored for publish/subscribe.\n - Events corresponding to various phases in mqtt workflow should be logged to Elasticsearch.\n\nWhen running from docker compose, the testing must happen in containers\nconnected to the same network, or with ports exposed to host to prevent\nconnectivity issues.\n\n# FAQ\n\n## Why is this necessary?\n\n### Short answer\n\nWe need a way to build emqttd in a deterministic manner, with strong assurance\nthat simply repeating the build won't result in a wildly different result.\n\n### Long answer\n\nA simple make in `emq-relx` repository will build emqttd, and there is a\nhelpful `emq-docker` repo to build a docker image.  However, there is no\nguarantee that if the build is repeated, the results are functionally\nidentical.\n\nErlang ecosystem has a dependency problem.  Semantic versioning was not around\nwhen a majority of projects were written.  Most projects still use source\ndependencies, often referring to a git branch or tag instead of an sha.  Lock\nfiles were unheard of until rebar3 or mix, but many projects still use\nerlang.mk or rebar.\n\nAs a result, it is possible (and not very uncommon) for two builds made at\ndifferent points of time to use different source versions of certain\ndependencies altogether.  Most of the time the updates may be\nbackwards-compatible, but it can be frustrating to have to hunt down the\nchanges.\n\nCommonly packages are deployed in compiled form, exact versions of sources are\nnot recorded.  When an issue is encountered in production, there is no fool\nproof way to identify the corresponding source.\n\nIn addition, erlang package namespace is global. i.e; there can only be one\nversion of a package loaded into the erlang VM at any point of time. (Ignoring\nhot code upgrades). However global dependency resolution is a relatively new\nfeature introduced with rebar3 and mix, both of which are not widely adopted.\nThis can result in multiple copies of the same dep being pulled and compiled\nduring a typical build.\n\nAll of this results in a lot of headache when trying to debug issues in\nproduction.  Not being able to reproduce a build is silly. It is an\nunforgivable sin to ship products that can not be reliably rebuilt from source.\n\n## How does this work?\n\n[Nix][nix] is a functional package manager. To quote \u003chttps://nixos.org/nix\u003e:\n\n\u003e Nix is a powerful package manager for Linux and other Unix systems that makes\n\u003e package management reliable and reproducible.\n\nThe internals of nix are out of scope here, and are described in detail in the\ndocumentation.\n\nWe just asked nix to build the dependencies and finally emqttd and the docker image.\n\nThe builds are repeatable since expected git revisions and the hashes of sources\nare tracked in the repository. The expressions allow building everything\n(including erlang, bash, gcc etc) from source in a predictable manner.\n\n## Why is the docker image so large?\n\nThe smallest image can probably be made in under 20MB, whereas the images built\nusing this repo are a little over half a GB in size. This is due to a few reasons:\n\n1. Some nix expressions we use are very conservative and include all possible\n   dependencies. This increases the closure size.\n2. The entire build closure is included in the image. This includes the exact\n   erlang version, the exact bash version, sources of erlang deps, etc. Most of\n   this is not strictly necessary, however, having this around does not\n   particularly hurt.\n3. The lean image uses musl libc whereas we use glibc. This alone makes a huge\n   difference, and replacing it takes some effort.\n\nWe believe the increased size is worth the benefits of reproducible builds,\nwhen storage is relatively cheap. We will consider replacing glibc with musl\nlibc to reduce the size if it becomes a pressing issue.\n\n## Won't updating the release be difficult?\n\nUpdating the release requires each dependency to be carefully considered, and\nits derivation to be updated. This is a _good_ thing.\n\nUpdating the revisions and hashes can be automated with some effort, and the\nresulting workflow can be made as simple as that of working with similar\nlocking and verifying package managers and build systems, for example yarn.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapinf%2Femqttd-nix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapinf%2Femqttd-nix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapinf%2Femqttd-nix/lists"}