Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tfc/nix_cmake_example
An example with nix: How to automatically and reproducibly build and integration-test a database C++ app in 60 different build-configurations (library exchange/compiler exchange/static-nonstatic)
https://github.com/tfc/nix_cmake_example
build-automation build-configuration c-plus-plus dependency-injection dependency-manager docker docker-image integration-testing nix python reproducible-builds virtualization workflow
Last synced: about 14 hours ago
JSON representation
An example with nix: How to automatically and reproducibly build and integration-test a database C++ app in 60 different build-configurations (library exchange/compiler exchange/static-nonstatic)
- Host: GitHub
- URL: https://github.com/tfc/nix_cmake_example
- Owner: tfc
- Created: 2018-09-25T10:21:51.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2022-08-28T13:38:17.000Z (about 2 years ago)
- Last Synced: 2023-03-11T01:38:32.260Z (over 1 year ago)
- Topics: build-automation, build-configuration, c-plus-plus, dependency-injection, dependency-manager, docker, docker-image, integration-testing, nix, python, reproducible-builds, virtualization, workflow
- Language: Nix
- Homepage:
- Size: 210 KB
- Stars: 50
- Watchers: 4
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# An example nix project with C++ and Python
This project consists of 2 simple example applications that model a writer and
a reader to/from a postgres database:
The "server" application (C++) listens on TCP socket 1300 and writes into the
database whatever it receives.
The "webserver" application (Python) is either a script that just prints the
database content, or a webserver script (both included) that listens on port
8000 and lists the latest 10 database "messages".
Both applications are relatively simple and stupid as they are mere
demonstration vehicles for showing:
- how to setup a reproducible working environment with toolchain and library
dependencies etc. with a single command using `nix`
- how to build the application(s) with different compilers, different library
versions, and select dynamic/static linking with a single parametrized
command using `nix`
- how to make the build process reproducible so that it still works in a few
years on completely different linux environments, by pinning all versions of
the toolchain down using `nix`
- how to automatically create minimal [`docker`](https://www.docker.com/) images
(so minimal that they contain no base distro image) using `nix`
- how to run completely automated integration test scenarios in VMs using `nix`
## Dependencies
At **compile time**, the applications need the following:
- server
- C++ compiler toolchain ([GCC](https://gcc.gnu.org/) or
[clang](https://clang.llvm.org/))
- [`cmake`](https://cmake.org/) (meta) build system
- [`boost`](https://www.boost.org/) library
- [`libpqxx`](http://pqxx.org/development/libpqxx) library (C++ postgres
interface)
- [`gtest`](https://github.com/google/googletest) unit test framework
- webserver
- [`python`](https://www.python.org/) version 3
- [`flask`](http://flask.pocoo.org/) library (webserver)
- [`psycopg2`](http://initd.org/psycopg/) library (postgres interface)
The applications can be compiled (the C++ app) and both be run without nix at
all.
But in that case you have to install/configure all dependencies yourself -
with `nix`, everything will be handled automagically.
## Build
In order to build the server app, do the following:
```bash
$ cd $project/server
$ nix-build
# You may now run ./result/bin/messagedb-server
```
In order to package the python app into a wrapper script that can be called
like any binary, do the following:
```bash
$ cd $project/python_client
$ nix-build
# You may now run ./result/bin/mdb-webserver
```
The python wrapper calls python with the dependencies in place, leaving no way
to accidentally call it with the wrong python interpreter.
(This way of packaging a python app is completely optional, `nix` does not force
you to do so)
`nix` can be obtained [here](https://nixos.org/nix/download.html).
There is always the quick and unsafe `curl | sh` way to install, but there are
also instructions on the project website that include binary verification.
Please note that tab completion in the following examples works out of the
box on NixOS, while users of other distros have to install it manually.
https://github.com/hedning/nix-bash-completions
## Build in different configurations
The server app can be compiled in different configurations:
- Compiler (can be easily extended/customized.)
- GCC 7.3.0 (That is the default choice in the current package list. Can be
easily parametrized, of course!)
- GCC 8.2.0
- clang 5.0.2
- clang 6.0.1
- clang 7.0.0
- `boost` versions 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.6.8
- Binary linkage
- static (single-file with no further runtime dependencies)
- dynamic
To select one configuration, run:
```bash
$ cd $project
$ nix-build release.nix -A
integrationtest-mdb-server-clang7-boost166 integrationtest-mdb-server-static-clang8-boost169 mdb-server-gcc8-boost166
integrationtest-mdb-server-clang7-boost167 integrationtest-mdb-server-static-gcc7-boost166 mdb-server-gcc8-boost167
integrationtest-mdb-server-clang7-boost168 integrationtest-mdb-server-static-gcc7-boost167 mdb-server-gcc8-boost168
integrationtest-mdb-server-clang7-boost169 integrationtest-mdb-server-static-gcc7-boost168 mdb-server-gcc8-boost169
integrationtest-mdb-server-clang8-boost166 integrationtest-mdb-server-static-gcc7-boost169 mdb-server-static-clang7-boost166
integrationtest-mdb-server-clang8-boost167 integrationtest-mdb-server-static-gcc8-boost166 mdb-server-static-clang7-boost167
integrationtest-mdb-server-clang8-boost168 integrationtest-mdb-server-static-gcc8-boost167 mdb-server-static-clang7-boost168
integrationtest-mdb-server-clang8-boost169 integrationtest-mdb-server-static-gcc8-boost168 mdb-server-static-clang7-boost169
integrationtest-mdb-server-gcc7-boost166 integrationtest-mdb-server-static-gcc8-boost169 mdb-server-static-clang8-boost166
integrationtest-mdb-server-gcc7-boost167 mdb-server-clang7-boost166 mdb-server-static-clang8-boost167
integrationtest-mdb-server-gcc7-boost168 mdb-server-clang7-boost167 mdb-server-static-clang8-boost168
integrationtest-mdb-server-gcc7-boost169 mdb-server-clang7-boost168 mdb-server-static-clang8-boost169
integrationtest-mdb-server-gcc8-boost166 mdb-server-clang7-boost169 mdb-server-static-gcc7-boost166
integrationtest-mdb-server-gcc8-boost167 mdb-server-clang8-boost166 mdb-server-static-gcc7-boost167
integrationtest-mdb-server-gcc8-boost168 mdb-server-clang8-boost167 mdb-server-static-gcc7-boost168
integrationtest-mdb-server-gcc8-boost169 mdb-server-clang8-boost168 mdb-server-static-gcc7-boost169
integrationtest-mdb-server-static-clang7-boost166 mdb-server-clang8-boost169 mdb-server-static-gcc8-boost166
integrationtest-mdb-server-static-clang7-boost167 mdb-server-docker mdb-server-static-gcc8-boost167
integrationtest-mdb-server-static-clang7-boost168 mdb-server-docker-static mdb-server-static-gcc8-boost168
integrationtest-mdb-server-static-clang7-boost169 mdb-server-gcc7-boost166 mdb-server-static-gcc8-boost169
integrationtest-mdb-server-static-clang8-boost166 mdb-server-gcc7-boost167 mdb-webservice
integrationtest-mdb-server-static-clang8-boost167 mdb-server-gcc7-boost168 mdb-webservice-docker
integrationtest-mdb-server-static-clang8-boost168 mdb-server-gcc7-boost169
$ nix-build release.nix -A mdb-server-gcc7-boost166
# You may now run ./result/bin/messagedb-server
```
## Generate `docker` images for both apps
Compiling and installing the docker images is simple.
The step of the docker image creation does not even require you to have `docker`
on your system.
server:
```bash
$ cd $project
$ nix-build release.nix -A mdb-server-docker
$ du -sh $(readlink -f result)
46M /nix/store/bnd9gzdgmrvdxd7cdrm7fxr39db8phfk-docker-image-mdb-server.tar.gz
```
The docker image with the static version of the library is a bit smaller:
```bash
$ nix-build release.nix -A mdb-server-docker-static
$ du -sh $(readlink -f result)
22M /nix/store/58lpaf0yh6skjdxdd0qfc4vrw9nj1j76-docker-image-mdb-server.tar.gz
```
Please note that this can of course be further reduced, but then the example
would go out of scope and we have a nice starting point already.
The webserver docker image:
```bash
$ nix-build release.nix -A mdb-webservice-docker
$ du -sh $(readlink -f result)
60M /nix/store/l4ipb3p325yaflsdna8h7nijla61vwrz-docker-image-mdb-webservice.tar.gz
```
What's particularly interesting about these docker images compared to
conventionally created images is, that they only contain the run-time
dependencies they need.
This means that for the statically linked binary, the corresponding docker
image, in addition to bash and its run-time dependencies, only contains the
binary itself. Likewise, for the dynamically linked binary, and the python
webserver, the docker images contain exactly the needed dependencies, e.g.
Python 3.xyz for the webserver.
In contrast, building a Docker image traditionally, e.g., with a Docker file
built upon a Ubuntu/Debian/Fedora base image, would quickly result in an image
size around hundreds or MB.
## Integration tests
There is only one integration test in this example.
It does the following
- Create a complete Linux VM that contains at least:
- A preconfigured postgres DB (with role and DB for test user)
- both applications
Running the test does the following:
- Boot the VM
- wait for postgres to start
- then start the server app
- then start the webservice app
- feed some data into the VM via the server app
- check if the DB is in the right state
- check if the webserver's output is correct
This integration test scenario can easily be run with all different build
configurations of the server app.
The NixOS distribution tests many of its packages with such integration tests.
These are however much more complex by running full herds of VMs in virtual
networks, to some extent even with the X desktop activated and OCR etc. - out
of scope for this example.
To run the integration tests, do:
```bash
$ nix-build release.nix -A integrationtest-mdb-server-
integrationtest-mdb-server-clang7-boost166 integrationtest-mdb-server-gcc7-boost169 integrationtest-mdb-server-static-clang8-boost168
integrationtest-mdb-server-clang7-boost167 integrationtest-mdb-server-gcc8-boost166 integrationtest-mdb-server-static-clang8-boost169
integrationtest-mdb-server-clang7-boost168 integrationtest-mdb-server-gcc8-boost167 integrationtest-mdb-server-static-gcc7-boost166
integrationtest-mdb-server-clang7-boost169 integrationtest-mdb-server-gcc8-boost168 integrationtest-mdb-server-static-gcc7-boost167
integrationtest-mdb-server-clang8-boost166 integrationtest-mdb-server-gcc8-boost169 integrationtest-mdb-server-static-gcc7-boost168
integrationtest-mdb-server-clang8-boost167 integrationtest-mdb-server-static-clang7-boost166 integrationtest-mdb-server-static-gcc7-boost169
integrationtest-mdb-server-clang8-boost168 integrationtest-mdb-server-static-clang7-boost167 integrationtest-mdb-server-static-gcc8-boost166
integrationtest-mdb-server-clang8-boost169 integrationtest-mdb-server-static-clang7-boost168 integrationtest-mdb-server-static-gcc8-boost167
integrationtest-mdb-server-gcc7-boost166 integrationtest-mdb-server-static-clang7-boost169 integrationtest-mdb-server-static-gcc8-boost168
integrationtest-mdb-server-gcc7-boost167 integrationtest-mdb-server-static-clang8-boost166 integrationtest-mdb-server-static-gcc8-boost169
integrationtest-mdb-server-gcc7-boost168 integrationtest-mdb-server-static-clang8-boost167
```
The result of such integration tests is a HTML/XML log document in the `result`
folder.
## Nix CI
NixOS comes with a service for continuous integration, called
[`hydra`](https://github.com/NixOS/hydra).
It does not look as polished as other CIs and its integration is very github
centric (because `nix` and `NixOS` source code and package lists are hosted on
github), but it is generally extensible with plugins.
A screenshot of a private hydra instance building this project:
