{"id":20472040,"url":"https://github.com/jonboh/declarative_data_pipelines","last_synced_at":"2025-06-30T03:06:17.867Z","repository":{"id":225904340,"uuid":"766618747","full_name":"jonboh/declarative_data_pipelines","owner":"jonboh","description":"A fully declarative data pipeline. This is an example project, intended as a learning resource.","archived":false,"fork":false,"pushed_at":"2024-03-04T21:39:10.000Z","size":3198,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-05T13:49:30.869Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Nix","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonboh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-03T18:52:49.000Z","updated_at":"2024-03-04T21:40:00.000Z","dependencies_parsed_at":"2024-03-04T22:45:30.143Z","dependency_job_id":null,"html_url":"https://github.com/jonboh/declarative_data_pipelines","commit_stats":null,"previous_names":["jonboh/declarative_data_pipelines"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jonboh/declarative_data_pipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonboh%2Fdeclarative_data_pipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonboh%2Fdeclarative_data_pipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonboh%2Fdeclarative_data_pipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonboh%2Fdeclarative_data_pipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonboh","download_url":"https://codeload.github.com/jonboh/declarative_data_pipelines/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonboh%2Fdeclarative_data_pipelines/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262702325,"owners_count":23350641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T14:18:16.746Z","updated_at":"2025-06-30T03:06:17.841Z","avatar_url":"https://github.com/jonboh.png","language":"Nix","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Declarative, Reproducible, Free and Open Source Data Pipelines\nThis repository is the accompanying material to a talk I gave to the students of\n[Deusto's Digital Industry dual degree](https://www.deusto.es/es/inicio/estudia/estudios/grado/grado-dual-industria-digital).\n\nIn this talk we walk through the skeleton of a data pipeline configured using the\nNix language. We use a Nix Flake to define the system, this provides us with a completely\ndeclarative and reproducible system.\n\nThe primary aim of the talk is to introduce to students the kinds of systems that they\nwill end up building to consume data in their professional career in a way that will\nallow them to reproduce the system in their own hardware.\n\nThe system presented is not intended to represent a production ready environment but an\nexperimentation playground.\n\n\n# Data pipeline sub-systems\n\n## OPC Server\nRepresenting the OT level of our pipeline we will deploy a very simple OPC Server\nthat will publish 3 variables that change with different frequencies.\n\n## Telegraf Collector Agent\nReading from the OPC server we have a Telegraf agent that will write the values read\nto InfluxDB and Kafka.\n\n## InfluxDB\nInfluxDB is a timeseries datatabase that will serve as our storage solution for the\nvalues read and the computations derived from these values.\n\n## Apache Kafka\nA message broker to stream data to the model and its \"predictions\" back to the Database.\n\n## Custom Rust Model\nA very simple Rust binary that will be subscribed to the `opc` topic where the Collector Telegraf\nagent will send all the OT data.\nThis model wil perform two computations and write them back to Kafka in the `model` topic.\n\n## Telegraf Kafka Agent\nThis telegraf agent reads from the `model` topic and writes the values into the database.\n\n## Grafana Dashboard\nFinally we display all the series in a Grafana dashboard.\n\n\n# Instructions\nThis guide assumes you are running this project in a Linux machine\n(not necessarily a NixOS one).\n\nMost commands are expected to be run from the root of this project.\n```bash\ngit clone git@github.com:jonboh/declarative_data_pipelines.git\ncd declarative_data_acquisition\n# continue with the following steps\n```\n\n## Prepare the development environment\n### Install Nix\nFollow the [official instructions](https://nixos.org/download) and install the Nix\npackage manager.\n\n### Activate Flake support\nYou can follow the [wiki's instructions](https://nixos.wiki/wiki/Flakes).\nBut in short, you need to write into `/etc/nix/nix.conf` the following content:\n```\nexperimental-features = nix-command flakes\n```\n\n### Activate the development environment\nGo to this projects' folder and run:\n```bash\nnix develop\n```\nNix will download all the necessary development tools to run this project.\nAfter that you can check that it has worked by running some of them, for example:\n```\nage --help\n```\nshould result in the help of the `age` tool.\n\n## Regenerate the secrets\nTo reproduce:\n1. Generate a new age key:\n```bash\nage-keygen -o ./master-age.key\n```\n2. Add your key as the master key of the project in .sops.yaml:\n```yaml\nkeys:\n  - \u0026master \u003cyour public age key here\u003e\ncreation_rules:\n  - path_regex: secrets/*\n    key_groups:\n    - age:\n      - *master\n```\nIf you open the .sops.yaml versioned in this repository you'll see that there's already\na public key, that's the public key of my age key, the one I used to reproduce this\nconfiguration in my computer. As unencrypted keys should never be versioned and I haven't\nshared the key with you, you need to create your own (previous step) and reencrypt\nthe token and password for InfluxDB (next step).\n\n3. Generate a new token and password for InfluxDB\n```bash\nsops secrets/influxdb_token\n```\nThis command will open your editor (configured value of $EDITOR).\nWrite in this file the content of the token that will be used by applications\nthat want to communicate with InfluxDB (Grafana and Telegraf).\nFor example:\n```\ndevtoken\n```\nNow repeat the process to generate the password for the InfluxDB user. This password\nwill be used to log in the InfluxDB Portal.\n```bash\nsops secrets/influxdb_password\n```\nand write a password (it should be longer than 8 characters), for example:\n```\ndevpassword\n```\n\n## Set the network device for communication between systems\nModify the `net_device` in `flake.nix`. You should set this to the network interface\nin which the Raspberry Pi will be connected. Usually it will be  your main ethernet\nnetwork device.\n\n## Add your ssh public key to the systems\nModify `common/configuration.nix` and add your key to the list of authorized ssh keys, for example:\n```nix\n  users.users.admin = {\n    openssh.authorizedKeys.keys = [\n      \"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAninVG6bOxD7bOi7od3WJJvPAV7DEiejNqHXrRqzdKW jon.bosque.hernando@gmail.com\"\n      # you should put here your ssh public key.\n    ];\n  };\n```\nThis will allow you to log into the admin user in each of the systems with through ssh.\nAlso it will allow you to remotely push configuration changes without the need to rebuild the virtualbox\nand raspberry images.\n\n## Build the systems\nJust build the output packages of the flake:\n```bash\nnix build \".#opc@raspberry\" --system aarch64-linux\n# or if you are running everything on virtualbox:\nnix build \".#opc@vbox\"\n```\n```bash\nnix build \".#collector@vbox\"\n```\n```bash\nnix build \".#db@vbox\"\n```\nEach of these commands will generate a symlink `result` to the output of the build.\nTo deploy each of the systems you will need the image of each one, so, build one,\ndeploy it (next step) and then build the next one and repeat.\n\n\n## Deploy the systems\n### First Time\n#### Raspberry\nBurn the image into your sd card.\n\nWARNING!: the following `dd` command will wipe any content in the location pointed to `of` (your sd card):\n```\nzstd -d result/sd-image/nixos-sd-image-23.11.20240228.c8e74c2-aarch64-linux.img.zst -o opc-rasp-image.img\ndd if=opc-rasp-image.img of=/dev/\u003cyour_sdcard\u003e bs=4M status=progress\nsync\n```\nRemove the sdcard, insert it into the raspberry and boot it up. The raspberry should get\nto the login screen in the TTY. At this point the OPC server will already be running.\n\nYou won't be able to log in into admin because a password is not set by default.\nTo enter the raspberry you need to use ssh into it. You don't need to do it though.\n\n#### VirtualBox Images\nOpen VirtualBox and import the generated images.\nThen start each of the images.\n\n#### Deploying the Master key for secrets\nNow you should have all the systems up and running, however they don't have the\nmaster key to decrypt the secrets that we generated in the first steps. This means\nthat Grafana and the Telegraf collector won't be able to read and write into the DB.\n\nIn a production environment you would not need to do this step as you could use a KMS.\nBut we have to deploy the `master-age.key` ourselves.\nCopy through ssh the master key into `collector@vbox` and `db@vbox`, and then reboot the system\nso that secrets are decrypted and made available.\n```bash\nscp ./master-age.key admin@collector-vbox:\nssh collector-vbox \"sudo reboot\"\nscp ./master-age.key admin@db-vbox:\nssh db-vbox \"sudo reboot\"\n```\nYou will need to define in your `.ssh/config` those two ssh hosts:\n```\nHost db-vbox\n   HostName 192.168.0.10\n   User admin\n   IdentityFile ~/.ssh/id_ed25519\n   IdentitiesOnly yes\n\nHost db-vbox\n   HostName 192.168.0.10\n   User admin\n   IdentityFile ~/.ssh/id_ed25519\n   IdentitiesOnly yes\n```\n\n### Changing Configurations\nThis section applies to both VirtualBox machines and raspberry ones.\nIf you have configured an ssh key in `common/configuration.nix` you can push\nconfiguration changes through ssh with a command like this:\n```bash\nnixos-rebuild switch --flake \".#db@vbox\" --target-host db-vbox --use-remote-sudo\n```\nNote that `db-box` in `--target-host` is an entry in my `.ssh/config`, as shown in the\nprevious step.\n\nYou can check that the configuration is correct by ssh-ing to the machine:\n```bash\nssh db-vbox\n```\nThis command should log you in the `db@vbox` machine. Running `hostname` there\nshould return `db-host`.\n\n## Checking the working systems\nAt this point we are done, you can use the services that have been deployed or tweak their configurations.\nWell done!\n\n### Grafana Dashboard\nUsing your browser go to the address of Grafana, which runs in `db-host`, by default: `192.168.0.10:3000`.\nLog in with the default admin user.\n```\nuser: admin\npassword: admin\n```\nGo to Home-\u003eDashboards-\u003eSensor Dashboard\nYou should see for plots. The three top ones correspond to the OPC variables read by `ot-collector`\nThe forth one corresponds to the computations performed by the model running in `db-host`.\n\n### Apache Kafka Topics\nYou can use kafkacat `kcat` command to listen to the two topics in Apache Kafka\nTo listen to the topic that receives the OT data (`opc` topic) run:\n```bash\nkcat -b 192.168.0.10:9092 -t opc -C\n```\nTo listen to the topic that receives the model computations (`model` topic) run:\n```bash\nkcat -b 192.168.0.10:9092 -t model -C\n```\n\n### InfluxDB\nUsing your browser go to the address of InfluxDB, which runs in `db-host`, by default: `192.168.0.10:8086`.\nLog in with `devuser`:\n```\nuser: devuser\npassword: \u003cwhatever you set on secrets/influxdb_password\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonboh%2Fdeclarative_data_pipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonboh%2Fdeclarative_data_pipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonboh%2Fdeclarative_data_pipelines/lists"}