{"id":24358792,"url":"https://github.com/tonyfahrion/howsystemswork","last_synced_at":"2026-02-23T01:31:07.684Z","repository":{"id":66761828,"uuid":"76367495","full_name":"tonyfahrion/howsystemswork","owner":"tonyfahrion","description":"How systems work, at least for me","archived":false,"fork":false,"pushed_at":"2016-12-22T17:56:06.000Z","size":47,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-30T03:42:36.690Z","etag":null,"topics":["principles"],"latest_commit_sha":null,"homepage":"https://twolfy.github.io/howsystemswork/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tonyfahrion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-13T14:44:30.000Z","updated_at":"2017-01-02T13:58:35.000Z","dependencies_parsed_at":"2023-03-13T20:29:34.844Z","dependency_job_id":null,"html_url":"https://github.com/tonyfahrion/howsystemswork","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tonyfahrion/howsystemswork","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyfahrion%2Fhowsystemswork","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyfahrion%2Fhowsystemswork/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyfahrion%2Fhowsystemswork/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyfahrion%2Fhowsystemswork/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tonyfahrion","download_url":"https://codeload.github.com/tonyfahrion/howsystemswork/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyfahrion%2Fhowsystemswork/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29734468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-22T20:09:16.275Z","status":"ssl_error","status_checked_at":"2026-02-22T20:09:13.750Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["principles"],"created_at":"2025-01-18T20:22:47.560Z","updated_at":"2026-02-23T01:31:07.616Z","avatar_url":"https://github.com/tonyfahrion.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# How systems work\n\n\u003e At least for me\n\nThis is my view of how my daily Systems world works. It is not an intend to produce the most profound Systems documentation out there. It does not aim for full coverage!\n\nIt just helps me to keep an overview about topics from my daily systems tasks. How stuff is related to each other and what are the different building blocks.\n\n## Topics to cover\n\n### How to keep control about what is running under our responsibility?\n\n* debugging is a lot more complex \\(what are ...?\\)\n\n  * linux kernel\n    * network / fs namespaces\n    * routing of network packages within the linux kernel \\(how do namespaces affect the normal package flow?\\)\n    * cgroups\n    * network stages like nftables ebtables iptables?\n    * how does the vfs work with ceph and trim commands?\n    * debugfs to debug distributed storage?\n    * pitfalls of distributed storage solutions?\n\n  * kubernetes / container management\n    * where are pitfalls?\n    * how to fix ordinary problems here?\n    * what are usual issues while operating a cluster?\n\n  * docker\n    * which stages does docker touch, while running an instance? is there any, or does docker just some command delegation to the linux stack?\n\n  * why might different versions of kubernetes, linux kernel and docker be a problem? \\(try to define each domain and the overlapping interfaces / area of operation\\)\n  * if a network package gets lost, why and how can I find the root problem?\n\n### Basic principles\n\n* Do data / communication verification (security rules) on the smallest layer possible\n  * to make use of easy to understand principles\n  * to require a general thumb of rules\n  * to require to understand the real problem\n  * to enforce a good design (trying to apply a general security layer will test your concept)\n  * to enforce an as simple as possible solution (at least, its more probable to get a simple one)\n\n* What are indicators for probably healthy and promising OpenSource projects?\n* Principles in programming?\n* Principles in infrastructure design?\n* What about small scripts - are they useful? =\u003e yes they are, why?\n* What about huge monolithic projects - are they useful? =\u003e yes they are, why?\n\n### How may github and reading code of projects be integrated in my daily doing?\n\n* while debuging problems, it often results in code reading\n\n  * how might this get easier?\n  * can I automate it a little bit more?\n  * are there common questions, so I might be able to develop a common query interface?\n\n* common projects of interest\n\n  * gRPC \\(nodejs, python, php, go\\)\n  * kubernetes\n  * coreos/\\* projects\n  * prometheus\n  * nagios\n  * inluxdb/\\* projects\n  * nodejs\n  * golang\n  * python\n  * php\n\n### Which programming languages I'm required to be fluent in writing?\n\n\u003e reading programming languages should be considered as always required and quick to learn\n\nFor each language, listed here, I might gather core facts about the language and easy entry points to understand the language quickly. Also I might cover common tasks and concepts, specific for the language. A chapter about common concepts might be required too. Just to give an overview.\n\n* golang\n\n  * good for programs to distribute\n  * high performance\n  * resilient systems\n\n* python\n\n  * scripting / automating system related tasks\n  * writing simple system daemons\n\n* nodejs\n\n  * writing APIs\n\n* php\n\n  * a lot of web apps are written in PHP, so I need to know how it works and how I might make the apps work as I want them to work\n\n* polymer \\(polymer-project from google\\) \\(not a language, but a key component in building web portals\\)\n\n### Package manager, their strenghts and weaknesses\n\n* bower\n* npm\n* composer\n* pip\n\n### systemd\n\n* what it does\n* how does it replace previous solutions\n* good use cases\n* how does journald fit in relation with graylog / logstash\n\n### logging while it's cloudy\n\n* where is graylog positioned\n* how does it differ from logstash\n* how about fluentd?\n  * https://cloud.google.com/solutions/real-time/fluentd-bigquery\n  * http://www.fluentd.org/architecture\n\n### modern metric gathering\n\n* what role does prometheus play?\n* how is this related to influxdb?\n* why might monitoring just be analyzing datapoints?\n* how about TS \\(timeseries\\) DBs in general? (compairson [spreadsheet](https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/edit#gid=0))\n  * influxdb\n  * dalmatinerdb (https://dalmatiner.io/)\n  * riak ts (http://basho.com/products/riak-ts/ (why is https not default here?), [architecture overview](http://info.basho.com/rs/721-DGT-611/images/RiakTS-Enterprise-Technical-Overview.PDF))\n  * prometheus \\(which is also \\(at first\\) a monitoring system\\)\n\n### Containers\n\n* see [Containerization](/containerization.md)\n\n### LKVM \\(Native Linux KVM Tool\\) by kvmtool\n\n\u003e Native Linux KVM tool\n\u003e\n\u003e =====================\n\u003e\n\u003e kvmtool is a lightweight tool for hosting KVM guests. As a pure virtualization\n\u003e\n\u003e tool it only supports guests using the same architecture, though it supports\n\u003e\n\u003e running 32-bit guests on those 64-bit architectures that allow this.\n\n\\(from [here](https://kernel.googlesource.com/pub/scm/linux/kernel/git/will/kvmtool/+/3695adeb227813d96d9c41850703fb53a23845eb/README)\\)\n\nInstall kvmtool via \\`apt install kvmtool\\`\n\n### How distributed / scaled storage systems work with modern workload requirements\n\n* what about object stores like Ceph, Swift and the cloud alternatives?\n* what about scaling ASID compliant databases, how to scale them best?\n\n  * where are the pitfalls and how to solve common multi-master issues?\n\n* when not using traditional database systems, how to scale then?\n\n  * mongodb\n  * redis\n  * etcd\n  * confd\n  * memcached\n\n### What about backup and disaster recovery in a distributed, many replicated world?\n\n* ....\n\n### Systems automation\n\n* why should I use ansible\n  * in a traditional szenario \\(VMs/bare metal\\) it's more or less clear\n  * but which role does it have in a kubernetes, autoscaling container world?\n  * how might it help with CI/CD, or not help?\n\n* [gitlab CI](https://about.gitlab.com/gitlab-ci/)\n  * how does it work\n  * and why is it powerful or not powerful?\n  * how is it related to ansible? \\(CD\\)\n  * how is it different to [Quay from CoreOS](https://quay.io)\n\n* how to package software, or, do I need to build packages anymore?\n\n### How does Machine Learning affect my daily tasks?\n\nmay I use it for...\n\n* monitoring\n* continues deployments\n* continues integration\n* data analyzing\n* security concerns\n* dependency awareness for firewalling, package management, system dependencies? \\(like an alerting for complexity\\)\n* config analyzing to discover service relations and how a request might be resolved, just by analyzing code and data sets \\(config, database \n\n# The second part of this book: Cloud Services\n\nBuilding blocks of the Cloud\n\n* API services which benefit from a scaled customer base like machine learning services and big data\n\n## What does it mean, if we think about migrations into the Cloud - for my position?\n\n* how may I need to adapt my skill sets \\(back to basics, deep dive into development - extending my complexity skills\\)\n* how are on prem solutions affected?\n* automate everything results in? \\(more complexity for humans, easier access for programs, lower human interaction or better: consolidated workforce\\)\n  * we are currently in a time of transitions - computing gets serious\n\n## What might come in the next 2 to 3 years in Clouding?\n\n* machine learning prepares good / \"easy\" access on complex data for programs\n\n  * we don't need to specify relations, or not all of them\n  * after preparation, an operation system and its apps is just a bunch of relations \\(data structures \\[config syntax, databases and tables\\] and context information \\[config values, generated data\\], like human languages\\), so why should machine learning not cover maintenance and systems setups?\n\n* automate maintenance\n\n* automate code analysis \\(for bugs and required dependencies like RDBs, Storage, Computing, other software packages\\) and its deployment\n\n# The third part of this book: Software Engineering\n\n## Useful algorithm for...\n\n* Sorting / Filtering a huge amount of data via Map / Reduce?\n* ...\n\n## Useful data structures\n\n* which perform well, are \"easy\" search/query able\n* are lightweight (in case of storage space and complexity)\n\n## How do ASTs work?\n\n* Which data structures are in use?\n* How to query?\n* How encode them for machine learning? (like natural language processing)\n* How does it work with...\n  * PHP\n  * Python\n  * Golang\n  * JavaScript\n* \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonyfahrion%2Fhowsystemswork","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftonyfahrion%2Fhowsystemswork","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonyfahrion%2Fhowsystemswork/lists"}