{"id":13679724,"url":"https://github.com/guettli/programming-guidelines","last_synced_at":"2025-05-16T10:06:27.318Z","repository":{"id":38421698,"uuid":"64375759","full_name":"guettli/programming-guidelines","owner":"guettli","description":"My personal programming guidelines","archived":false,"fork":false,"pushed_at":"2024-11-08T22:20:02.000Z","size":3161,"stargazers_count":300,"open_issues_count":2,"forks_count":43,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-05-16T10:06:20.270Z","etag":null,"topics":["best-practices","guidelines","programming-languages","python"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guettli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-28T07:58:41.000Z","updated_at":"2025-02-28T03:02:54.000Z","dependencies_parsed_at":"2024-02-16T09:31:18.451Z","dependency_job_id":"bf5a7ee9-52bb-462e-990a-4ab563e545ef","html_url":"https://github.com/guettli/programming-guidelines","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guettli%2Fprogramming-guidelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guettli%2Fprogramming-guidelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guettli%2Fprogramming-guidelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guettli%2Fprogramming-guidelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guettli","download_url":"https://codeload.github.com/guettli/programming-guidelines/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254509475,"owners_count":22082891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["best-practices","guidelines","programming-languages","python"],"created_at":"2024-08-02T13:01:08.738Z","updated_at":"2025-05-16T10:06:27.287Z","avatar_url":"https://github.com/guettli.png","language":null,"readme":"# Programming Guidelines\n\nMy opinionated programming guidelines. \n\n[1. Introduction](#1-introduction)\n\n[2. Data structures](#2-data-structures)\n\n[3. Dev](#3-dev)\n\n[4. Remote APIs](#4-remote-apis)\n\n[5. Op](#5-op)\n\n[6. Networking](#6-networking)\n\n[7. Monitoring](#7-monitoring)\n\n[8. Communication with others](#8-communication-with-others)\n\n[9. Epilog](#9-epilog)\n\n## 1. Introduction\n\n### About this README\n\nI was born in 1976. I started coding with basic and assembler when I was\n13. Later turbo pascal. From 1996-2001 I studied computer science at\nHTW-Dresden (Germany). I learned Shell, Perl, Prolog, C, C++, Java, PHP, and finally Python.\n\nSometimes I see young and talented programmers wasting time. There are\ntwo ways to learn: Make mistakes yourself, or read from the mistakes\nwhich were done by other people.\n\nThis list summarises a lot of mistakes I did in the past. I wrote it, to\nhelp you, to avoid these mistakes.\n\nIt's my personal opinion and feeling. No facts, no single truth.\n\n### I need your feedback\n\nIf you have a general question, please start a [new discussion](https://github.com/guettli/programming-guidelines/discussions/new).\n\nIf you think something is wrong or missing, feel free to open an issue or pull request.\n\n### Relaxed focus on your monitor\n\nDo not look at the keyboard while you type. Have a relaxed focus on your\nmonitor.\n\nI type with ten fingers. It's like flying if you learned it. Your eyes\ncan stay on the rubbish you type, and you don't need to move your eyes\ndown (to keyboard) and up (to monitor) several hundred times per day.\nThis saves a lot of energy. This is a simple tool to help you to learn touch typing:\n[tipp10](https://www.tipp10.com/en/)\n\nMeasure your typing speed: [10fastfingers.com](//10fastfingers.com/)\n\nAvoid switching between mouse and keyboard too much.\n\nI like Lenovo keyboards with track point. If you want more grip, then\nread [Desktop Tips \"Keyboard\"](https://github.com/guettli/desktop-tips-and-tricks/blob/master/README.md#keyboard)\n\nOnce I was fascinated by the copy+paste history of Emacs and PyCharm.\nBut then I thought to myself: \"I want more. I am hungry. I want a\ncopy+paste history not only in one application, but I also want it for the whole\ndesktop\". The solution is very simple, but somehow only a few people use\nit. The solution is called a clipboard manager. I use [CopyQ](https://hluk.github.io/CopyQ/). I use ctrl+alt+v to open the list of last\ncopy+paste texts. CopyQ supports regex searches in the history. \n\n### Avoid searching with your eyes\n\nAvoid searching with your eyes. Search with the tools of your IDE. You\nshould be able to use it \"blind\". You should be able to move the cursor\nto the matching position in your code without looking at your keyboard,\nwithout grabbing your mouse/touchpad/TrackPoint and without looking\nup/down on your screen.\n\nCompare two files with a diff tool, otherwise, you might get this ugly skeptical frown.\n\nHow often per day do you search for the mouse cursor on your screen?\nSupport your eyes by increasing the cursor size. If you use Ubuntu,\nyou can do it via [Universal Access / Cursor Size](https://askubuntu.com/questions/1266951/increase-mouse-cursor-size-on-ubuntu-20-04/1266961#1266961)\n\n### Increase font size\n\nDuring daily work, you often jump from one information snippet to the next\ninformation snippet.\n\nWhen was the last time you read a text with more than 20 sentences?\n\nI think from time to time you should do so. Slow down, focus on one\ntext, and read slowly. It helps to increase the font-size. `ctrl-+` is\nyour friend.\n\n### KISS\n\nKeep it simple and stupid. The most boring and most obvious solution is\noften the best. Although it sometimes takes months until you know which\nsolution it is.\n\nFrom the book \"Site Reliability Engineering\" (O'Reilly Media 2016)\n\u003chttps://landing.google.com/sre/book/chapters/simplicity.html\u003e\n\nQuote:\n\n:   The Virtue of Boring\n\n    Unlike just about everything else in life, \"boring\" is a\n    positive attribute when it comes to software! We don’t want our programs to be spontaneous and interesting; we want them to stick to the script and predictably accomplish their business goals.\n\nExample: [Pure Functions](https://en.wikipedia.org/wiki/Pure_function) are great. They are stateless, their output can be cached forever, they are easy to test.\n\n### Increase the obviousness\n\nBut it is not only about code. It is about the experience of all stakeholders: Users, salespeople, support hotline, developers,...\n\nIt is hard work to keep it simple. \n\nOne thing I love to do: \"Increase the obviousness\". \n\nOne tool to get there: Use a central wiki (without spaces), and\ndefine terms. Related text from me: [Documentation in Intranets: My point of view](https://github.com/guettli/intranets)\n\n\n### Avoid redundancy\n\nSee heading.\n\n### Premature optimization is the root of all evil.\n\nThe famous quote \"premature optimization is the root of all evil.\" is true.\nYou can read more about this here [When to optimize](https://en.wikipedia.org/wiki/Program_optimization#When_to_optimize).\n\n### MVP\n\nYou should know what an [MVP (minimum valuable product)](https://en.wikipedia.org/wiki/Minimum_viable_product) is. Building an MVP means to bring something useable to your customer, and then listen to their feedback. Care for their needs, not for your vision of a super performant application.\n\nAvoid i18n in MVP. German is my mother tongue. If I develop a MVP for German users, than I won't to i18n. This can be done later, if needed.\n\n------------------------------------------------------------------------\n\n## 2. Data structures\n\n### Introduction\n\n\"Bad programmers worry about the code. Good programmers worry about data\nstructures and their relationships.\" -- Linus Torvalds (creator and\ndeveloper of the Linux kernel and the version control system git)\n\n### Cache vs Database\n\nThere is a fundamental fact which you need to understand: The difference between\na cache and a database.\n\nRemember the basic Input-Process-Output pattern.\n\nIn a cache you store data which is **output**. That's handy since you can access the output\nwithout doing the processing again. But cache-invalidation is hard. Maybe\nthe input has changed, and the value in the cache is outdated? Who knows?\nIf possible avoid caching, since this will never give you outdated data.\nYou don't need to backup your cache data. You can create it again.\n\nIn a database you store data which is **input**. Usually it was entered by a human\nby hand, or generated by measuring some real word data. You can use the data\nin database to create a nice HTML page. It is important to backup your valuable\ndatabase data, since you can't create it again. The generated output (HTML, JSON, ...)\nhas no value.\n\nData which is input usualy has value. Data which is output has only little value,\nsince you can re-create it again.\n\n### Relational Database\n\nI know SQL is..... It is either obvious or incomprehensible. And, yes, it is\nboring.\n\nA relational database is a rock-solid data storage. Use it.\n\nWhen I studied computer science, I disliked SQL. I thought it was an\noutdated solution. I tried to store data in files in XML format, used\nin memory Berkley-DB, I used an object-oriented database written in Python (ZODB),\nI used NoSQL .... And finally, I realized that boring SQL is the best solution\nfor most cases.\n\nI use PostgreSQL.\n\nI don't like NoSQL, except for caching (simple key-value DB).\n\nThe [PostgreSQL Documentation](https://www.postgresql.org/docs/current/index.html) contains\nan introduction to SQL and is easy to read.\n\nIf you want to share small SQL snippets, you can use https://dbfiddle.uk/\n\n### Cardinality\n\nIt does not matter how you work with your data (struct in C, classes in\nOOP, tables in SQL, ...). Cardinality is very important. Using 0..\\* is\noften easier to implement than 0..1. The first can be handled by a\nsimple loop. The second is often a nullable column/attribute. You need\nconditions (IFs) to handle nullable columns/attributes.\n\n\u003chttps://en.wikipedia.org/wiki/Cardinality_(data_modeling)\u003e\n\nIf this is new to you, I will give you two examples:\n\n-   1:N --\u0026gt; One invoice has several invoice positions. For example,\n    you buy three books in one order, the invoice will have three invoice positions. This is a 1:N relationship. The invoice position is contained in exactly one invoice.\n-   N:M --\u0026gt; If you look at tags, for example at the Question+Answer\n    site StackOverflow: One question can be related to several\n    tags/topics and of course a topic can be set on several questions.\n    For example, you have a strange UnicodeError in Python then you can set the tags \"python\" and \"unicode\" on your question. This is an N:M\n    relationship. One well know example of N:M is user and groups.\n\n### Conditionless Data Structures\n\nIf you have no conditions in your data structures, then the coding for\nthe input/output of your data will be much easier.\n\n### Avoid nullable Foreign Keys\n\nImagine you have a table \"meeting\" and a table \"place\". The table\n\"meeting\" has a ForeignKey to table \"place\". In the beginning, it might\nbe not clear where the meeting will be. Most developers will make\nthe ForeignKey optional (nullable). WAIT: This will create a condition\nin your data structure. There is a way easier solution: Create a place\ncalled \"unknown\". Use this [senitel value](https://en.wikipedia.org/wiki/Sentinel_value) as default. This data\nstructure (without a nullable ForeignKey) makes implementing the GUI\nmuch easier.\n\nIn other words: If there is no NULL in your data, then there will be\nless NullPointerException in your source code while processing the data\n:-)\n\nFewer conditions, fewer bugs.\n\n### Avoid nullable boolean columns\n\n\\[True, False, Unknown\\] is not a nullable Boolean Column.\n\nIf you want to store data in a SQL database that has three states\n(True, False, Unknown), then you might think a nullable boolean column\n(here \"my\\_column\") is the right choice. But I think it is not. Do you\nthink the SQL statement \"select \\* from my\\_table where my\\_column = %s\"\nworks? No, it won't work since \"select \\* from my\\_table where\nmy\\_column = NULL\" will never return a single line. If you don't\nbelieve me, read: [Effect of NULL in WHERE clauses\n(Wikipedia)](https://en.wikipedia.org/wiki/Null_(SQL)#Effect_of_Unknown_in_WHERE_clauses).\nIf you like typing, you can work-around this in your application, but I\nprefer straightforward solutions with only a few conditions.\n\nIf you want to store True, False, Unknown: Use text, integer, or a new\ntable and a foreign key.\n\n### Avoid nullable characters columns\n\nIf you allow NULL in a character column, then you have two ways to\nexpress \"empty\":\n\n-   NULL\n-   empty string\n\nAvoid it if possible. In most cases, you just need one variant of\n\"empty\". Simplest solution: avoid that a column holding character data is allowed to be null.\n\nIf you think the character column should be allowed to be NULL (for example you want a unique, but optional identifier for rows),\nthen consider a constraint: If the character string in the column is not\nNULL, then the string must not be empty. This way ensure that there are\nis only one variant of \"empty\".\n\n### SQL: I prefer subqueries to joins\n\nIn most cases, I use an ORM to access data and don't write SQL by hand.\n\nIf I do write SQL by hand, then I often prefer [SQL Subqueries](https://en.wikipedia.org/wiki/SQL_syntax#Subqueries)\nto SQL Joins. \n\nHave a look at this example:\n```\nSELECT id, name\nFROM products\nWHERE category_id IN\n   (SELECT id\n    FROM categories\n    WHERE expired = True)\n```\nI can translate this to human language easily: Select all products, which\nbelong to a category that has expired.\n\n\n### Use all features PostgreSQL does offer\n\nIf you want to store structured data, then PostgreSQL is a safe default\nchoice. It fits in most cases. Use all features PostgreSQL does offer.\nDon't constrain yourself to use only the portable SQL features. It's ok\nif your code does work only with PostgreSQL and no other database if\nthis will solve your current needs. If there is a need to support\nother databases in the future, then handle this problem in the future,\nnot today. PostgreSQL is great, and you waste time if you don't use its\nfeatures.\n\nImagine there is a Meta-Programming-Language META (AFAIK this does\nnot exist) and it is an official standard created by the ISO (like SQL).\nYou can compile this Meta-Programming-Language to Java, Python, C, and\nother languages. But this Meta-Programming-Language would only support\n70% of all features of the underlying programming languages. Would it\nmake sense to say \"My code must be portable, you must use META, you must\nnot use implementation-specific stuff!\"?. No, I think it would make no\nsense.\n\nMy conclusion: Use all features PostgreSQL has. Don't make your life more\ncomplicated than necessary and don't restrict yourself to use only\nportable SQL.\n\nGreat features PG has, which you might not know yet:\n\n* [Insert/Update/Delete Trigger](https://www.postgresql.org/docs/current/sql-createtrigger.html)\n* \"SELECT FOR UPDATE .... SKIP LOCKED\" gives you the perfect foundation for a task-queue. For example [Procrastinate](https://github.com/peopledoc/procrastinate)\n* [PGAdmin](https://www.pgadmin.org/) nice GUI to configure your databases.\n* [Fulltext Search](https://www.postgresql.org/docs/current/textsearch.html)\n\nThere is just one hint: Avoid storing binary data in PostgreSQL. An S3 \nservice like [minio](https://min.io/) is a better choice.\n\n### Where to not use PostgreSQL?\n\n-   For embedded systems SQLite may fit better\n    \\* Prefer SQLite if there will only be one process accessing the database at a time. As soon as there are multiple users/connections,\n    you need to consider going elsewhere\n-   TB-scale full-text search systems.\n-   Scientific number crunching:\n    [hdf5](https://en.wikipedia.org/wiki/Hierarchical_Data_Format)\n-   Caching: Redis fits better\n-   Go with the flow: If you are wearing the admin hat (instead of the\n    dev hat), and you should install (instead of developing) a product,\n    then try the default DB (sometimes MySQL) first.\n\nSource: PostgreSQL general mailing list:\n\u003chttps://www.postgresql.org/message-id/5ded060e-866e-6c70-1754-349767234bbd%40thomas-guettler.de\u003e\n\n### Transactions do not nest\n\nI love nested function calls and recursion. This way you can write easy\nto read code. For example recursion in quicksort is great.\n\nNested transactions ... sounds great. But stop: What is\n[ACID](https://en.wikipedia.org/wiki/ACID) about? This is about:\n\n-   Atomicity\n-   Consistency\n-   Isolation\n-   Durability\n\nDatabase transactions are atomic. If the transaction was successful,\nthen it is **D**urable.\n\nImagine you have one outer-transaction and two inner transactions.\n\n1.  Transaction OUTER starts\n2.  Transaction INNER1 starts\n3.  Transaction INNER1 commits\n4.  Transaction INNER2 starts\n5.  Transaction INNER2 raises an exception.\n\nIs the result of INNER1 durable or not?\n\nConclusion: Transactions do not nest\n\nRelated:\n\u003chttp://stackoverflow.com/questions/39719567/not-nesting-version-of-atomic-in-django\u003e\n\nThe \"partial transaction\" concept in PostgreSQL is called savepoints.\n\u003chttps://www.postgresql.org/docs/devel/sql-savepoint.html\u003e They capture\nlinear portions of a transaction's work. Your use of them may be able to\nexpress a hierarchical expression of updates that may be preserved or\nrolled back, but the concept in PostgreSQL is not itself hierarchical.\n\n### My customer wants to extend the data schema...\n\nImagine you created some kind of issue-tracking system. Up until now, you provide attributes like \"subject\", \"description\", \"datetime created\",\n\"datetime last-modified\", \"tags\", \"related issues\", \"priority\", ...\n\nNow the customer wants to add some new attributes to issues. It would be quite easy for you to update\nthe database schema and update the code.\n\nMaybe you are lucky and you have 100 customers. Then you would like to prefer to spend your time\nimproving the core product. You don't want to spent too much time on the features which\nonly one customer wants.\n\nOr the customer wants to update the schema on its own. \n\nWhat can you do now?\n\nOne solution is EAV: The [Entity–attribute–value model](https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model)\n\n### Why I don't want to work with MongoDB\n\n\u003e MongoDB is a cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with optional schemas. ([Wikipedia](https://en.wikipedia.org/wiki/MongoDB))\n\nOne document in a collection can differ in its structure. For example, most all documents in a collection have an integer value on the attribute \"foo\", but for unknown reasons, one document has a float instead of an integer. Grrr.\n\nWhat does the solution look like?\n\n```\nreturn try {\n    this.getLong(key)\n  } catch (e: ClassCastException) {\n    if (this[key] is Double) this.getDouble(key).toLong() else null\n  }\n```\n\nNo! I want a clear schema where all values in a column are of the same type.\n\nOf course, my wish has a draw-back: If you want to upgrade a table in a production relational database, you might have downtime, because the database needs some\nminutes to convert all rows to the new schema. But at least in my context, this was never a big problem up until now.\n\nRelated: [StackOverflow \"class java.lang.Double cannot be cast to class java.lang.Long\"](https://stackoverflow.com/questions/65141475/mongodb-class-java-lang-double-cannot-be-cast-to-class-java-lang-long)\n\n\n------------------------------------------------------------------------\n\n## X. UI\n\n### Mockups help\n\nStart with painting. A [Mockup](https://en.wikipedia.org/wiki/Mockup#Software_engineering) helps.\n\nIf you improve an existing application, then take a screenshot and then paint it with expressive colors. I like #ff00ff.\n\nIf you doing something from scratch, then create some slides paint it roughly, add numbers to buttons and add a little\ntext on what should happen if someone pushes the button. Again, use expressive colors, so that it easy to see what is ideation and\nwhat is existing GUI.\n\nYou don't need expensive tools like Figma or InVision for this. Especially if I create something new, I like to do it on paper with a pencil and crayons.\n\nOf course, the above hints make no sense if you write a device driver that has no graphical user interface.\n\n### Faceted search\n\nYou should know this term: [Faceted search](https://en.wikipedia.org/wiki/Faceted_search)\n\n### FTUE\n\n[First-time user experience](https://en.wikipedia.org/wiki/First-time_user_experience) is very important. Does a user who has never used the application before understanding it immediately? \n\n### Don't make me think\n\n[Don't make me think](https://en.wikipedia.org/wiki/Don%27t_Make_Me_Think) is the title of a book. I don't think it is necessary to read it. Just remember this title and try to create user interfaces that are easy to understand.\n\n### SEO\n\nI think the best docs about search engine optimization are from the company which creates the currently most popular internet search engine:\n\n[developers.google.com/search](https://developers.google.com/search)\n\n------------------------------------------------------------------------\n\n## 3. Dev\n\n### Input-Processing-Output\n\nThere are thousands of programming languages and thousands of ways to exchange data. But finally, it is one concept:\n\n[Input-Processing-Output](https://en.wikipedia.org/wiki/IPO_model)\n\nIf you tell your navigation system of your car \"Please show me the route to Casablance Pub, Leipzig\" or if you write your first program which adds two integers and prints the result.\n\n### Less code, fewer bugs\n\n-   Not existing code is the best: Less code, fewer bugs\n-   Code maintained by a reliable upstream (like Python, PostgreSQL,\n    Django, Linux, Node.js, Typescript, ...) is more reliable than my code.\n\n### Avoid low-level stuff\n\nFor me this means to avoid: Assembler, C, C++, Rust, golang ...\n\nThese tools are great if you want maximum performance.\n\nMy goal is to create something useful. Maybe I optimize later.\n\n### fewer resources, fewer bugs\n\nThere are several ways to give data to a method.\n\nLet's have a look at this simple method call: `my_method(some_string)`\n\nYou might think there is only of variable which gets accessed by the method?\n\nLet's find more ways this method could get input:\n\n* Environment variables: Maybe setting LANG=de_DE influences the output?\n* Filesystem: Maybe the existence or content of a file in the local file system influences the method.\n* `my_method()` could access a database, storage, or a cache to read additional data\n* Maybe there is a global variable that contains a value that was set by a previous call to `my_method()`\n* Maybe the date influences the method. Maybe the method creates a different output at a full moon.\n* ...\n\nAFAIK there is no clear name that distinguishes between explicit and implicit input.\n\nYou can't avoid implicit input, and it is 100% ok if it is obvious. If your method\nshould return the data of the user with the id 12345, then your code needs to access\nthe database.\n\nIf the same code works in one environment, but not in a different environment, and you don't know why\nthen this tool might help: [dumpenv](https://github.com/guettli/dumpenv) it writes the environment to\na list of files, which you can compare with your favorite diff tool (e.g. Meld).\n\n### Environment variables for configuration?\n\nEnvironment variables are great for providing applications/containers values for database connection strings, URL \nto a storage server ...\n\nAs soon as an environment variable is used in a condition like `if $FOO equals \"BAR\", then ... else ...`, then\nit is some kind of magic input.\n\nI prefer \"clear\" input: For a http request this means the GET/POST data. Using the http header is some kind of magic,\nand should be avoided.\n\nFor commands called via the command line it is the same: I prefer command line arguments instead of environment variables.\n\nImagine you have a typo in the environment variable name. A dirty shell script will use an empty string and it is likely that\nit will do something wrong. Compare this to a script: If you have a typo in the argument for the command, it will fail and tell\nyou that the given argument is unknown.\n\nA shell script might make you faster during the first 10 minutes. But it will make you slower in the long run. \n\nWriting a Python script with `argparse` takes longer, but will provide you much more reliablity.\n\nI know [12factor App](https://12factor.net/):\n\n\u003e III. Config\n\u003e\n\u003e Store config in the environment\n\nI agree with connection URLs and passwords/keys/tokens which connect the app to the environment. But if the\nconfiguration influences the behaviour, then I think traditional configuration or configuration stored in a database\nmakes more sense.\n\nFor connection-URL and passwords the data type is easy: It is a string.\n\nBut you configuration needs booleans or other data types, then environment variables are not well suited.\n\n\n### Zen of Python\n\n[Zen of Python](https://www.python.org/dev/peps/pep-0020/) (Written by\nTim Peters in the year 1999)\n\n-   Beautiful is better than ugly.\n-   Explicit is better than implicit.\n-   Simple is better than complex.\n-   Complex is better than complicated.\n-   Flat is better than nested.\n-   Sparse is better than dense.\n-   Readability counts.\n-   Special cases aren't special enough to break the rules.\n-   Although practicality beats purity.\n-   Errors should never pass silently.\n-   Unless explicitly silenced.\n-   In the face of ambiguity, refuse the temptation to guess.\n-   There should be one-- and preferably only one --obvious way to do it.\n-   Although that way may not be obvious at first unless you're Dutch.\n-   Now is better than never.\n-   Although never is often better than *right* now.\n-   If the implementation is hard to explain, it's a bad idea.\n-   If the implementation is easy to explain, it may be a good idea.\n-   Namespaces are one honking great idea -- let's do more of those!\n\nIn the year 2001, I knew these programming languages: Basic, Pascal,\nAssembler, C, C++, Prolog, Lisp, Visual Basic, Java, JavaScript, Tcl/Tk,\nPerl.\n\nI was unhappy with all of them and looked for a new language. I narrowed\ndown the languages, I was interested in and there were two choices left.\nOne was ruby, the other was Python. I choose Python. It looked simpler,\nlike executable pseudo-code. Since 2001 I use it nearly every work-day.\nI like it, and till now, no other language attracts me.\n\nI am not married to Python. I am willing to change. But the next\nlanguage needs to be better. Up until now, I see no alternative.\n\nJavaScript has a big benefit, that it can be executed in the browser.\nBut I don't like it. Why I don't like it? I don't know. Sometimes\nfeelings are more important than facts.\n\n### CRUD --\u0026gt; CRD\n\nIn most cases, the software does create, read, update, delete data. See\n[CRUD](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete)\n\nThe \"update\" part is the most difficult one.\n\nSometimes CRD helps: Do not implement the update operation. Use\ndelete+create. But be sure to use transactions to avoid data loss, if your\ndata storage supports this:\n\"BEGIN; DELETE ...; INSERT ...; COMMIT;\"\n\nTranslating to SQL terms:\n\n|  CRUD Term   |SQL                                |\n|  ------------|-----------------------------------|\n|  create      |insert into my\\_table values (...) |\n|  read        |select ... from my\\_table          |\n|  update      |update my\\_table set col1=...      |\n|  delete      |delete from my\\_table where ...    |\n\nTake a look at virtualization and containers ([Operating-system-level\nvirtualization](https://en.wikipedia.org/wiki/Operating-system-level_virtualization)).\nThere CRD gets used, not CRUD. Containers get created, then they\nexecute, then they get deleted. You might use configuration management\nto set up a container. But this gets done exactly once. There is one\nupdate from the vanilla container to your custom container. But this is like\n\"create\". No updates will follow once the container was created. This\nmakes it easier and more predictable.\n\nThe same is true for operating on data-structures in memory. In most cases, you should not alter the data structure which is iterating. Create a new data structure while iterating the input data. In other words: no in-place editing.\n\n### Stateless\n\nWhen I was a student I was excited and fascinated by [CORBA (Common Object Request Broker Architecture)](https://en.wikipedia.org/wiki/Common_Object_Request_Broker_Architecture). I thought this is the future of machine to machine communication. Today I smile about how childish I was 19 years ago. CORBA is dead, stateless [http](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) has won.\n\nThings are much easier to implement and predict if you just have one method call. One request and one response. You don't have an open connection and a reference to a remote object which executes on a remote server.\n\nLook at all the dated protocols which are like a human conversation between a client and a server: SMTP, IMAP, FTP, ... Nobody wants the client and the server to have a chatty dialog like this: \n\n```\nClient: My name is Bob\nServer: Hi Bob, nice to meet you.\nServer: But are you really Bob?\nServer: Please prove to me that you're Bob. You can use method foo, bar, blu for authentication\nClient: I choose method \"blu\"\nServer: Ok, then please tell send the magic blu token\nClient: Here it is xyuasdusd8... I hope you like it.\nServer: Fine, I accept this. Now I trust you. Now I know you are Bob\nClient: Please show me the first message\nServer: here it is:\nServer:...\nClient: looks like spam. Please delete this message\nServer: Now I know that you want to delete this message. \nServer: But won't delete it now. Please send me EXPUNGE to execute the delete.\nClient: grrrr, this is complicated. I already told you that I want the message to be deleted.\nClient: EXPUNGE\n...\n```\n\nOf course roughly the same needs to be done with HTTP. But HTTP you can cut the task into several smaller HTTP requests. This gives the service the chance of delegating request-1 to server-a and request-2 to server-b. In the cloud, environment containers get created and destroyed in seconds. It is easier without a long-living connection.\n\nIn the above case (IMAP protocol) the EXPUNGE is like a COMMIT in relational databases. It is very handy to have a transactional database to implement a service. But it makes no sense to expose the transaction to the client.\n\nStateless is like IPO: Input-Processing-Output.\n\n\n\n### No Shell Scripting\n\nThe shell is nice for interactive usage. But shell scripts are\nunreliable: Most scripts fail if filenames contain whitespaces.\nShell-Gurus know how to work around this. But quoting can get complicated. I use the shell for interactive stuff daily. But I stopped\nwriting shell scripts.\n\nReasons:\n\n- If an error happens in a shell script, the interpreter steps silently to the next line. Yes, I know you can use \"set -e\". But you don't get a stack trace. Without a stack trace, you waste a lot of time analyzing why this error happened.\n- It makes sense to use (or run) an application monitoring platform. For example \"Shell\" is not a [supported plattform of Sentry](https://docs.sentry.io/platforms/). If you configure it for your prefered environment once, then you get great error reporting in once place. Even if your small backup-script is only a three lines long shell script: It is unreliable, use a real language!\n- Shell-Scripts tend to call a lot of subprocesses. Every call to grep, head, tail, cut creates a new process. This tends to get slow.\n    I have seen shell scripts that start thousands of processes per second.\n    After re-writing them in Python they were 100 times faster and 100\n    times more readable.\n- I do this `find ... | xargs` daily, but only while using the shell interactively. But what happens if a filename contains a space character? Yes, I know `find ... -print0 | xargs -r0`. BTW, I switched from find+xargs to [rg](https://github.com/BurntSushi/ripgrep) for most cases.\n- Look at all the pitfalls: [Bash\n    Pitfalls](https://mywiki.wooledge.org/BashPitfalls)\n- Even Crontab lines are dangerous. Look at this cron-job which should clean the directory of the temporary files:\n\n\u003e @weekly . ~/.bashrc \u0026\u0026 find $TMPDIR -mindepth 1 -maxdepth 1 -mtime +1 -print0 | xargs -r0 rm -rf\n\nDo you spot the big risk?\n\nShell scripts are fine if they are conditionless. This means no \"if\", no \"else\", no \"for\".\nFor example in a Dockerfile you can use \"RUN ....\" commands to create a custom image. But I would not call things like this a shell script. \nIt is just a sequence of commands to execute.\n\n\n### Portable Shell Scripts\n\nI think writing portable shell scripts and avoiding bashism (shell\nscripts that use features that are only available in the bash) is a\nuseless goal. It is wasting time. It feels productive, but it is not.\n\nAvoid `#!/bin/sh`. The interpreter could be bash, dash, busybox, or something else.\nSee [Comparison of command\nshells](https://en.wikipedia.org/wiki/Comparison_of_command_shells).\nPlease be explicit. Use `#!/bin/your-favorite-shell`.\n\nIf I look at this page\n([DashAsBinSh](https://wiki.ubuntu.com/DashAsBinSh)), which explains how\nto port shell scripts to /bin/dash I would like to laugh, but I can't\nbecause I think it is sad that young and talented people waste their\nprecious time which this nonsense. Since systemd gets used, the shell\ngets started less often (compared to the old system-V or BSD init). This\narchitectural change brought improvement. And I think that using dash\ninstead of bash brings no measurable benefit today. If you want it\nminimal, then use Alpine Linux with Busybox.\n\nIf you are not able to create a dependency to bash, then solve this\nissue. Use rpm/dpkg or configuration management to handle \"my script\nfoo.sh needs bash\".\n\nI know that there are some edge cases where the bash is not available,\nbut in most cases, the time to get things done is far more important.\nExecution performance is not that important. First: get it done\nincluding automated tests.\n\n### Server without a shell is possible\n\nIn the past, it was unbelievable: A Unix/Linux server that does not\nexecute a shell while doing its daily work. The dream is true today.\nThese steps do not need a shell: operating system boots. Systemd starts.\nSystemd spawn daemons. For example a web server. The web server spawns\nworker processes. An HTTP request comes in and the worker process handles\none web request after the other. In the past, the boot process and the\nstart/stop scripts were shell scripts. I am very happy that systemd\nexists.\n\nBut time has changed. Today applications run in containers. Containers\ndon't need systemd. In [Kubernetes](https://en.wikipedia.org/wiki/Kubernetes) containers\nget started and stopped, not services. There is no need for a daemon starting and\nstopping services since this gets done on a higher level.\n\n### Avoid calling command-line tools\n\nI try to avoid calling a command-line tool if a library is available.\n\nExample: You want to know how long a process is running (with Python).\nYes, you could call `ps -p YOUR_PID -o lstart=` with the subprocess\nlibrary. This works.\n\nBut why not use a library like\n[psutil](https://pypi.python.org/pypi/psutil)?\n\nWhy do you want to avoid a third-party library?\n\nIs there a feeling like \"too much work, too complicated\"? Installing a\nlibrary is easy, do it.\n\nCheck the license of the library. If it is BSD, MIT, LGPL, or Apache like, then\nuse the library.\n\nCalling a subprocess is slow, especially if it gets done often you will notice\nthe difference soon.\n\nThat's one reason I dislike shell scripting. Calling `grep`, `cut`, `sed` again and\nagain wastes a lot of CPU time. You can see this with the command line tool `top`.\nIf the `sy` value is high, then your server is busy starting new processes. A library is\nway more efficient, since you don't start new processes again and again.\n\n### Shell Scripts are ok, if ...\n\nShell Scipts are ok, if they are almost aconditionless: Few \"if\", \"else\" or \"for\".\n\nI use this heading, to ensure that the script is using Bash and stops if something is wrong (aka \"Bash strict mode\"):\n\n```\n#!/bin/bash\ntrap 'echo \"Warning: A command has failed. Exiting the script. Line was ($0:$LINENO): $(sed -n \"${LINENO}p\" \"$0\")\"; exit 3' ERR\nset -Eeuo pipefail\n\n...\n```\nAnd:\n\n* you should check your script in CI with [shellcheck](https://github.com/koalaman/shellcheck).\n* Use an IDE plugin which uses shellcheck.\n* For formatting you can use [shell-format](https://marketplace.visualstudio.com/items?itemName=foxundermoon.shell-format) based on [shfmt](https://github.com/mvdan/sh)\n\nI elaborated that here [Bash Strict Mode](https://github.com/guettli/bash-strict-mode)\n\n### Avoid toilet paper programming (wrapping)\n\nWhat is \"toilet paper programming\"? This is a pattern which was often\nused in the past: There is something wrong inside - something is\nsmelling. Let's write a wrapper. Still something wrong? Let's write a\nsecond wrapper.....\n\nAll these wrappers do not solve the underlying issue.\n\nIn the past, there were fewer alternatives. And since you had no choices,\nyou were forced to use a particular tool. If this did not work the way\nyou wanted it, you need to write a wrapper.\n\nToday you have many more alternatives. If tool x does not work\nthe way you want it to, you can use tool y.\n\nI am happy that the anti-pattern \"toilet paper programming\" gets used\nless often today.\n\nExample: WxPython (GUI toolkit) wraps WxWindows wraps gtk wraps xlib.\n\nThere are still some places where toilet paper wrappers need to get coded again and again.\n\nFor example, JSON does not support datetime, timedelta, and binary data. See [Let's fix JS](https://github.com/guettli/lets-fix-js). Speak to the upstream, to whoever is responsible for this, even if you think they are way too big, and you are way too small.\n\n\n### If unsure use MIT License\n\nThe [MIT License](https://en.wikipedia.org/wiki/MIT_License) is simple and short. Most projects\nat Github use it.\n\nSome licenses are much too long. I tried to read the GPL twice, but I fell\nasleep. I don't like things that I don't understand.\n\nNext argument: The GPL and AGPL licenses are [viral](https://en.wikipedia.org/wiki/Viral_license). If you want\nto create a commercial product, you can't use this.\n\nFor me \"freedom\" means no constraints. That's why I prefer\nthe MIT License, since GPL and AGPL have the constraint\nthat you must open your source, too.\n\nSee [Code licensed under AGPL MUST NOT be used at Google](https://opensource.google/docs/using/agpl-policy/)\n\n\n\n### Loop in DB, not in your code\n\nDo the filtering in the database. In most cases, it is faster than the\nloops in your programming language. And if the DB is not fast enough,\nthen I guess there is just the matching index missing up until now.\n\n### Do permission checking via SQL\n\nImagine you have three models (users, groups, and permissions) as tables\nin a relational database system.\n\nMost systems do the permission checking via source code. Example: `if\nuser.is_admin then return True`. \n\nSooner or later you need the list of items: Show all items which the\ncurrent user may see.\n\nNow you write SQL (or use your ORM) to create a queryset that returns\nall items which satisfy the needed conditions.\n\nNow you have two implementations. The first `if user.is_admin then\nreturn True` and one which uses set operations (SQL). This is redundant and looking\nfor trouble. Sooner or later your permission checks get more complex and then one implementation will get out of sync.\n\nThat's why I think: do permission checking via SQL\n\nSome call this \"Authorization predicate push-down\"\n\n### Real men use ORM\n\n[ORM (Object-relational mapping)](https://en.wikipedia.org/wiki/Object-relational_mapping) makes daily\nwork much easier. The above heading is a stupid joke. Clever people use tools to make work simpler, more fun, and more\nconvenient. ORMs are great. \n\nSome (usually elderly) developers fear that an ORM is slower than hand-crafted and optimized SQL. Maybe\nthere are corner cases where this prejudice is true. But that's not a reason to avoid ORMs. Just use them,\nand if you hit a corner case, then use raw SQL.\n\nSee [premature optimization is the root of all evil](#premature-optimization-is-the-root-of-all-evil)\n\nMake your life easy, use ORM.\n\nExample: [Django ORM \"Filtering on a Subquery() or Exists() expressions\"](https://docs.djangoproject.com/en/dev/ref/models/expressions/#filtering-on-a-subquery-or-exists-expressions). \n\n```\n# Select all rows of the model Post, which have a comment which was created a day ago:\n\none_day_ago = timezone.now() - timedelta(days=1)\nrecent_comments = Comment.objects.filter(\n     post=OuterRef('pk'),\n     created_at__gte=one_day_ago,\n)\n\nPost.objects.filter(Exists(recent_comments))\n```\nFor me above code is super easy to read.\n\n### SQL is an API\n\nIf you have a database-driven application and a third party tool wants\nto send data to the application, then sometimes the easiest solution is\nto give the third party access to the database.\n\nYou can create a special database user that has only access to one table.\nThat's easy.\n\nNitpickers will disagree: If the database schema changes, then the\ncommunication between both systems will break. Of course, that's true.\nBut in most cases, this will be the same if you use a \"real\" API. If\nthere is a change to the data structure, then the API needs to be\nchanged, too.\n\nI don't say that SQL is always the best solution. Of course, HTTP based\nAPIs are often better for services which get consumed by third paries.\n\nBut for internal services PostgreSQL with a custom role (only access to one table, and\nonly allowed to do INSERT) works fine. You can use NOTIFY, so that you can handle the inserted data\nimmediately.\n\nFor professional internal services you can use [nats.io](https://nats.io/). But using NATS for small\nproject makes no sense.\n\n### C is slow\n\n... looking at the time you need to get things implemented. Yes, the\nexecution is fast, but the time to get the problem done takes \"ages\". I\navoid C programming, if possible. If Python gets too slow, I can optimize\nthe hotspots. But do this later. Don't start with the second step. First,\nget it done and write tests. Then clean up the code (simplify it). Then\n.... What is the next step? Optimize? In most cases, the customer has new\nneeds and he likely wants new features not faster execution.\n\nHigher-level languages have a better \"zero to [MVP](https://en.wikipedia.org/wiki/Minimum_viable_product)\" speed.\n\n### Three time dimensions\n\nI think in software development there are three dimension of \"time\".\n\nMost developer immediatley think about \"execution time\": How fast is the code? How can I make the code even faster?\n\nBut there are:\n\nTime for \"From wish to wow\": How long does it take to implement and deploy a feature, so that the customer is happy?\n\nTime for \"From ? to Aha!\": How fast can an other developer understand your code.\n\nI think in most cases the proprity is like this: First \"From wish to wow\", then \"From ? to Aha!\", then \"execution time\".\n\nOf course this depends on your context. If you developing on PostgreSQL-core, Python-core, Kubernetes-core or Linux-kernel then\nexecution time is very important.\n\nBut mere mortals do application development.\n\nOf course the application should have a good performance.\n\nBut my hint is to optimize the performance of the application by using statistical profiling\nof the production system. But just looking at the code and guessing how to optimize performance \nwon't help, if you have not measured the performance of the production system.\n\n### Version Control: git\n\nFor version control of software, I use git. I think all other tools (svn,\nmercurial, CVS, darcs, bazaar) can be considered \"dead\". See\n[StackOverflow TagTrend](http://sotagtrends.com/?tags=git+svn+mercurial+cvs+darcs+bazaar)\n\nThe only exception to the rule \"use git\" is Google. They use their [own gigantic monorepo](https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext). \n\n\n### Avoid long-living branches\n\nAvoid long-living branches in your git repos. The more time that passes,\nthe less likely is that your work will ever get merged. For me one week\nis ok, but three weeks are too long.\n\nTen lines of improvement that get pushed to main today have much more value\nthan 1000 lines which are in a branch which will never get pushed to main.\n\nTrunk based development goes further. Sounds good:\n\n\u003e ... each developer divides their work into small batches and merges that \n\u003e work into the trunk at least once (and potentially several times) a day.\n\nSee [Google DevOp Guide \"Trunk based development\"](https://cloud.google.com/solutions/devops/devops-tech-trunk-based-development)\n\n\n### Don't put generated code into version control\n\nPlease read [Source code vs generated code](#source-code-vs-generated-code). Generated code or binary\ndata should not be in a git repository. It is possible but strange.\n\n### The best commits remove code\n\nFor me, the best commits add some lines to the docs, add some lines to\ntests and removes more lines than it adds to the production code.\n\n### Time is too short to run all tests before commit+push\n\nIf the guideline of your team is: \"Run all tests before commit+push\",\nthen there is something wrong. Time is too short to watch tests running!\nRun only the tests of the code you touched `py.test -k my_keyword`.\n\nIt's the job of automated CI (Continuous Integration) to run all tests.\nThat's not your job.\n\n### Time is too short to care for \"E302 expected 2 blank lines, found 1\"\n\nStyle Guide Enforcements (like [flake8 for Python](https://flake8.pycqa.org/en/latest/)) don't help much.\n\nTime is too short to manually make the style guide checker happy by \nediting the source code.\n\n\u003e  E302 expected 2 blank lines, found 1\n\nI don't want to waste my time with \"errors\" like above. This is no error.\nThe code is great and makes the customer happy.\n\nReading the message, understanding it, opening the file, editing it, re-runing\nthe checker .... No, this is not productive.\n\nThe solution is (like almost always) **automation**\n\nStyle guide enforcement does not help.\n\nAutomated source code styling helps.\n\nUnfortuantely this is not solved yet.\n\nFor the Python there is [black](https://github.com/psf/black), but it is not ready yet.\n\n### CI\n\nUse continuous integration. Only tested code is allowed to get deployed.\nThis needs to be automated. Humans make more errors than automated\nprocesses.\n\nGithub Actions are great.\n\nIncreasing the version number can be done with [BumpVer](https://pypi.org/project/bumpver/) which\ncan use [Calendar Versioning](https://calver.org/) (for example YYYY.MM.X)\n\n\nAll I need to do is to commit. All other steps are automated :-)\n\n### Tests should work offline\n\nImagine a developer sits on a train and has an unreliable network connection.\n\nNevertheless, I want that all tests can get executed.\n\nFor simple unit-tests that don't need a server, this is easy.\n\nBut if your test needs an HTTP-server, a database (PostgreSQL, MySQL),\na key-value DB (Redis), ... What can you do?\n\nAutomation is the solution. You can use a tool like Ansible to set up\nthe needed environment.\n\n\n### CI Config\n\nCI tools (GitLab, Travis, Jenkins) usually have a web GUI. Keep the\nthings you configure with the GUI simple. Yes, modern ci tools can do a\nlot. With every new version, they get even more [turing complete](https://en.wikipedia.org/wiki/Turing_completeness) (this was\na joke, I hope you understood it). Please do separation of concerns. The\nCI tool is the GUI to start a job. Then the jobs run, and then you can\nsee the result of the job in your browser. If you do configure condition\nhandling \"if ... then ... else ...\" inside the web-GUI, then I think you\nare on the wrong track.\n\nThe ci tool calls a command line. To make it easy for debugging and\ndevelopment this job should be callable via the command line, too. In\nother words: the web GUI gets used to collect the arguments. Then a\ncommand-line script gets called. Then the web GUI displays the result\nfor you. I think it is wise to avoid a complex CI config. If you want to\nswitch to a different ci tool (example from Jenkins to GitLab), then\nthis is easy if your logic is in scripts and not in ci tool\nconfiguration.\n\n### Avoid Threads, Async and Promises\n\nThreads and Async are fascinating. BUT: It's hard to debug. You will\nneed much longer than you initially estimated. Avoid it, if you want to\nget things done. It's different in your spare time: Do what you want and\nwhat is fascinating for you.\n\nThere is one tool and one concept that is rock solid, well known, easy\nto debug, and available everywhere and it is great for parallel\nexecution. The tool is called \"operating system\" and the concept is\ncalled \"process\". Why re-invent it? Do you think starting a new process is\n\"expensive\" (\"it is too slow\")? Just, do not start a new process for\nevery small method you want to call in parallel. Use a [Task\nQueue](https://www.fullstackpython.com/task-queues.html). Let this tool\nhandle the complicated async stuff and keep your code simple like\nrunning in one process with one thread. It is all about IPO:\nInput-Processing-Output.\n\nThere is a good reason to use async: The [C10k\nProblem](https://en.wikipedia.org/wiki/C10k_problem). BUT: I guess you\ndon't have this problem. If you don't have this problem, then don't use\ntechnology which was invented to solve this issue :-)\n\nThe related part of the [Google Codereview Guidelines \"Functionality\"](https://google.github.io/eng-practices/review/reviewer/looking-for.html#functionality)\n\nThere is a huge difference between implementing a task-queue and using\na task-queue. If you implement a task-queue, then threads/async/promises/multiprocessing are\nthe building blocks. But taks-queues exist. There is no need to re-invent them.\n\nI like to use task-queues, and write my code in a very predictable single-thread,\nsingle-process synchronous way.\n\n\n[Hick's Law](https://en.wikipedia.org/wiki/Hick%27s_law):\n\n\u003e increasing the number of choices will increase the decision time logarithmically.\n \nEverytime I need to deal with async or task-queues (like celery or rq) my output decreases.\nThere are so many ways to handle parallelism. Now you could argue: \"Thomas, parallelism is not the problem. The problem is that you are too stupid.\"\nMaybe this is correct. Maybe I am too stupid (or not familiar with this topic). I guess I am just an medicore developer.\nMy experience is that the environment should be optimized for medicore (normal) people. This will provide the best result.\nThus my rule of thumb: keep it simple and try to avoid Threads, Async and all this parallel computing.\n\n\n### Functions should return values, Not Promises.\n\nEspecially in JavaScript, functions often return [Promises](https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/Promise).\n\nThe `Promise` represents the eventual completion (or failure) of an asynchronous operation and its resulting value.\n\nI don't like this. I want a method to execute synchronously and then return the result.\n\nIf I want a method to be executed asynchronously, then I (the caller) can use a Promise. But I don't want the function to decide \"async or sync?\".\n\nI want to decide this, and I want the default to be \"synchronous execution\".\n\nPseudo Code (synchronous):\n```\nresponse = fetch('https://example.com')\nmy_json = response.json()\n```\n\nJavaScript (asynchronously)\n```\nconst my_json = async () =\u003e {\n    const response = await fetch('https://example.com');\n    return response.json();\n}\n```\n\nThe second code snippet is way more complicated. \n\nI think this can be compared to hyperlinks on web pages. The default is to follow the\nhyperlink (synchronous). If the user wants to open the hyperlink in a new tab (asynchronous), \nthen this decision should\nbe done by the user, not by the one who created this hyperlink.\n\nI have seen JavaScript code where almost every line contained `await`. That's childish.\n\n### Don't waste time doing it \"generic and reusable\" if you don't need to\n\nIf you are doing some kind of software project for the first time, then\nfocus on getting it done. Don't waste time to do it perfectly, reusable,\nfast, or portable. You don't know the needs of the future today. One main\ngoal: Try to make your code easy to understand without comments and make\nthe customer happy. First, get the basics working, then tests and CI,\nthen listen to the new needs, wishes, and dreams of your customers.\n\nExample: If you are developing web or server applications, don't waste\ntime making your code working on Linux and MS-Windows. Focus on one\nplatform.\n\nSee [Minimum viable\nproduct](https://en.wikipedia.org/wiki/Minimum_viable_product)\n\nRelated Book: [The Lean Startup](http://theleanstartup.com/book)\n\nSeveral months after writing the above text I found this \n\n[Google Codereview Guidelines \"Complexity\"](https://google.github.io/eng-practices/review/reviewer/looking-for.html#complexity)\n\u003e A particular type of complexity is over-engineering, where developers have made the code more generic than it needs to be, or added functionality that isn’t presently needed by the system. Reviewers should be especially vigilant about over-engineering. Encourage developers to solve the problem they know needs to be solved now, not the problem that the developer speculates might need to be solved in the future. The future problem should be solved once it arrives and you can see its actual shape and requirements in the physical universe.\n\nRelated: [YAGNI (You aren't gonna need it)](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it)\n\n### Use a modern IDE\n\nTime for vi and emacs has passed. Use a modern IDE on modern hardware\n(SSD disk). For example PyCharm. I switched from Emacs to PyCharm in\n2016. I used Emacs from 1997 until 2015 (18 years).\n\n### Easy to read code: Use guard clauses (early return)\n\nGuard clauses (early return) help to avoid indentation. It makes code\neasier to read and understand. See\n\u003chttp://programmers.stackexchange.com/a/101043/129077\u003e\n\nExample:\n\n    # Code with unnecessary complexity\n    \n    def my_method(my_model_instance):\n        if my_model_instance.is_active:\n            if my_model_instance.number \u003e MyModel.MAX_NUMBER:\n                if my_model_instance.foo:\n                    ....\n                    ....\n                    ....\n                    ....\n                    ....\n\nBetter:\n\n    # Less complex because less indentation\n    \n    def my_method(my_model_instance):\n        if not my_model_instance.is_active:\n            return\n        if not my_model_instance.number \u003e MyModel.MAX_NUMBER:\n            return\n        if not my_model_instance.foo:\n            return\n        ....\n        ....\n        ....\n        ....\n        ....\n\nLook at the actual code which does something. I used five lines with\n.... points for it. I think more indentation, makes the code more\ncomplex. The \"return\" simplifies the code. For me, the second version is\nmuch easier to read.\n\nPlease tell me, if you know a tool which can detect and maybe fix missing early returns for Python\ncode.\n\nFor Python there exists a \"complexity checker\": [radon](https://pypi.org/project/radon/), but AFAIK\nit can't be used to detect missing early-returns.\n\n### Source code vs generated code\n\nI guess every young programmer wants to write a tool that automatically\ncreates source code. Stop! Please think about it again. What do you\ngain? Don't confuse data and code. Imagine you have a source code\ngenerator that takes DATA as input and creates SOURCE as output. What\nis the difference between the input (DATA) and the output (SOURCE)? What\ndo you gain? Even if you have some kind of artificial intelligence, you\ncan't create new information if your only input is DATA. It is just a\ndifferent syntax. Why not write a program which reads DATA and does the\nthing you want to do?\n\nFor the current context, I see only two different things: **source code**\nfor humans and **generated code** for the machine.\n\nJust because a file contains code of a programming language, this does\nnot means that this file is source code.\n\nIf the TypeScript compiler creates JavaScript, then the output is\ngenerated code since the created JavaScript source is intended for the\ninterpreter only. Not for humans. If you create JavaScript with a\nkeyboard and a text editor it is source code. Don't mix source code and\ngenerated code in one file.\n\nIn other words: source code gets created by humans with the help of an\neditor or IDE.\n\n### Don't believe the \"automatically create foo\" hype\n\nIf you are new to software development you are fascinated by the magic.\nYou can create things! In this section, I call the magic output \"foo\".\n\nYes, you can automatically create foo with a script. Whatever \"foo\" is\nin your context: It has no value. It is worth nothing. It is dust in the wind like a web page that displays the current time. \nThis output is only temporarily valuable. \n\nLook at the basic IPO pattern: Input - Processing - Output (in this case\n\"foo\").\n\nDo not store \"foo\", the output of your script, in a database. Do not\nstore \"foo\" in version control.\n\nIt has no value since you can always create \"foo\" again. You just need\nthe input and your script.\n\nYou can store \"foo\" in a cache to improve performance. But do not store\nit permanently. Don't make a backup of it.\n\nDon't store automatically created data in your database. Instead re-calculate the data again\nand again. Maybe a\n[Materialized View (PostgreSQL)](https://www.postgresql.org/docs/current/rules-materializedviews.html) helps\nyou do improve speed.\n\nA term that is often a hint to this anti-pattern is \"generator\". Yes,\nyou can generate a lot of data. But this bloated, generated data is just hot air with\nlittle value.\n\nDevOps who prefer \"Op\" to \"Dev\" tend to create a configuration with a script.\nYou can do this but then create the config again daily. Do not edit\nthe generated config by hand.\n\nRelated: [Single source of truth](https://en.wikipedia.org/wiki/Single_source_of_truth)\n\n### Regex are great - But it's like eating rubbish\n\nYes, I like a regular expression. But slow down: What do I do, if I use a\nregex? I think it is \"parsing\". I remember to have read this some time\nago: \"Time is too short to rewrite parsers\". Don't parse data! We live\nin the 21 century. Consume high-level data structures like JSON, YAML, or\nprotocol buffers. If possible, refuse to accept CSV or custom text format\nas input data.\n\nFrom time to time you need to do text processing. Unfortunately, there\nare several regex flavors. My guide-line: Use PCRE. They are available\nin Python, Postfix, in `grep -P` and many other tools. Don't waste time with other\nregex flavors, if PCRE is available.\n\nCurrent Linux distributions ship with a grep version which has the -P\noption to enable PCRE. AFAIK this is the only way to grep for special\ncharacters like the binary null: [How to grep for special\ncharacter](https://superuser.com/a/612336/95878)\n\n### Use a password manager\n\nI use keepass. And sync it via Nextcloud.\n\nDon't forget to add the content of your ~/.ssh/id_rsa file to it. \n\n### CSV - Comma-separated values\n\nCSV is not a data format. It is an illness. See the introduction at:\n\u003chttps://docs.python.org/3/library/csv.html\u003e\n\nIf your customer sends you tabular data in Excel, read the excel\ndirectly. Do not convert it to CSV just because you think this is\neasier.\n\nIf a customer wants you to send him CSV, ask if he can consume JSON.\n\nThere are great libraries for reading and writing Excel. For example:\n[openpyxl](https://openpyxl.readthedocs.io/en/stable/)\n\nOther alternatives to CSV:\n\n* [zarr](https://zarr.readthedocs.io/en/stable/) (For data science (very long arrays))\n* [jsonlines](http://jsonlines.org/) (for example for logfiles)\n\n### Give booleans a \"positive\" name\n\nI once gave a DB column the name \"failed\". It was a boolean indicating\nif the transmission of data to the next system was successful. The\noutput as a table in the GUI looked confusing for humans. The column\nheading was \"failed\". What should be visible in the cell for failed\nrows? Boolean usually get translated to \"Yes/No\" or \"True/False\". But if\nthe human brain reads \"Yes\" or \"True\" it initially thinks \"all right\".\nBut in this case \"Yes\" meant \"Yes, it failed\". The next time I will call\nthe column \"was\\_successful\", then \"Yes\" means \"Yes, it was successful\".\nSome GUI toolkits render \"True\" as a green (meaning \"everything is ok\")\nhook and \"False\" as a red cross (meaning \"it failed\").\n\n### Love your docs\n\nI have seen it several times on Github. Just have a look\nat the README files on GitHub. They start with \"Installing\", \"Configuring\", then \"Special Cases\"... \n\nWhat is missing? An introduction! Just some sentences\nabout what this great project is all about. Programmers prefer the details to the big picture,\nthe overview. \n\nBut \"Project simple-foo simplifies foo\" is not enough. What is \"foo\"?\n\nDear programmers, learn to relax and look at the thing you create like a newcomer. Imagine a newcomer who knows how to add two integers with his favorite programming\nlanguage. What is missing to make him understand why the project/lib/tool is needed?\n\nFirst, you need to convince him that this project is worth a try, then if he knows\nthe \"why?\", then explain how to install it.\n\nIf you have this mindset \"I do the important (programming)\nstuff. Someone else can care for the docs\", then your open source\nproject won't be successful.\n\nIf you write docs, then do it for newcomers. Start with the\nintroduction, define the important terms, then provide simple and straightforward use\ncases. Put details and special cases at the end.\n\nIf your library gets used and you add a bug, you will get feedback soon.\n\nTests fail or even worse customers will complain.\n\nBut if you write broken docs, no one will complain.\n\nEven if someone reads your mistake, it is unlikely that you get\nfeedback. Unfortunately, only a few people take this seriously and tell you\nthat there is a mistake in your docs.\n\nHow to solve this?\n\nYou need to act.\n\nLet someone else read your docs.\n\nThe quality of feedback you get depends on the type of person you ask to\nread your docs.\n\nIf it is a programmer, likely, he does not read your docs\ncarefully. Most software developers do not care for orthography and it\nis hard for them to read the docs like a newcomer. They already know\nwhat's written there, and they will say \"it is ok\".\n\nMy solution: resubmission: Read the text again 30 days later.\n\nA good example is [gVisor](https://github.com/google/gvisor) the README starts with \"What is gVisor?\" and \"Why does gVisor exist?\"\n\n### Test your docs\n\nKeeping your docs in the same git repo like your code makes sense. This has the benefit that you have a review and testing process.\n\nIntegrate automated spell checking into the CI process.\n\n### Canonical docs\n\nLook at the question concerning OpenSSH options at the Q+A site [serverfault.com](https://serverfault.com/).\nThere is a lot of guessing. Something is wrong. Nobody knows where the\ncanonical upstream docs are. Easy linking to a specific configuration is not\npossible. What happens? Redundant docs. Many blog posts try to explain\nstuff... Don't write blog posts, instead, you should improve the upstreams docs. Talk with\nthe core developers. Open an issue in the issue tracker if you think something is missing in the docs.\n\nOpen an issue if the docs start with the hairy details and don't start\nwith an introduction/overview. Developers don't realize this, since they\nneed to deal with the hairy details daily. Don't be shy: Help them to\nsee the world through the eyes of a newcomer.\n\nI am unsure if I should love or hate \"wiki.archlinux.org\". On the one\nhand, I found there valuable information about systemd and other Linux\nrelated secrets. On the other hand, it is redundant and since a lot of\nusers take their knowledge from this resource, the canonical upstream\ndocs get less love. First, determine where the canonical upstream docs\nare. Then communicate with the maintainers. Avoid redundant docs.\n\nIn other words: Blog posts are nice, but they are like dust in\nthe wind. They explain a snapshot. Three months later they are outdated.\nIt makes more sense to add one missing sentence to the upstream docs,\nthen to create a blog post explaining something which is not explained\nin the docs. At least in the open-source world. Since it is more likely\nthat you can influence the upstream docs.\n\nRelated: [Single Source of Truth (Wikipedia)](https://en.wikipedia.org/wiki/Single_source_of_truth)\n\nRelated: [Canonical URL](https://en.wikipedia.org/wiki/Canonicalization#URL)\n\nRelated: [\"Don't repeat yourself\" vs \"We enjoy typing\"](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself#DRY_vs_WET_solutions)\n\n### One central glossary: One page per term\n\n\u003e There are only two hard things in Computer Science: cache invalidation and naming things.\n\u003e -- Phil Karlton\n\n[Martin Fowler](https://martinfowler.com/bliki/TwoHardThings.html)\n\nMy best practice to solve the \"naming things\" challenge\n\n* Define your terms, your terminology. For small projects, a glossary is enough, but for bigger projects, every term should have its page. It should be easy to create a hyperlink to this term. That's why I prefer the \"one term, one-page\" approach. Creating hyperlinks into a page (https://..../...#foo) are possible but less fun.\n* The defined terms should not differ too much from the spoken words (or the words used in your chat/mail messages). If there is a difference, then alter the written definition. \n* Someone should be responsible for the docs. \"Everybody is responsible for it\" does not work.\n* Encourage and motivate people, again and again, to speak up if the docs are outdated.\n\nMore about this topic from me: [Intranets](https://github.com/guettli/intranets)\n\n### Do not send long instructions to customers via mail\n\nIf you send long instructions to customers via mail, then these docs in\nthe mail are hidden magic. Only the customer who receives this mail\nknows the hidden magic.\n\nPublish your docs in your app. Send your customer a link to the online\ndocs.\n\nDespite all myths: Some users read the docs!\n\nAnd that's great if the user has more knowledge. Because this means you\nhave less work. Fewer emails, fewer interrupts, fewer phone calls :-)\n\nThis even applies to public discussion forums. Don't write too much. Create great docs and answer questions by\nproviding links to the docs. And be polite and include the question if this answers the question of the user.\n\n[Permalinks](https://en.wikipedia.org/wiki/Permalink) are great, since they provide a single source of truth.\n\n\n###  Don't write tech-docs in a non-English language\n\nGeneral rule: don't waste time.\n\nIt is feasible to write high-level blog posts about tech topics in your favorite language.\n\nSometimes it is easier to communicate the holistic view in your mother-tongue.\n\nBut it is not feasible to write detailed tech stuff in a non-English language.\n\nExample:\n\nhttps://wiki.ubuntuusers.de/Installation_auf_externen_Speichermedien/\n\nI came across this page because I want to install Linux on an external hard disc.\n\nUnfortunately, there seemed to be no good English guide on how to do this.\n\nThe most solid guide I found during the first minutes was the above link. Unfortunately, the above guide was outdated.\n\nGrrrrrr. Now I needed to choose:\n\n* V1: Should I update the outdated german guide? It is a wiki editable by everybody.\n\n* V2: I use an English guide, but they look not solid. \n\nGrrr. I don't like thinking.\n\nThe people who created the German guide thought they help the world. They felt good\nwhile doing what they did. I think they wasted time. Automatic translations are quite\ngood today. At least if you translate English to your favorite language.\nI won't update the outdated German guide in the wiki. This would help only very few people.\nMost people which want to install Linux on an external hard drive can either\nread English text or they know who to translate English text to their favorite\nlanguage. I would update an Englisch wiki page since this would help a lot of people.\n\nDon't get me wrong: Docs for applications you write should be in the language of your customers. Above text\nis about tech-related docs.\n\nMy conclusion: Don't write tech-docs in a non-English language\n\n### Care for newcomers\n\nIn the year 1997, I was very thankful that there was a hint \"If unsure\nchoose ...\" when I needed to compile a Linux kernel. These days you\nneed to answer dozens of question before you could compile the invention of\nLinus Torvalds.\n\nI had no clue what most questions were about. But this small advice \"If\nunsure choose ...\" helped me get it done.\n\nIf you are managing a project: Care for newcomers. Provide them with\nguidelines. But don't reinvent docs. Provide links to the relevant\nupstream docs, if you just use a piece of software.\n\n\n### Good example for \"care for newcomers\"\n\n\u003e Writing plugins\n\u003e \n\u003e It is easy to implement local conftest plugins .... Please refer to [Installing and Using plugins](https://docs.pytest.org/en/stable/plugins.html#using-plugins) if you only want to use but not write plugins.\n\nThat's great. That's newcomer focused documentation.\n\n### Keep custom IDE configuration small\n\nImagine you lost your PC and you lost your development environment:\n\n-   IDE configuration\n-   Test data\n-   Test database\n\nAll that's left is your source code from version control, CI servers and\ndeployment workflow.\n\nHow much would you lose? How much time would you waste to set up your\npersonal development environment again?\n\nKeep this time small. This is related to \"care for newcomers\". If you\nneed several hours to set up your development environment, then a new team member would need even much more time.\n\nAlthough I use PyCharm and VSCode, the introduction of [Gitpod](https://www.gitpod.io/) gets it to the point:\n\n\u003e Gitpod does to Dev Environments what Docker did to Servers. Today we are emotionally attached (for better or worse) to our dev environments, give them names \u0026 massage them over time. They are pets - similar to servers before docker took advantage of namespaces and cgroups in Linux and turned these nice puppies into cattle. \n\u003e With Gitpod it is the same - we treat dev environments as automated resources you can spin up when you need them and close down (and forget about) when you are done with your task. Dev environments become fully automated and ephemeral. Only then you are always ready-to-code - immediately creative, immediately productive with the click of a button, and without any friction.\n\n\n\n### Setting up a new development environment should be easy\n\nThis happened to me several times: I wanted to improve some open source\nsoftware. Up until now I only used the software, now I want to write a\npatch. If setting up a new development environment and running the tests\nis too complicated or not documented, then I will resign and won't\nprovide a patch. These steps need to be simple for people starting from\nscratch:\n\n-   check out the source from version control\n-   check that all tests are working (before modifying something)\n-   write a patch and write a test for your patch\n-   check that all tests are working (after modifying something)\n\n### Passing around methods make things hard to debug\n\nEven in C, you can pass around method-pointers. It's very common in\nJavaScript and sometimes it gets done in Python, too. It is hard to\ndebug. IDE's can't resolve the code: \"Find usages\" don't work. I try to\navoid it. I prefer OOP (Inheritance) and avoid passing around methods or\ntreating methods like variables.\n\nBut maybe this is just my strong backend related roots. I have never\ncoded in a big modern JavaScript-based environment.\n\nI like it simple: Input-Processing-Output.\n\nWith \"Input\" being 100% data. Not a method.\n\n### Software Design Patterns are overrated\n\nIf you need several pages in a book to explain a software design\npattern, then it is too complicated. I think Software Design Patterns\nare overrated.\n\nWhy are so many books about software design patterns and nearly no books\nabout database design patterns?\n\n### OOP is overrated\n\nAbout OOP (Object-oriented programming)\n\n**Stateless** has won. OOP is stateful:\n\n1. Create an instance of a class\n2. Call a method of this instance\n3. Destruct the instance\n\nThree steps vs one step.\n\nOOP is great for implementing an [ORM (Object-relational mapping)](https://en.wikipedia.org/wiki/Object-relational_mapping). But implementing this should be done by people who have more experience than I have :-)\n\nHere is code that uses the well-known jUnit style:\n```\n# OOP way\nimport unittest\n\n\nclass TestSMTP(unittest.TestCase):\n    def smtp_connection(self):\n        import smtplib\n        return smtplib.SMTP(\"smtp.gmail.com\", 587, timeout=5)\n    \n    def test_helo(self):\n        response_code, msg = self.smtp_connection().ehlo()\n        self.assertEqual(response_code, 250)\n```\n\nThe non-object-oriented way:\n```\n# pytest way\nimport pytest\n\n\n@pytest.fixture\ndef smtp_connection():\n    import smtplib\n    return smtplib.SMTP(\"smtp.gmail.com\", 587, timeout=5)\n\n\ndef test_ehlo(smtp_connection):\n    response_code, msg = smtp_connection.ehlo()\n    assert response_code == 250\n```\n\nMy rule of thumb: Less indentation, means less complexity, means better code.\n\nTwo things are simplified: The second version does not need a class or inheritance. Nice, since less code means fewer bugs.\n\nIn the second example the method `smpt_connection()` is not an instancemethod of a class, it just an unbound method. If a test\nasks for a parameter with this name, then pytest gives the test the result of this method.\n\nAnd look at the assertion: `self.assertEqual(response_code, 250)` vs `assert response_code == 250`. Namespaces\nintroduced by dots are great (`assertEqual` is in the namespace of `self`). But if one level is enough, then\nthis is even better.\n\nOf course, this is opinionated, and it is 100% ok if you prefer the OOP-way and not the shorter solution.\n\n### \"Dependency injection\" is just \"Configuration\"\n\nFor me, the term [Dependency injection](https://en.wikipedia.org/wiki/Dependency_injection) and the corresponding Wikipedia article are way too complicated.\n\nFor me, it is just \"Configuration\". But some people don't like it simple, they prefer .... (I removed this phrase since it was provocative. Feel free to add your favorite phrase here)\n\nFrom Wikipedia \"Dependency injection\"\n\n\u003e In the following Java example, the Client class contains a Service member variable that is initialized by the Client \n\u003e constructor. The client controls which implementation of service is used and controls its construction.\n\u003e In this situation, the client is said to have a hard-coded dependency on ExampleService.\n\nNow have a look at these docs [Database Settings](https://docs.djangoproject.com/en/3.0/ref/settings/#databases)\n\n```\nDATABASES = {\n    'default': {\n        'ENGINE': 'django.db.backends.sqlite3',\n        'NAME': 'mydatabase',\n    }\n}\n```\n\nThat's all: Instead of hard-coded dependencies, you provide a way to configure your software.\n\nI avoid the term \"Dependency injection\", since it is unclear to me.\n\n### Test-Driven Development\n\nred, green, refactor. More verbose: make the test fail, make the test\npass, refactor (simplify) code.\n\n\n### Extract Method to get full coverage\n\nImagine you have a method like this:\n```\ndef my_method(a, b, c):\n    # ten \n    # lines\n    # of \n    # code\n    \n    if a \u003e b:\n        # ....\n        \n    # again\n    # ten\n    # lines \n    # of \n    # code\n```\n\nOne thing is 100% sure: You can get full coverage with one test. You would\nneed to call the method twice: Once with `a \u003e b` and once with opposite.\n\nBut you don't want to call this method twice, since useless executing\nof the code above and below the \"if\" statementent. You want to avoid\nthat you test suite gets too big and too slow.\n\nMaybe you could extract the condition into an new method:\n\n```\ndef my_method(a, b, c):\n    # ten \n    # lines\n    # of \n    # code\n    \n    d = handle_case_foo(a, b)\n        \n    # again\n    # ten\n    # lines \n    # of \n    # code\n\ndef handle_case_foo(a, b):\n    if a \u003e b:\n        return ...\n    return ...\n```\n\nThis way you can test `my_method()` with one test, and you can write\na small test for `handle_case_foo()`.\n\n### From bug to fix\n\nFirst, make your bug reproducible. If it is reproducible, then it is easy\nto fix it.\n\nMake it reproducible in a test.\n\nImagine there is a bug in your method do\\_foo(). You see the mistake\neasily and you fix it. Done?\n\nI think you are not done yet. I try to follow this guideline:\n\nBefore fixing the bug, search test\\_do\\_foo(). There is no test for this\nmethod up until now? Then write it.\n\nNow you have test\\_do\\_foo().\n\nYou have two choices now: extend test\\_do\\_foo() or write\ntest\\_do\\_foo\\_\\_your\\_special\\_case(). I use the double underscore\nhere.\n\nMake the test fail (red)\n\nFix the code. The test is green now.\n\nSlow down. Take a sip of tea. Look at your changes (\"git diff\" in your\npreferred IDE). Is there a way to simplify your patch? If yes, simplify\nit.\n\nRun the \"surrounding tests\". If do\\_foo() is inside the module \"bar\".\nThen run all tests for module \"bar\" (I use py.test -k bar). But if this\nwould take more than three minutes, then leave the testing to the CI\nwhich happens after you commit+push (you have a CI, haven't you?)\n\n### Tests and production code go hand in hand.\n\nYou implemented the great method foo() and you implement a corresponding\nmethod called test\\_foo(). It does not matter if you write foo() first,\nand then test\\_foo() or the other way round. But it makes sense to store\nboth methods with one commit to one git repo.\n\nSeveral months later you discover a bug in your code. Or worse: your\ncustomer discovers it.\n\nIf you fix foo() you need to extend test\\_foo() or write a new method\ntest\\_foo\\_with\\_special\\_input(). Again both changes (production code\nand testing code) walk into the git repo like a pair of young lovers\nholding hands :-)\n\nRelated [Guideline of Google: Codereview \"Tests\"](\nhttps://google.github.io/eng-practices/review/reviewer/looking-for.html#tests)\n\n### 80% unit-tests\n\n* 80% unit-tests\n* 15% integration tests\n* 5% end-to-end tests\n\nFrom [Software Engineering at Google](https://www.oreilly.com/library/view/software-engineering-at/9781492082781/)\n\n### pre-commit hook\n\nFor basic syntax checking (aka linting) before commit I use [pre-commit](https://pre-commit.com/)\n\nAdding simple checks is very easy: [hook to reject commit if a file contains a specific string](https://stackoverflow.com/a/66171121/633961)\n\n### aaa-tests (smoke tests)\n\nIf you have a huge test-suite, which takes more than ten minutes to execute, then I recommend\nto flag some tests. I call these tests \"aaa\" tests. These tests should be fast and check the basic\nstuff. \n\nThis way you can check if most parts are all right before pushing code and triggering CI.\n\nSome call these \"smoke tests\".\n\nWhy \"aaa\"?\n\nMost test runners allow you to execute all tests which match a certain pattern. I name the tests \"test_aaa_...\",\nand then I can easily run all these tests. Example: `pytest -k aaa`.\n\nRunning all aaa-tests should take less then a minute\n\nBut I don't call it automatically before each commit. \n\n### Creating test data is much more important than you initially think\n\nCreating test data is very important. It can help you with several\nthings:\n\n1: It can help you to create a re-usable application. If you have only\none customer, it does not matter. But the real benefit of software is its re-usability.\nYour code wants to get executed by several customers. As soon as you have two or more \ncustomers you need a neutral test environment that is no specific to one of your customers.\nIt is a lot of work to create a neutral test environment if you have not done it from\nday one. But the work only needs to be done once and helps in the long run.\n\n2: It can help you to create presentation/demo systems.\n\n3: It can help you in automated tests.\n\nYour tests should not run on real data from customers.\n\nIf you create test data this should be automated. This way you can fill a new database with useful data. You should be able to create a\ndemo system with one command (or one click).\n\nWrite the creation of test data once and use it for both: presentions\nand automated tests.\n\n### Don't use random data for tests.\n\nDo not use random data for tests. It just makes no sense: the test environment should\nbe reproducible, not flaky.\n\nSome people use libraries which create random user names and addresses (street, city, postal code, .....) like [Faker](https://pypi.org/project/Faker/).\n\nI don't see why a special library for creating test data is needed. Random data leads to flaky tests.\n\nIf you need some a list of names/addresses/ to fill you database, then I see these options:\n\n* Option0: If you users have different roles, use a corresponding name: like \"Admin\", \"Staff\", \"User\", ...\n* Option1: Be creative and/or use names which come to your mind: Bob Geldof, Steve Wonder, Mr. Bean, ...\n* Option2: you can take data from here by hand: https://github.com/joke2k/faker/tree/master/faker/providers\n* Option3: Use the [faker](https://faker.readthedocs.io/en/master/) library **once** and create some JSON. Store this JSON in your code or in an extra file. Then uninstall faker.\n\n\nThis way it is far easier to debug a test which works on your machine, but fails in CI. If you use random data, then\nthis is much harder. Imagine in CI a mail gets send to only three users, although four users should get an email. If you\nuse random data you can't differentiate between the users. If you use a predictable naming scheme, then you can distinguish between\nthe users.\n\nThis guideline is about writing tests. If you create demo-systems, then it is the same: Don't use\nrandom data. The output should repeatable. Although for a demo-system you usualy want nice names.\n\nIf you use an ORM in your production code, then use the ORM to create your\ntest data.\n\nI like [pytest fixtures](https://docs.pytest.org/en/latest/explanation/fixtures.html).\n\nI know that there special cases and corresponing libraries which use fuzzing to test edge cases.\nFor example Golang has the package [fuzz](https://go.dev/doc/fuzz/). But that's only for special cases,\nif is not needed for most application programming.\n\n### How to create QA and staging systems?\n\nMany teams create the QA and staging systems by copying the production system.\n\nThis works, but I think it is better to create these systems from code stored in\nversion control.\n\nCreating a test system via code looks complicated at first, but it helps you to create\nreliable, reproducible systems. This makes you faster in the long run.\n\n### Don't check for counts in unittests.\n\n\u003e AssertionError: 8 != 9\n\nThat's a useless error message.\n\nYou have absolutely no clue if a test fails with a message like this.\n\nIt is much more useful to compare a list of strings.\n\n### This is untestable code\n\nIf you are new to software testing, then you might think ... \"some parts\nof my code are *untestable*\".\n\nI don't think so. I guess your software uses the [IPO pattern](https://en.wikipedia.org/wiki/IPO_model): Input, Processing, Output. The\nquestion is: How to feed the input for testing to my code? Mocking,\nvirtualization, and automation are your friends.\n\nThe \"untestable\" code needs to be cared for. Code is always testable,\nthere is no untestable code. Maybe your knowledge of testing is limited\nup until now. Finding untestable code and making it testable is the\nbeginning of an interesting adventure.\n\n### Flaky Tests\n\nTests are never flaky. If the same code ran fine yesterday, and it the same code\nfails today, then the test itself is stable. \n\nThe environment is flaky. Some small bit in the environment is different today.\n\nMaybe the servers are under more load today, which results in slower responses, which\nresults in timeouts.\n\nMaybe it fails because there is a new test that executes before the flaky test and which\nmodifies the database.\n\nMaybe a shared resource contains different data today.\n\n...\n\nThe bigger your environment, the more likely you have flaky tests.\n\nThis is the way to avoid flaky tests:\n\n* Keep your test, simple. Try to write stateless methods that receive only a few input.\n* keep the environment, simple. If you can avoid Selenium, then avoid it. This will save you time.\n* Avoid shared resources. Tests should have their own database, their own cache, ...\n* ...\n\n### Hermetic Testing\n\nThis blog from [Google Testing Blog \"Hermetic Servers\"](https://testing.googleblog.com/2012/10/hermetic-servers.html) explains it in-depth: \nEnd-to-End tests are faster and less flaky if they run on localhost and don't need other resources.\n\nThis usualy means:\n\n* The database is running on localhost\n* storage server (S3) is running on localhost or in-memory\n* Cache Server (Redis) is running on localhost or in-memory.\n\nFor storage and cache it is easy to find an in-memory solution (for Django [dj-inmemorystorage](https://github.com/waveaccounting/dj-inmemorystorage),\nbut for the database it is more difficult. My opinion: Use PostgreSQL during development.\nDon't use SQLite, since it does not support all features of PostgreSQL.\n\n### Hermetic Testing: N times on localhost\n\nIt should be easy for developers to set up several test-systems on his local machine.\n\nIf you are working on a larger change, it is really helpful to have one system with the old state, and\na second system with the new state.\n\nBoth systems should be hermetic, which means that they don't share resources.\n\nUsing the same database server is fine, but both should use different databases.\n\n### Unit-Tests may use the ORM.\n\nImagine you use a framework that provides you a nice ORM to create, read, update, and delete your data.\n\nNow you write some backend-methods on top of this ORM.\n\nAnd on top of your methods, you might provide an HTTP API.\n\nImagine you have a class `Ticket` which has a method called `resolve()`. This method uses the ORM.\n\nYou want to write a unit-test for this method.\n\nA purist argues: I only want to unit-test the method, I must not use the ORM since blablabla.\n\nI understand what the purist wants. But I want to get things done. I want to make\ncustomers happy, not unit-test purists.\n\nFor me, it is 100% ok if unit-tests use the ORM.\n\nIn other words: Only mock away things that take too long or things that need resources\nwhich are not available (e.g. an SMTP server).\n\nRelated Podast: [Don't Mock your Database (Jeff Triplett)](https://testandcode.com/154)\n\n### Is config code or data?\n\nThe heading \"Is config code or data?\" could be phrased as \"config: DB or git?\", too.\n\nWhere should configuration be stored?\n\nThis is a difficult question. At least at the beginning. For me, most\nconfiguration is data, not code. That's why the config is in a\n**database**, not in a text or source code file in a version control\nsystem.\n\nThis has one major drawback. All developers love their version control\nsystem. Most developers love git. It is such a secure place. Nothing can get lost\nor accidentally modified. And if a change was wrong, you can always revert\nto an old version. It is like heaven. Isn't it?\n\nNo, it is not. The customer can't change it. The customer needs to call\nyou and you need to do stupid repeatable useless work.\n\nFor me, the configuration should be in the database. This way you can provide\na GUI for the customer to change the config.\n\nThe configuration and recipes for the configuration management are\nstored in git. But this is a different topic. If I speak about\nconfiguration management, then I speak mostly about configuring Linux\nservers and networks (aka [Infrastructure as code](https://en.wikipedia.org/wiki/Infrastructure_as_code)). In my case, this is nothing which my customer\ntouches.\n\n### ForeignKey from code to DB\n\nThis code uses the ORM of Django\n\n``` {.sourceCode .python}\nif ....:\n    issue.responsible_group=Group.objects.get(name='Leaders')\n```\n\n`Group` is a class and refers to a table with the same name. Each group has a name. There\nis one group (one row) with the name \"Leaders\".\n\nThe above code is dirty because 'Leaders' is like a ForeignKey from code to\na database row.\n\nHow to avoid this?\n\nCreate a global config table in your database. This table has exactly one\nrow. That's the global config. There you can create a column called\n\"Leaders\" and there you store the ForeignKey to the matching group.\n\n### Testcode is conditionless\n\nTest code should not contain conditions (the keyword `if`). If you have\nloops (`for`, `while`) in your tests, then this looks strange, too.\n\nTests should be straight forward:\n\n\u003e 1.  Build environment: Data structures, ...\n\u003e 2.  Run the code which operates on the data structures\n\u003e 3.  Ensure that the output is like you want it to.\n\n### Code Review: Start to look at the tests first\n\nIf I do a code review, I like to look at the tests first. This hides\nthe implementation from my eyes and shows how the method get used.\n\nA clean interface is more important than a clean implementation. The\nimplementation can get refactored easily. The interface is harder to change,\nsince in most cases all usages of the interface need to be updated.\n\n### Don't search the needle in a haystack. Inject dynamite and let it explode\n\nImagine you have a huge codebase that was written by a nerd who is\ngone for several months. Somewhere in the code, a row in the database gets\nupdated. This update should not happen, and you can't find the relevant\nsource code line during the first minutes. You can reproduce this\nfailure in a test environment. What can you do? You can start a debugger\nand jump through the lines which get executed. Yes, this works. But this\ncan take longer, it is like \"Searching the needle in a haystack\". Here is\na different way: Add a constraint or trigger to your database which\nfires on the unwanted modification. Execute the code and BANG - you get\nthe relevant code line with a nice stack trace. This way you get the\nsolution provided on a silver platter with minimal effort :-)\n\nIn other words: Don't waste time searching.\n\nSometimes you can't use a database constraint to find the relevant\nstack trace, but often there are other ways.....\n\nIf you can't use a database constraint, maybe this helps: Raise\nException on unwanted syscall (Python+GDB)\n\u003chttp://stackoverflow.com/a/42669844/633961\u003e\n\nIf you want to find the line where unwanted output in stdout gets\nemitted: \u003chttp://stackoverflow.com/a/43210881/633961\u003e\n\nIf you have a library that logs a warning, but the warning does not\nhelp, since it is missing important information. And you have no clue\nwhere this warning comes from. You can use this solution:\n\u003chttp://stackoverflow.com/a/43232091/633961\u003e\n\nYou can use `strace -e inject...` to perform syscall tampering for \nthe specified set of syscalls.\n\n\n### Avoid magic or uncommon things\n\n- hard links in Linux file systems.\n- file system ACLs (Access control lists). Try to use as little as possible chmod/chown.\n- git submodules (Please use dependency management, configuration management, deployment, tools, ...)\n- [seek()](https://en.cppreference.com/w/c/io/fseek). Stateless is better. If you use seek() the file position is a state. Sooner or later the position (state) will be wrong.\n- Scripts which get executed via OpenSSH [ForceCommand](http://man.openbsd.org/OpenBSD-current/man5/sshd_config.5#ForceCommand) or \"command\" in .ssh/authorized_keys. SSH is not an API, use http.\n- I think even [symbolic links](https://en.wikipedia.org/wiki/Symbolic_link) are strange and outdated. Just some minutes ago I got confused because `grep -r foo .` did not show a result, but `grep foo ./my-dir/abc.txt` showed a result. Root-cause: `my-dir` was a symlink.\n\n### Avoid writing a native GUI\n\nImagine you have developed web applications up until now. You have never\ndeveloped a native GUI before. Now a new potential customer has a use\ncase and you think: This time a native GUI would be a good solution.\n\nCaution: slow down. Developing a native GUI is much more work and needs\nmuch more time than you think.\n\nThe edit, compile, run cycle is much longer. This will slow you down.\n\nIf you develop a native GUI, you might need several mouse clicks until\nyou reach the part where you improving the current code. And like all\nhumans, you are not perfect, and you have a typo. The application\ncrashes, and you need to do the edit, compile, run, five clicks cycle\nagain...\n\nCompare this to a web application: You do not need to do five clicks to\nreach the part where you improve the current code. You just hit ctrl-r\nand reload the page. The stateless HTTP protocol makes this possible. I\nlove it.\n\nNext argument: The native GUI community is tiny compared to web\ndevelopment. If you have a question, you have only a few people to talk\nto.\n\nI am at the Chemnitzer Linux Days yearly and meet a lot of newcomers\nthere. Some people new to software development think: \"I just want to\ndevelop a simple app for me. No need to run a web server. I want a real\napplication running on my pc.\"\n\nMy advice: use Python and Django. The things you learn have more value.\nThe knowledge you gain can be used to build cool stuff. If you have a\nquestion, there is always someone who has an useful advice.\n\nSee the [TagTrend gtk, qt,\ndjango](http://sotagtrends.com/?tags=%5Bgtk,qt,django%5D)\n\n\n### Avoid writing native Apps\n\nDeveloping a mobile-friendly web application is much easier than writing a \nnative app. If you can avoid it, then avoid writing a native app.\n\nThe development and release process is much slower.\n\nOf course, the age of [Progressive Web Apps](https://web.dev/progressive-web-apps/) has just begun.\nA lot of things are not possible in a web app up until now. Just be warned, that this road is\nslow and in the long run deprecated, since the environments for PWAs are getting better every year.\n\n### My prefered Web Stack\n\nPython, Django, Gunicorn, Nginx, PostgreSQL, [htmx](https://htmx.org/), Bootstrap5.\n\nThis way I can write responsive mobile friendly applications.\n\nI think React/Vue are in general overrated and not needed for my use cases.\n\n### Learn one programming language, not ten.\n\nMost young developers think they need to learn many programming languages\nto be a good developer.\n\nMy opinion: Learn Python, SQL, and some JavaScript.\n\nThen learn other topics: PostgreSQL, Configuration management,\ncontinuous-integration, organizing, teamwork, learn to play a musical\ninstrument, long-distance running, family\n\n\n### git \n\nMoved here [git tips](https://github.com/guettli/git-tips)\n\n### Avoid Conditional Breakpoints\n\nImagine, you can reproduce a bug in a test. But you could not\nfix it at the moment. If you want to create a conditional breakpoint to find\nthe root of the problem, then you could be on the wrong track. Rewrite\nthe code first, to make it more fine-grained debuggable and testable.\n\nModify the source and test where a normal (non-conditional) breakpoint is enough.\n\nThis likely means you need to move the body of a loop\ninto a new method.\n\n``` {.sourceCode .}\n# Old\ndef my_method(...):\n    for foo in get_foos():\n        do_x(foo)\n        do_y(foo)\n        ...\n```\n\n``` {.sourceCode .}\n# new\ndef my_method(...):\n    for foo in get_foos():\n        my_method__foo(foo)\n\ndef my_method__foo(foo):\n    do_x(foo)\n    do_y(foo)\n    ...\n```\n\nNow you can call `my_method__foo()` in a test, and you don't need a\nconditional breakpoint anymore. This helps you now (during debugging), but raises\nthe overall value of the source code in the long run, too. Instead of a few big monster methods,\nyou have more small and easy to understand methods that follow the simple input-processing-output model.\n\n### Make a clear distinction between Authentication and Permission Checks\n\nIt is important to understand the difference.\n\n**Authentication** happens first: Is the user really Bob, or is there\njust someone who pretends to be Bob?\n\n**Permission Checks** Is Bob allowed to do action \"foo\"? Here we already\ntrust that the user is Bob and not someone else. I use the term\n\"Permission Checks\" on purpose since the synonym \"Authorization\" sounds\ntoo similar to \"Authentication\".\n\nRelated question:\n\u003chttps://softwareengineering.stackexchange.com/questions/362350/synonym-for-authorization/363690#363690\u003e\n\nEven the http-spec confuses both similar sounding words: \n\n\u003e There's a problem with 401 Unauthorized, the HTTP status code for authentication errors. And that’s just it: it’s for authentication, not authorization. Receiving a 401 response is the server telling you, “you aren’t authenticated–either not authenticated at all or authenticated incorrectly–but please reauthenticate and try again.\n\nSource: [403 Forbidden vs 401 Unauthorized HTTP responses](https://stackoverflow.com/questions/3297048/403-forbidden-vs-401-unauthorized-http-responses)\n\nGeneral guidelines: Avoid [Homonyms](https://en.wikipedia.org/wiki/Homonym)\n\n### Idempotence is great\n\nIdempotence is great, since it ensures that it does no harm if\nthe method is called twice.\n\nErrors (for example power outage) can happen in every millisecond.\nThat's why you need to decide what you want:\n\n-   if the power outage happened, some jobs do not get executed.\n    Cronjobs work this way.\n-   if the power outage happened, some jobs do get executed twice to\n    ensure they get done.\n\nFurther reading:\n\u003chttp://docs.celeryproject.org/en/latest/userguide/tasks.html\u003e (I don't\nuse celery, but I like this part of the docs)\n\n\u003chttps://en.wikipedia.org/wiki/Idempotence\u003e\n\n### File Locking is deprecated\n\nIn the past [File Locking](https://en.wikipedia.org/wiki/File_locking)\nwas a very interesting and adventurous topic. Sometimes it worked,\nsometimes not, and you got interesting edge cases to solve again and\nagain. It was fun, especially on NFS (Network File System). Only hardcore experts know the difference between\nfcntl, flock, and lockf.\n\n.... But on the other hand: It's too complicated, too many edge cases,\ntoo much wasting time.\n\nThere will be chaos if there is no central dispatcher.\n\nI like tools like \u003chttp://python-rq.org/\u003e. It is simple and robust.\nBut the next time I create something like this, I will try [django-pg-queue](https://github.com/SweetProcess/django-pg-queue)\n\nBTW, the topic is called\n[Synchronization](https://en.wikipedia.org/wiki/Synchronization_(computer_science)).\n\nFurther reading about \"task queues\":\n\u003chttps://www.fullstackpython.com/task-queues.html\u003e\n\n### No nested directory trees\n\nIf you store files, then avoid nested directory trees. It is complicated\nand if you want to use a storage server like\n[S3](https://en.wikipedia.org/wiki/Amazon_S3) later, you are in trouble.\n\nMost storage servers support containers and\n[blobs](https://en.wikipedia.org/wiki/Binary_large_object) inside a\ncontainer. Containers in containers are not supported, and that's good\nsince it makes the environment simpler.\n\n### Code doesn't call mkdir\n\nCode runs in an environment. This environment was created with\nconfiguration management. This means: source code usually does not call\nmkdir. In other words: Creating directories is part of configuration management. Setting up the environment and executing code\nin this environment are two distinct parts. If your software runs, the\nenvironment does already exist. Code creating directories if they do not\nexist yet should be cut into two parts. One part is creating the\nenvironment (gets executed only once) and the second part is the daily\nexecuting (which is 100% sure that the environment is like it is. In\nother words: the code can trust the environment that the directory\nexists). These two distinct parts should be separated.\n\nHow to create directories if I should not do it with my software? With\nautomated configuration management (Ansible, Chef, ...) or during\ninstallation (RPM/DPKG).\n\nException: You create a temporary directory that is only needed for\nsome seconds. But since switching from subprocess/shell calling to using\nlibraries (see \"Avoid calling command line tools\") temporary files get\nused much less.\n\n### Debugging Performance\n\nI use two ways to debug slow performance:\n\n\u003e -   Logging and profiling, if you have a particular reproducible use\n\u003e     case\n\u003e -   Django Debug Toolbar to see which SQL statements took long in a\n\u003e     HTTP request.\n\u003e -   Statistics collected on production environments. For Python:\n\u003e     \u003chttps://github.com/uber/pyflame\u003e or\n\u003e     \u003chttps://github.com/benfred/py-spy\u003e\n\n### You provide the GUI for configuring the system. Then the customer (not you) uses this GUI\n\nI developed a workflow system for a customer. The customer gave me an\nexcel sheet with steps, transitions, and groups.\n\nThe coding was the difficult part.\n\nThen I configured the system according to the excel sheet.\n\nThe code was bug-free, but I made a mistake when I entered the values\n(from excel to the new web-based workflow GUI).\n\nThe customer was upset because the configuration contained mistakes.\n\nI learned. Now I ask if it would be ok if I provide the GUI and the\ncustomer enters the configuration. In most cases, the customer likes to\ndo this.\n\nThere is a big difference. The customer feels productive if he does\nsomething like this. I hate it. I care for the database design and the\ncode, but entering data with copy+paste from the Excel sheet ... No I\ndon't like this. Results will be better if you like what you do :-)\n\nFor detail lovers: No, it was not feasible to write a script that\nimported the excel sheet to the database. The excel sheet was not well\nstructured.\n\n*give a man a fish and you feed him for a day; teach a man to fish and\nyou feed him for a lifetime*\n\n\n\n### Better error messages\n\nIf you have worked with Windows95, then you must have seen them: Empty\nerror messages with just a red icon and a button labeled \"OK\". You had\nno clue what was wrong. On the one hand, it was great fun, on the other\nhand, it was very sad since you wasted your precious time.\n\nDo it better.\n\nImagine user \"foo\" wants to access data (let's call it \"pam\") which you\nonly can see if you are in the group \"Baywatch\". Unfortunately, user\n\"foo\" is not in the group. You could show him the simple message\n\"permission denied\". And no further information.\n\nI don't like messages like this. They create extra work. The user will\ncall the support and ask the question \"Why am I not allowed to see the\ndata?\". The support needs to check the details.... and soon a half-hour\nof two people is gone.\n\nProvide better error messages: In this particular case be explicit and\nlet the code produce a message like: \"to access the data you need to be\nin one of the following groups: Baywatch, Admin, ...\".\n\nSoftware security experts might disagree. I disagree with their disagreement.\nHiding the facts is just \"Security through obscurity\".\n\n### \"false\" means failure... The root cause is gone.\n\nIn the early days, when the C programming language was predominant, it was common for the return value of a method call to specify\nwhether the call was successful or not.\n\nI run a [Nextcloud](https://nextcloud.com/) server, but\nthe synchronization fails for some files. In the logs, I see that GenericFileException()\ngets thrown. Let's have a look at these lines. \n\n```\nif ($this-\u003eview-\u003efile_put_contents($this-\u003epath, $data) === false) {\n    throw new GenericFileException('file_put_contents failed');\n}\n```\n\nI try to find the implementation of `file_put_contents()` and see that it\nis implemented 19 times! There several different backends (ObjectStore, Local, DAV, ...)\n\nThe error message is completely meaningless: \"file_put_contents failed\".\n\nThe root cause is unclear. I would like to know more. I want to know why it failed.\n\nDebugging this would be much easier if `file_put_contents()` would throw\nan exception on failure instead of returning `false`.\n\nThe [man page of errno](https://man7.org/linux/man-pages/man3/errno.3.html) lists all the common errors which can happen.\nIt would help me if I would know if it is EDQUOT (Disk quota exceeded), ENAMETOOLONG (Filename too long), ENOSPC (No space left on device) ...\n\nHow would the above application code look like, if `file_put_contents()` would raise an exception instead of returning `false`?\n\nIt would be much simpler:\n\n```\n$this-\u003eview-\u003efile_put_contents($this-\u003epath, $data);\n```\n\nNext issue with returning \"false\" on error: I guess there are several calls to `file_put_contents()` which don't check the return\nvalue and silently don't realize that something failed.\n\nGuideline: Use exceptions to signal that something went wrong.\n\n\n### Avoid clever guessing\n\nThese days I needed to debug a well-known Python library. It works fine,\nbut you don't want to look under the hood.\n\nOne method accepted an object with three different meanings types as\nthe first argument:\n\n-   case1: a string containing HTML markup\n-   case2: a string containing a file path. This file contained the HTML\n    to work on.\n-   case3: a file descriptor with a read() method.\n\nThis looks convenient at the first sight. But in the long run, it makes\nthings complicated. This kind of guessing can always lead to false\nresults. In my case, I always used case1 (it contained a small HTML\nsnippet). But, once the string was accidentally the name of an existing\ndirectory! This crashed, because the library thought this is was a\nfile...\n\nConclusion: STOP GUESSING.\n\nIn Python, you can use classmethods for alternative constructors.\n\n``` {.sourceCode .}\n# case 1\nobj = MyClass.from_string('.....')\n\n# case2\nobj = MyClass.from_file_name('/tmp/...')\n\n# case3\nwith io.open('...') as fd:\n    obj = MyClass.from_file_object(fd)\n```\n\n### Don't stop with \"permission denied\"\n\nIn most non-trivial projects there are several reasons why the\npermission was denied.\n\nIf you (the software developer) only return \"permission denied\", then\nthe user/admin doesn't know the **reason**.\n\nIf you add a reason, then it is more likely that the user/admin can help\nthemselves.\n\nThis means they don't call you, our teammate, to solve this.\n\nFewer interruptions for your and happy customers, it's easy.\n\nOr more general: Add enough information to error messages, to make it\neasier to understand the current situation.\n\nFor example, you can add hyperlinks to docs/wiki/issue-tracker in you\nerrors messages.\n\n### OOP: Composition over inheritance\n\nIf unsure, then choose \"has a\" and not \"is a\".\n\n\u003chttps://en.wikipedia.org/wiki/Composition_over_inheritance\u003e\n\n### Cache forever or don't cache at all\n\nCaching is like a false friend or a drug. It makes you happy today, but\nin the long run, it brings you a headache.\n\nCaching is easy, but cache invalidation is hard.\n\nThat's why the pattern \"cache for every\" is handy: You don't need\nto invalidate anything.\n\n\u003e [Two Hard Things](https://martinfowler.com/bliki/TwoHardThings.html): There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton\n\n\nAvoid \"maybe\". If your HTTP code returns a response you have two choices\nconcerning caching:\n\n-   the web client should cache this response forever.\n-   the web client should not cache this response at all.\n\nIf you follow this guide you will get great performance since\nrevalidation and ETag magic is not needed.\n\nI possible, avoid fiddling with ETag and If-Modified-Since HTTP headers.\n\nBut you have to care for one thing: If you cache forever, whenever you\nupdate your data, you need to give your resource a new URL. That's easy:\n\nFor example: Instead of serving the file `/css/base.css` you serve `/css/base.27e20196a850.css`. The string \"27e20...\" is the md5 sum of the content of the file. Configure your webserver to serve this file with the appropriate \"cache forever\" headers, and your client will not ask for this file again. \n\n\nIf you use Django, you can use the [ManifestStaticFilesStorage](https://docs.djangoproject.com/en/3.0/ref/contrib/staticfiles/#django.contrib.staticfiles.storage.ManifestStaticFilesStorage)\n\n\n[Best Practices for Speeding Up Your Web Site (Yahoo)](https://developer.yahoo.com/performance/rules.html)\n\nA good introduction to caching: [Caching (Mozilla Foundation)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching)\n\nAn other good article about caching [Caching Header Best Practices](https://simonhearne.com/2022/caching-header-best-practices/)\n\n### Robust Cache-Invalidation: Database Index\n\nA database index is like caching: Redundant data gets created to achieve faster lookups.\n\nIf possible, use this robust caching and cache-invalidation provided by a database index instead of creating your implementation.\n\nRelated: https://www.postgresql.org/docs/current/indexes-expressional.html\n\n### CDN for private data?\n\nYou might think unguessable URLs are enough to protect data. Only people with the URL can access it.\n\nNo, likely, your customers don't like this. The URL could be shared easily. \n\nYou might argue that unguessable URLs are fine since an evil user could\ndownload and upload the content, but this is a different case.\n\nMy rule of thumb: Put only public data on a CDN.\n\nNginx and Apache have this feature called [X-Sendfile](https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/). This\nhandy, since you can do authentication and permission checking in your application and tell the webserver to serve the blob data.\n\n### Avoid coding for one customer\n\nTry to avoid writing software just for one customer. If you write code\nfor one customer, you miss the great benefit of software: You can write\nit once and make several customers happy. Of course, every business\nstarts small. But try to create a re-usable product soon.\n\n# Release daily.\n\n[Release early, release often](https://en.wikipedia.org/wiki/Release_early,_release_often)\n\nContinuous Integration is nowadays done by almost everybody. But Continuous Deployment not.\n\nPush it! If you are able to release daily your development speed will increase, and you\nhave less pressure that a change did not make it into the current release.\n\nAnd if something brakes, than your change set which I need to debug is much smaller. \n\nSince you can't release native apps daily, I avoid to develop native apps.\n\n\n### Misc\n\n-   [Rough consensus and running code](https://en.wikipedia.org/wiki/Rough_consensus)\n\n### Background tasks should preserve Buffer Cache State\n\nYou should know what this article talks about. But of course, you don't\nneed to recall every detail.\n\n\u003chttps://insights.oetiker.ch/linux/fadvise/\u003e\n\n\u003chttps://github.com/Feh/nocache\u003e\n\nUse case: you use rsync to backup a Linux machine. The rsync process\nshould not slow down the production environment.\n\nBy default Linux thinks \"A process just read file 'foo'. Let's keep the\ncontent in the buffer cache\". But rsync runs in the background and it does\nnot touch the same file twice.\n\nIt makes no sense to store the files which get read by rsync in the\nbuffer cache. The buffer cache should be available for the production\nenvironment.\n\n\n### Avoid POSIX locale\n\nAvoid the [locale](https://en.wikipedia.org/wiki/Locale_(computer_software)). It causes your code to behave differently in different environments. Your code might be working during development and CI. But it might fail in production if there is a different locale active.\n\nIt is broken by design: First, you call `setlocal()` and after that methods do different things. That's stateful and confusing.\n\nIt does not follow the simple input-processing-output model.\n\nI wasted too many hours with it. For example: [SAP PyRFC Bug #142](https://github.com/SAP/PyRFC/issues/142)\n\nA library should always return the same output given the same input. The result should not\nbe different for different locales.\n\nAnd the GUI? The user wants to use her/his favorite language. But native GUIs fade, and web GUIs come. And modern\nweb GUIs don't use locale anymore. Here you see \"i18next vs locale\" on [Stackoverflow tag trend](http://sotagtrends.com/?tags=[i18next,locale])\n\n### Datetime class vs \"seconds since 1970\"\n\nThere are two ways to work with dates:\n\n* old: Seconds since 1970 (the [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time))\n* new: Datatypes like [datetime](https://docs.python.org/3/library/datetime.html#datetime.datetime)\n\nIf unsure use the new datatypes. Don't fiddle with seconds since 1970 anymore.\n\nException: Since file systems store the mtime (modification time) in \"seconds since 1970\" and you only want the age of the file in seconds, then it is simpler to stick to the old way. Related: [getmtime() vs datetime.now()](https://stackoverflow.com/questions/58587891/getmtime-vs-datetime-now)\n\n\n### Statistical Profiler\n\nDebugging and profiling are easy in a development environment. But how to debug a running production system?\nA [Statistical Profiler](https://en.wikipedia.org/wiki/Profiling_%28computer_programming%29#Statistical_profilers) (or Sampling Profiler) is very cool. Every N millisecond the stack traces of the processes get dumped. This does not slow down your production environment at all. These dumps can reveal interesting facts. In which source code lines does the running application spend the most time?\n\nThere are commercial tools and some open-source tools.\n\nFor Python, there is [py-spy](https://github.com/benfred/py-spy) to dump the stack traces. The dumps can get analyzed by [speedscope](https://github.com/jlfwong/speedscope).\n\nFor Kubernetes there is [Parca](//parca.dev)\n\n### Use \"main\" for development\n\nYou confuse newcomers if your development branch has a different name. If you call the development branch \"main\", then all introduction material at Github does apply. And if your code is at Github, all people can see that your project is still alive, since the main branch gets displayed per default.\n\nAnecdote: The [tinelic](https://github.com/sergeyksv/tinelic/) project did all the coding in the \"development\" branch. The main branch was not updated for three years. I thought this project was dead. The maintainer was upset because he recently pushed changes into this branch. See [issue #9](https://github.com/sergeyksv/tinelic/issues/9#issuecomment-558557925)\n\n### monorepo != monolith\n\nUsing a [monorepo](https://en.wikipedia.org/wiki/Monorepo) can be handy. You have one git repository which contains several projects.\n\nThis does not mean you need to run a [monolithic application](https://en.wikipedia.org/wiki/Monolithic_application).\n\n\n\n------------------------------------------------------------------------\n\n## 4. CI/CD\n\nContinuous Integration, Continuous Deployment\n\n### CI++\n\nContinuous Deployment enhanced:\n\n* run tests (everybody does this)\n* check coverage. Allow merge to the main branch only if the coverage is about a treshold. You can start with a low treshold and increase it every month slightly until you reach feasible value\n* check code quality\n\n### Canary release\n\n\n\u003e Canary release is a technique to reduce the risk of introducing a new software version in production by \n\u003e slowly rolling out the change to a small subset of users before rolling it out to the entire \n\u003e infrastructure and making it available to everybody.\n\nSource: [martinfowler.com](https://martinfowler.com/bliki/CanaryRelease.html)\n\nImagine you have two versions: \n\n* Version A (the stable mainstream version)\n* Version B (the version containing a new feature)\n\n\nThere are three ways you can implement the switch between the versions:\n\n* Via routing: You have one system and several servers. Some servers use version A, some version B. A router decides which server should handle the request\n* Via system: You have several systems (for example one system for each customer). You can update the system of one customer to version B, while the other customers use version A.\n* Via code: The source code contains both implementations. According to some conditions either code version A or code version B gets executed. You usually use a feature-flag for this. This feature-flag can be set per user (or per customer).\n\nFor example, you updating an internal tool, but you don't want to interrupt the work of all developers.\n\nSince it is just for internal tooling, you can implement canary releasing in an early adaptors list like this:\n\n```\nif user in ['thomas', 'peter', 'hugo', ....]:\n    #### new way\nelse:\n    #### old way\n```\n\nThis way you ensure to don't annoy colleagues with your great new, but not yet mature features.\n\nExample 2: You can use a mixture of \"via system\" and \"via code\". I guess you have some system-wide configuration\nin your database. Then you can deploy the same source code to all systems and use code like this:\n\n```\nif system_config.feature_flag_foo:\n    ### new way\nelse:\n    ### old way\n```\n\nI like the \"via code\" way because this simplifies your CI/CD. You only have one current version which gets deployed to all places. Of course, this gets a bit more difficult if your change requires a database schema update.\n\n\n### fewer conditions, less code, fewer bugs\n\nI don't know if this is fake or real, but I guess it is true. In [this article from a former Oracle developer](https://news.ycombinator.com/item?id=18442941)\nyou can read the reason why developing on the huge codebase is hard. It is not the size, it is the number of flags.\n\nFlags are nice, but they introduce complexity. Every flag is a condition. It changes the environment and the simple Input-Processing-Output method\ndoes not work anymore. You have the explicit input and the indirect input from the environment.\n\nIf you can avoid conditions, then do it. Conditionless is the goal.\n\n### Avoid to catch all exceptions\n\nThis is good:\n\n```\ntry:\n   something()\nexcept SpecificException as exc:\n   ...\n```\n\nThe try..except block is small.\n\nThe `except` catches a specific exception.\n\nDon't catch all exception. More about this in [The Most Diabolical Python Antipattern](https://realpython.com/the-most-diabolical-python-antipattern/)\n\n### Live at head\n\n[Semantic Versioning (SemVer)](https://semver.org/) is well-known because it promises stability.\n\nBut if it is useful or slowing down, depends on the context.\n\nIf you develop something for end-users, the version plays no big role. Do you know the version of your mail-user-agent (gmail, thunderbird, ...) or browser? \nUsers should not need to care about that.\n\nOn the other hand, if you develop a software library, or if you provide APIs, then versioning, release notes and a clear deprecation time-line are important.\n\nRelated: [Philosophy of Abseil](https://abseil.io/about/philosophy)\n\nRelated: [Does SemVer work?](https://books.google.de/books?id=V3TTDwAAQBAJ\u0026pg=PA445\u0026dq=does%20semver%20work) [Software Engineering at Google](https://www.oreilly.com/library/view/software-engineering-at/9781492082781/)\n\nBTW, I like [Calendar Versioning](https://calver.org/) with [BumpVer](https://pypi.org/project/bumpver/).\n\n### How to structure libraries?\n\nIn the past we had a handy (closed source) library called `djangotools`. It contained a lot of methods which helped us to re-use code in different projects.\n\nBut there was a problem: The methods in djangotools needed third-party libraries. For example:\n\n* for reading excel files our methods needed xlrd\n* for thumbnailing images our methods needed Pillow\n* ....\n\nThis means the list of dependencies of our library djangotools got very long.\n\nIf a project needed a single small method from djangotools, it needed to install the dependecies of djangotools. Diskspace\nis cheap, but nevertheless it eats your time in the long run if you bloat your projects.\n\nHow to solve this?\n\nThe solution sounds easy: Create serveral libraries, not one huge library.\n\nBut how to structure libraries?\n\nNow I know that it makes sense structure the libraries by their dependecies.\n\nExamples:\n\n* xlrd_utils\n* pillow_utils\n* ....\n\nThis way a project which needs a method from pillow_utils just need Pillow, and not xlrd.\n\n\n### List of API Types\n\n#### Client: Human or machine-to-machine?\n\nYou create applications? You create a server. Whom do you serve? \n\nThere is a huge difference between creating an API for a GUI for humans and an API for a machine-to-machine communication.\n\nAn machine-to-machine API needs to be stable. If you create a new version, you usualy support the old version for some months in parallel to the new API.\n\nIf you create a GUI, then you do fancy things like A/B testing.\n\n#### IPC\n[IPC (Inter-Process Communication](https://en.wikipedia.org/wiki/Inter-process_communication)\n\nFor example you can configure an Nginx webserver to talk to your Python code via an unix-domain socket. Or you can access your PostgreSQL database running on localhost via an unix-domain socket.\n\n#### Command line\n\nCommand line tools are like an API. The command receives structured input and creates output.\n\n#### TCP/UDP\n\nFor example Wireguard uses UDP to create a VPN.\n\n### http based \n\nRESTful\n\ngrpc\n\n#### Library\n\nA library can provide methods which you can use. For example the Python Standard Library provides many usefull methods which make your live easier.\n\n### GraphQL / SQL\n\nI have not found a matching generic term for this.\n\n#### Conclusion?\n\nMost people think \"API\" mean \"http\". But that's not true. \n\nFor me, as a developer a library provides more value than an http API.\n\nFor example: Reducing the size of an image. If I have library which reduces\nthe size of an image, I have a handy tool. An http API which provides me\nthis services is not that easy. I need bandwith, what happens if the service is down,\nI need to pay the bill for the service, ....\n\n\n\n------------------------------------------------------------------------\n\n## 4. Remote APIs\n\n### Don't split Backend and Frontend\n\nDon't split frontend and backend into two seperare git repos. This will make things more\ncomplicated and will slow down your development.\n\nSoon your frontend code will need a change in the backend. If you have one git repo, then\nyou don't need to handle the dependency. You can be sure if the changes in the backend\nand the changes in the frontend get into the main branch in one Pull-Request, then\nyou don't need to handle the dependency.\n\nIf you have two independent git repos, than you need to handle the dependency somehow. This \nis a interesting task. I guess some developers will like this challenge. They will invest\na lot of time, energy and emotions into this interesing task ..... And the clock on the\nwall goes tick-tack-tick-tack. And you customers won't notice and won't honor all the effort.\n\nI think you gain nothing if you split the backend code and the frontend code.\n\n### Use http, avoid ftp/sftp/scp/rsync/smb/mail (to import data)\n\nUse http for data transfer. Avoid the old ways\n(ftp/sftp/scp/rsync/smb/mail).\n\nEspecialy if the sender is a different company or different department.\n\nThe old protocols ftp/sftp/scp/rsync/smb/mail don't let you validate the\ndata you receive.\n\nThis means invalid data can be send to you, and then you are responsible\nto care for the invalid data. This is usually manual work which causes\nuseless interruptions.\n\nIf it just a simple file backup, then these protocols (except mail) might\nbe ok. But if you want to impport the data into your system, and\nthe data could be invalid, then avoid these old protocols.\n\nIf you want to transfer files via HTTP from shell/cron you can use:\n[tbzuploader](https://github.com/guettli/tbzuploader). This way\nthe sender can call a simple script, and the receiver can validate\nthe data before accepting it.\n\nThe next step is to avoid clever\n[inotify](https://en.wikipedia.org/wiki/Inotify)-daemons. You don't need\nthis anymore if you receive your data via HTTP.\n\n\n### Avoid Polling\n\nPolling means checking for new data again and again. Avoid it, if\npossible. Try to find a way to \"listen\" for changes. In most databases,\nyou can execute a trigger if new data arrives.\n\n### Provide specific import directories, not one generic\n\nIf you still receive files via FTP/SCP since you have not switched to\nHTTP-APIs yet, then be sure to provide specific input directories.\n\nIn the past, I received files in a directory called \"import\". Several\nthird-party systems sent data to this directory. It looks easy in the\nfirst place. But sooner or later there will be chaos since you need to\nknow where the data came from. Was it from third party system FOO or was\nthe data from the third party system BAR? You can't distinguish anymore if\nyou provide only one import directory.\n\nNow we provide import-FOO, import-BAR, import-qwerty ...\n\n### Don't set up an SMTP daemon\n\nIf you can avoid it, then refuse to set up an SMTP daemon. If the\napplication you write should import mails, then do it by using POP3 or\nIMAP and poll for new mail N seconds. Setting up an SMTP daemon is easy,\nbut being responsible for it is effort. Dealing with attacks, keeping an\neye on security announces... Live is easier without being responsible\nfor an SMTP server.\n\nAn SMTP daemon needs to run 24 hours a day. You get into trouble if it is\ndown. Or even worse: it is misconfigured and rejects all mails. These\nemails get lost and won't come back.\n\nIf the [getmail](http://pyropus.ca/software/getmail/) job is down or is misconfigured it just won't fetch\nmails. But it is unlikely that mails get lost.\n\nI know this conflicts with the general guideline \"avoid polling\".\n\n### Don't do Virus/Malware/Spam detection\n\nVirus/Malware/Spam detection and prevention is an endless battle. It will take a lot of time\nto do it properly. So if possible, avoid the responsibilty for it. Except\nis is your main task and you have customers who are willing to pay.\n\n------------------------------------------------------------------------\n\n## 5. Op\n\nOperation. The last two characters of DevOp.\n\n### Configuration Management\n\nIn the past configuration management tool like Ansible, SaltStack, Puppet or Chef\nwhere used. Roughly since 2020 they are less important.\n\nConfiguration management is great for updating servers. But there are fewer\nservers these days which get updated. Code runs in containers. \n\nTo configure a container, you don't need fancy configuration management.\n\nA simple conditionless (no \"if\", no \"else\", no \"for\") shell script (with `set -e`) \nis enough to create a container image.\n\n#### The magic reload feature of Config Management is not needed anymore\n\nConfig management tools have magic reload feature. Imagine you update\nthe configuration of a webserver. The config management tools can detect\nif a restart of a server is needed or not. For example: If the\nconfiguration of the webserver was changed, then the webserver gets\nreloaded. If the configuration of the webserver was not changed, then\nthere is no need to restart it. Great feature?\n\nAnsible Docs: [Handlers: Running Operations On\nChange](https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html#handlers-running-operations-on-change)\n\nI liked this feature in the past.\n\nTime has changed.\n\nSee above for \"From CRUD to CRD\". Kubernetes is coming. You create\ncontainers, you run containers, you delete them. You don't update them\nanymore.\n\nThe magic reload feature is not needed anymore.\n\nOf course, this change from stateful servers to stateless containers\ndoes not happen from one day to the next. One thing is sure: Stateful\nservers and the need to reload running services after an update will\ndecrease.\n\n### Config Management: Change file vs put file\n\nOften there are two ways to do configuration management:\n\n-   change a part of a file: \"replace\", \"append\", \"patch\"\n-   put a whole file under configuration management.\n\nYou have far less trouble if you use \"put a whole file\". Example: Do not\nfiddle with the file /etc/sudoers. Put a whole file into\n/etc/sudoers.d/.\n\n### Config Management: No need for custom RPMs/DPKGs\n\nIn the past, it was common to create a custom RPM or Debian package to\ninstall a file on a server.\n\nFor example an SSL cert.\n\nIf you have a configuration management tool, then this extra container\n(RPM/DPKG) does not make much sense.\n\n### Cron Jobs\n\nA server exists to serve. If the server does not receive requests, why\nshould the server do something? This results in my rule of thumb:\nAvoid cron jobs.\n\nSometimes you need to have a cron job for housekeeping.\n\nKeep cron jobs simple.\n\nIn general, there are two ways to configure the arguments of a cron job:\n\n-   the command line arguments which are part of the crontab line\n-   additional source of configuration: config files or config from a\n    database\n\nAvoid mixing these two ways of configuring a cron job. I prefer to\nconfigure the cron job via the latter of both ways. This keeps the cron\njob simple. My guideline: Do not configure the cron job via optional\ncommand-line arguments. Only use the required arguments.\n\n### SSH to production-server\n\nI still do interactive logins to production remote-server (mostly via\nssh). But I want to reduce it.\n\nSooner or later you will make a typo. See this article from GitLab for an\nexciting report on what happened during a denial of service:\n\u003chttps://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/\u003e\nWe are humans, and humans make mistakes. Automation helps to reduce the\nrisk of data loss.\n\nIf you are doing \"ssh production-server ... vi /etc/...\" or \"... apt\ninstall\": Configuration management is much better. For example ansible.\n\nIf you are doing \"ssh production-server .... less /var/log/...\": No\nlog-management yet? Get your logs to a central place.\n\nIf you are doing \"ssh production-server ... rm ...\": Please ask yourself\nwhat you are doing here. How can you automate this, to make this\nunnecessary in the future?\n\n### VPS vs Kubernetes vs PaaS\n\nThere are several ways to execute your code and make your application available to the public.\n\nVPS: Virtual Private Server. These are cheap, you can get one for 3 Euro per month.\nBenefit: You can install and configure it right the way you want it to be.\nDrawback: You need knowledge about Linux.\n\nKubernetes: This is the current hype. It's great for huge data centers... but ...\ndo you have a huge data center?\n\nPaaS: Platform as a Service. For the example provided by Heroku, Google, Amazon. They try\nto make your life easier.\nPro: easy to use.\nCon: more expensive.\n\nMy hint: if unsure use PaaS. If you want to learn the basics of running a server, then use a VPS.\n\n\n### Keep your directories clean\n\nThere are two kinds of files in the context of backup: Files that should\nbe in the backup and temporary files which should not be in the backup.\nKeep your directories clean. In a directory, there should be either only\nfiles which should be in the backup xor only files which should not be\nin the backup. This will make life easier for you. The configuration of\nyour backup is easier and cleaning temporary files is easier and looking\nat the directory makes more joy since it is clean.\n\nAnd overall, storing data in directories and files is outdated. If you\nstart from scratch, then put structured data in a database and binary data in an S3\ncompatible server.\n\n### Avoid logging to files\n\nI still do this, but I want to reduce it. Logs are endless streams.\nFiles are a bunch of bytes with a fixed length. Both concepts don't fit\ntogether. Sooner or later your logs get rotated. Now you are in trouble\nif you want to run a log checker for every line in your log file. I mean\nthe mathematical version of \"every line\". This gets complicated\nif you want to check every line. Rotating log files needs to be done\nsooner or later. But how to rotate the file, if a process still writes to\nit? This is one problem, which was solved several hundred times and each\ntime different ...\n\nIn other words: Avoid logging to files and avoid logrotate. Logging is\nan endless stream.\n\nOf course, somewhere on the hard disk data gets stored in files. But it is highly\nrecommended using a tool where don't fiddle with files daily.\n\nSee [12factor App: #11 Treat logs as event streams](https://12factor.net/logs)\n\n### Logging: Symptom vs root-cause\n\nFiltering or ignoring errors is easy. People love it. But wait, why not fix the root-cause?\n\nOne example from hosting a Django application on a VPS:\n\nYou will see these messages soon after you set up your VPS:\n\n\u003e DisallowedHost at / Invalid HTTP_HOST header: '198.211.x.y'. You may need to add '198.211.x.y' to ALLOWED_HOSTS.\n\nI guess you don't want your site to be accessed via IP, but some script-kiddies scan all public accessible\nIPs all the time and check if http-host-header-validation is active.\n\nThe easy solution: Extend the ALLOWED_HOST setting and add your IP. But this does solve the issue just for some hours.\nSoon some script-kiddies will use fakted HTTP_HOST header and you get useless warnings again.\n\nWhat the next step? You use google and find a way to ignore the annoying error messages.\n\nYou could solve this by filtering the logs. You could ignore all `django.security.DisallowedHost` logs.\n\nNow to the heading of this chapter: \"Symptom vs root-cause\"\n\nFiltering logs just hides the symptom.\n\nFixing the root-cause means to configure the webserver in front of Django (for example Nginx)\nto handle the broken request. These broken requests should not be forwarded to Django,\nand then you don't need to add IP addresses to ALLOWED_HOSTS or ignore logs.\n\n### Use Systemd\n\nIt is available, don't reinvent it. Don't do double-fork magic anymore.\nUse a systemd service with Type=simple. See [Systemd makes many daemons\nobsolete](https://stackoverflow.com/a/30189540/633961)\n\n### Don't use Systemd \"instantiated units\"\n\nSystemd allows you to create a template and create several services from\nthis template. See: \u003chttp://0pointer.de/blog/projects/instances.html\u003e\n\nFirst, I thought this was great. But some months later I realized: It is\nbetter to have one source for templates: Your configuration management.\nIf you want several almost equal services, then use templates in your\nconfiguration management.\n\nThis makes it simpler.\n\n### etckeeper is handy\n\nIf you are using Kubernetes, then this makes no sense. But if you\nare running services in a Linux server, then you want to know\nwhat has changed?\n\nThe tool etckeeper stores changes in the /etc directory in a git\nrepository. This does not make much sense for containers. But for\nservers that live several weeks, it makes sense. You don't need to push\nthe changes to a different location. It is very handy. Example:\n\n    cd /etc/apt\n    git log .\n    \n    ---\u003e you see all git commits which changed files in the /etc/apt directory.\n\nBut etckeeper is no backup tool. It is just a handy tool to see what has\nchanged and when this change happened.\n\nWe wrap it and dump additional information into /etc/etckeeper/extra/\nbefore \"git commit\". We add: /var/spool/cron, output of hwinfo, lsblk,\nfdisk, pvdisplay, vgdisplay, lvdisplay, dpkg/rpm package list, postgres\nconfig.\n\nDoes it make sense to add the output of df into /etc/etckeeper/extra/?\n\nI think it makes no sense since this changes daily. If no change was\nmade to the configuration, then there should be no commit in /etc/.git.\n\n### If you do coding to implement backup ...\n\nIf you do coding/programming to implement your backup of data, then you\nare on the wrong track.\n\nYou will likely do it wrong, and this will be a big risk.\n\nWhy? Because you will notice your fault if you try to recover your data.\n\n**Use** a backup tool, even if you love to do programming. **Configure**\nit, but don't write it yourself.\n\n### Avoid re-inventing replication\n\nThat's what the customer wants you to implement:\n\nYou should transfer data from database A to database B. Every time there\nis an update in database A, data should get copied to database B.\n\nSlow down: What you are doing is replication. Replication creates\nredundancy and redundancy need to be avoided.\n\nWhy do you want redundancy in your data storage? The only reasons I can\nthink of are speed/performance and fault-tolerant (like DNS/LDAP).\n\nIf replication is needed, then take the replication tools the\ndatabases offers. Do not implement replication yourself. This is not\ntrivial and experts with more knowledge than you and me have solved this\nissue before.\n\n### Master-Master Replication\n\nThe real magic is Master-Master Replication. Here are some examples where it gets used:\n\ntodo: add examples\n\n### Data transfer rates\n\nIt makes sense to have some rough numbers for rough estimates.\n\nUSB-2: 1.9 GByte per minute (rsync from PC to external hard disk). 3.6 GByte per minute (theory)\n\nOffice: Download: 200 Mbit/s, Upload: 20 Mbit/s\n\nHome: Download 60 Mbit/s, Upload 12 Mbit/s, Latency 10 ms\n\nRelated: [List of interface bit rates](https://en.wikipedia.org/wiki/List_of_interface_bit_rates)\n\n## 6. Networking\n\n### No routing on servers\n\nImagine there are 20 servers in your network. Imagine there are two\nnetwork routes. One route goes to a second internal network and the\nother route goes to the internet. All 20 servers should be able to\naccess both networks. There are two ways to solve this:\n\n-   V1: Each of the 20 servers have the two routes configured.\n-   V2: There is one default gateway for the 20 servers. Every server has one route. (The common term is \"default gateway\")\n\nPlease choose V2. It is simpler, it is easier to understand, it is less\nerror-prone, it is saner.\n\n### traceroute won't help you\n\nIf you have trouble with a TCP connection, then use tcptraceroute. It\ncan help to find the firewall which blocks your IP packages trying to\nget from host A to host B. Again \\*tcp\\*traceroute. It is the tool for\nTCP connection tests (HTTP, HTTPS, ssh, SMTP, pop3, IMAP, ...). Reason:\nnormal traceroute uses UDP, not TCP.\n\n## 7. Monitoring\n\n### Nagios Plugin API (0=ok, 1=warn ...)\n\nWriting Nagios like checks is very simple. The exit status has this\nmeaning:\n\n-   0: ok\n-   1: warn\n-   2: error\n-   3: unknown\n\nIs this KISS (keep it simple and stupid)? Yes, I think it is **simple**.\nYou can write a Nagios plugin with any language you like. Often less\nthen ten lines of source code are enough to implement a Nagios check.\n\nBut on the other hand, it is not **stupid**. The checks do two things:\nIt collects some numbers (for example \"How much disk space is left\") and\nit does evaluate and judge (\"only N MByte left, I think this is a\nwarning\"). That's not stupid this is some kind of intelligence.\n\nAfter writing and working with Nagios checks for several years I think\nthe evaluation of the data should not be done inside the check. Some\ndata-collector should collect data. Then a different tool should\nevaluate the data and judge if this ok, warn, or error.\n\n### Checks vs Logs\n\nChecks are mostly for operators and logs are mostly for developers.\n\nSince there are always some temporary network failures, checks help more\nthan logs do.\n\nExample:\n\n1.  yesterday night at 3:40 there was a temporary network failure and this results in log messages.\n2.  At 3:45 the network failure was gone.\n3.  You look at the log message at 9:15. You don't know: Is this message still valid?\n\nChecks get executed again and again.\n\nIf a check fails at 3:41 it will be ok some minutes later.\n\nThen you know immediately that there was **temporary** failure.\n\nLogs are important for developers for debugging.\n\nBut in this case, the developer can't do anything useful. Temporary\nnetwork failures happen again and again. That's live. Looking at the log\nwhich was created by a temporary network failure wastes the time of the\ndeveloper.\n\nLogs should contain the stack trace and the local variables of each frame\nin the stack trace (a tool like sentry could be used), if real errors\noccur.\n\n### Logs vs Rows\n\nImagine you have a application-to-application synchronization process. Data from\nsystem A needs to be pushed to system B. It is a custom synchronization and\nyou won't support this syncing, you just want to develop it. After developing this plugin\nfor system A you want it hand over to the customer.\n\nOf course there are several error-conditions which can occur. The network between\nboth systems can be broken. One system can be down for some minutes, or there\nis data which does not validate ....\n\nIf you implement traditional logging you have a time-stamp and some additional \ndata like error messages or snippets of data which provide additional information.\n\nYou could dump this log together with thousand completely unrelated logs into NoSQL\nbased logging solution. The Logs coming from different sources have nothing in\ncommon, except that they accidently have all a timestamp and a message.\n\nDoes this provide the best possible user experience? I don't think so.\n\nThere is big gap between system A and the important logging information.\n\nImagine ther is a data-set called \"foo\". This data-set could not get synced.\nOf course system A has a page for \"foo\". \n\nThe best user-experience would be a link from the page \"foo\" of system A to the\ncurrent state of the sync.\n\nHow to implement this? It is easy: Stop logging, start creating rows in a table.\n\nIf you have a table and rows in system A, then it is easy to provide a useful interface\nfor the operator of system A to see what's going. The big gap is gone. Information\nis visible where you need it.\n\nOf course this does not apply to every kind of logging. It hardly makes sense to \nwrite every http request/response into a database table. \n\n\n## ???. Web-Development\n\n### HTML and CSS over JS\n\nSee [Bootstrap 5 \"HTML and CSS over JS\"](https://v5.getbootstrap.com/docs/5.0/extend/approach/#html-and-css-over-js)\n\n### Use VanillaJS\n\nVanillaJS is a very cool framework. By the way, it is a joke. VanillaJS means \"Use Javascript without a framework\".\n\nSome think modern web applications need to use React or Vue. I don't think so. It is perfectly fine to send [html-over-the-wire](https://github.com/guettli/html-over-the-wire) and add some JS if needed.\n\n### Validate Forms on the Server\n\nYou need to validate your form on the server anyway. So why implement it on the client-side? I think that in most cases it is perfectly fine\nto validate forms only on the server and not on the client.\n\n\n### A simple homepage? SSG!\n\nIf you create a simple homepage (without the need for a database), then an SSG (Static Site Generator) might be enough. A CMS is not needed.\n\nSee [Liste of Static Site Generators](https://github.com/guettli/static-site-generators)\n\n\n\n## ?. Sofware Lifecycle\n\n\n## Don't waste time supporting old software.\n\nDon't waste time supporting old software. The clock is ticking faster today than in the past.\n\nMy rule of thumb: There is no need to support software which is older than the Ubuntu LTS (long-term-support) of last year.\n\nExample: \n\nToday is July 2021. Last year was July 2020. The current Ubuntu LTS of July 2020 was 20.04. This shipped with Python 3.8.\nThis means for me: Today (July 2021), there is no need to support Python 3.7 or lower. \n\nThis applies if you create a library which gets used by other parties.\n\nIf you run an application which is only run in an environment which you control, then do what you want.\nThere is no need to support older versions.\n\nBut on the other hand this means: Your software needs to run on the software stack of the Ubuntu LTS\n12 months ago. Don't fall behind this. If your software stack is older then the Ubuntu LTS of last year,\nthen you should upgrade your stack.\n\nExample: Ubuntu 20.04 shipped with PostgreSQL 12.8. This long-term-support version of Ubuntu was released in April 2020. This means\nthat you should upgrade your applications to at least PostgreSQL 12.8 around April 2021. Otherwise your\nstack gets too old.\n\n------------------------------------------------------------------------\n\n## 8. Communication with others\n\n### Avoid getting a nerd\n\nIf you do \"talk\" with software to databases and APIs daily, your ability\nto communicate with humans might decrease.\n\nYou might start to think like a computer (at least a bit).\n\nThe human mind works completely differently, not just bits and bytes. It\nhas [Emotions](https://en.wikipedia.org/wiki/Emotion)\n\nAvoid getting a Nerd https://en.wikipedia.org/wiki/Nerd\n\nHere some hints:\n\n-   Nerds like complaining. This book can help: \"Rethinking Positive\n    Thinking: Inside the New Science of Motivation\" by Gabriele\n    Oettingen. The method is called WOOP.\n-   Nerds like to think about their problems first. [Nonviolent\n    Communication](https://en.wikipedia.org/wiki/Nonviolent_Communication#Four_components)\n    can help.\n-   Meet with \"normal\" people. With \"normal\" I mean people who do not do\n    IT stuff.\n-   Raise a family.\n-   Do sport. Physical health is important.\n-   Relax. Creativity raises if there is no input (no noise, no visible motion, ...)\n\n### Avoid stress\n\nStress triggers your body’s “fight or flight” response. It pushes your\nblood into the muscles. That's great if you need to jump onto the sidewalk because a fast red race car would hit you. But in your daily life,\nthis \"fight or flight\" response is hardly needed. You need the energy in\nyour brain :-)\n\nAvoid stress, relax daily.\n\nOn the other hand, stress is fun: I like tennis and long-distance\nrunning.\n\nCare for both: brain and body.\n\n### Discussion, but no progress? V1, V2, V3, ...\n\nThis and the following parts are about \"Requirement Engineering\".\n\nIf a discussion does not bring progress, then grab a pen. Start with V1. The\nletter V stands for \"Solution Variant\" or \"One strategy of several to\nget to a goal\". Find a term or short description of the first possible\nstrategy. Write it down. Then: which other ways could be used? V2, V3,\n...\n\nRember, there is always the last variant: Leave things like they are\ntoday and think about this again N days later.\n\nIf you have found several solution variants, then look at them in\ndetail. Most of the time it is useful to define the needed sequence of\nsteps. You can use the letter \"S\" for this: S1, S2, S3 ...\n\nA simple example:\n\nIn the morning, you wake up.\n\n-   V1: Go to work now\n-   V2: Do some more sleeping\n-   V3: Try to remember what you dreamed, write it down\n-   V4: Do some sports\n-   V5: Play piano\n-   V6: Recall your personal goals, what is the next step?\n-   ...\n\nIf you look at V1 in detail you get to a list of steps:\n\n-   S1: get up\n-   S2: make the bed\n-   S3: wash yourself\n-   S4: get dressed\n-   S5: eat\n-   S6: take the bike and ride to work\n\nI think the first letter (V, S) helps if you are brainstorming.\n\nThis can help if you or your team is stuck in [Analysis paralysis](https://en.wikipedia.org/wiki/Analysis_paralysis) (aka \"overthinking\" if\nyou are on your own, or endless discussions if there is a team).\n\nGetting out of thinking/talking into writing and \"naming solutions\" helps to get an actionable plan.\n\nIn a professional environment, these notes about the options and the decision can get used as an entry in the \"Decision Log\".\n\n### Avoid Office Documents or UML-tools\n\nChoose a way to edit content (use cases, specs, ...) over the internet. Use\nan issue tracking system or wiki.\n\nDon't waste time with UML tools. UML is like\n[esperanto](https://en.wikipedia.org/wiki/Esperanto). It is (in theory)\na great solution that solves a lot of problems. But somehow it does not\nwork.\n\nWrite down the high-level use case, the cardinality, and the steps.\nSequence diagrams can be simplified to enumerations: first step, second\nstep, third step ...\n\n[Sketch](https://en.wikipedia.org/wiki/Sketch_(drawing)) screenshots you\nwant to build with your team with a pen. I avoid any digital device for\nthis since paper or a whiteboard is far more real. If you\nneed the result in digital format, just take a picture with your cell\nphone at the end.\n\n### Public ChangeLog\n\nThe public ChangeLog will never be perfect.\n\nIt is in the nature of things. Some customers want to know every small change.\nSome customers only want to know the important changes. Some say that\nthey want to know every change, but then never look at it.\n\nIf you are working in a modern SaaS environment where you can\nrelease often, then you should provide a ChangeLog, but don't take it too seriously for GUI changes.\n\nExample: I see small changes in the GUI of GSuite weekly. I guess they don't have\na public ChangeLog, and I think it is not needed, too. \n\nA GUI should be self-explanatory.\nThat's important. Maybe the GUI changed, maybe not. It is like in the game chess: \nYou look at the board and you face the question of what to do next. It does not matter\nwhich chess piece was moved before. You see the board and now it is your turn.\nThe past does not matter in this context.\n\nThings are different if you provide a public API. This must not change without clear communication.\nChanges to public APIs should be announced several months in advance.\n\n\nThings are different if you are working in an environment that has more constraints\n(Medical devices, Banking, Insurance, ...), then you don't need my advice, since\nyou get told what to do.\n\n### Communication with Customers: Binary decision \"next\" or \"later\" lists.\n\nDefine \"done\" with your customers. Humans like to be creative and if\nthing X gets changed, then they have fancy ideas about how to change thing Y.\nBe friendly and listen: Write these fancy ideas down on the \"do later\nlist\".\n\nIf the customer has new ideas, let them decide: Should this be on the\n\"do list\" or the \"do later list\".\n\nIf you don't have a definition of done/ready, then you should not start\nto write source code. First, define the goal, then choose a strategy to\nget to the goal.\n\nFocus on a simple working solution first. Add optional stuff to the \"do\nlater\" list.\n\n### Tell customers what they should test\n\nI have seen it several times: Software gets developed. The customer was\ntold to test and ... nothing happens. That's not satisfying since\nsoftware developers want to hear that their work does help. If you (the\ndeveloper) provide a checklist of things to test, then the likelihood to\nget feedback is bigger.\n\nIt is wise to create this checklist for testing as early as possible. It\ntells the customer the desired result.\n\n### Dare to say \"Please wait, I want to take a note\"\n\nMost people can listen and write at once. I can't. And I guess a lot of\ndevelopers have this problem. I can only do one thing at a time. If you\nare telephoning with a customer and he has a lot of things to tell you,\ndon't fool yourself. You will only remember 4 of 5 issues. Dare to say\n\"please wait, I want to take a note\". This way, you can take care of all\nissues, which results in happy customers.\n\n\n### Avoid Gossip\n\nGossip creates an atmosphere that promotes negativity (bad karma).\nAvoid making jokes about other teammates or customers. Yes, some people do strange stuff and have strange attitudes. Making jokes\nabout them makes everything worse. Please be aware that this guideline\nhas a major drawback. Sometimes, the people around you are laughing about\na customer or a teammate in their absenc ... and you are\nthe only one who is not laughing. It is up to you how to react. Be patient.\n\nSame for irony and sarcasm. You and your friends might think it is funny. New team members and\nother people won't understand you. It is not funny, it is confusing and childish.\n\nWhat can you do if your team mates complain and laugh about customers daily? If you are an employee, then it easy. You can leave. And maybe\nyou should leave. Laughing and complaining about customers is the beginning of the end.\n\n### Avoid bike-shedding\n\nAvoid [bike-shedding](https://exceptionnotfound.net/bikeshedding-the-daily-software-anti-pattern/). Please read this link. It is fun.\nAfter you read this, you will see it happen again and again.\n\nExamples: \"Should we switch to microservices?\". This question is very broad and almost every enganged developer has his own opinion. This is a great topic for general blublublabla which might just eat your time without improving the customer satisfaction. \n\nYou can make it much more actionable by asking three questions: \n\n* Which part of our could would be a good candidate for a microservice?\n* What are the benefits and draw-backs?\n* How to implement this particular change?\n\n### Cargo Culting\n\n[Cargo Culting](https://en.wikipedia.org/wiki/Cargo_cult_programming) is everywhere:\n\n* Scrum\n* Fat JS frontends which consume JSON APIs (React, Vue)\n* Microservices\n* Forced Pull-Request reviews\n\n[You ain't gonna need it (YAGNI)](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it)\n\nAbout Microservices: \n\n\u003e #1 rule of distribute computing: Don’t distribute your computing! At least if you can in any way avoid it.\n\nSource: [The Majestic Monolith](https://m.signalvnoise.com/the-majestic-monolith/)\n\nWhat to do instead?\n\nMake customers happy, don't beautify your tech. Under the hood every machine is ugly. That's ok, as long it works reliably.\nDon't beautify internals. Improve customer experience, raise test coverage, automate boring stuff. \n\n### Microservices and Esperanto\n\nImagine you have ten years old code base which is a monolith: One git repo, one DB, one logical http service.\n\nThe overall motivation is mediocre and development feels slow.\n\nManagement wants to improve the situation.\n\nBriliant software architects provide a simple solution: Microservices and Kubernetes!\n\nThey provide presentation where the overall architecture is drawn on one slide.\n\nIt looks simple. Noone dares to disagree, because the company does not foster\nthe habit of disagreeing. Nobody has the role of playing the devil's advocate.\n\nManagement decides to go this route: Microservices and Kubernetes. \n\nEverybody does what the upper management wants. Some leave and go to a different company.\n\nHow does this relate do \"Esperanto\"? Esperanto is a constructed international auxiliary language.\nThe idea is briliant: Let's create a language that everybody understands, and then we are all \nhappy because we understand each other. There is a huge difference between briliant theory\nand practice.\n\nBack to IT: You don't change a ten years old code base with 100k lines of code to a Kubernetes\nbased Microservice in some weeks.\n\nAnd all the time and energy you invest into getting to microservices does not improve the customer\nexperience. \n\nI think instead of microservices and Kubernetes these things provide more:\n\n* release daily.\n* relaxed PR review.\n* Automated code quality checks (coverage). \n* Automated code formatting (like \"black\" for Python)\n* Teamleads need skill to foster collective intelligence and avoid groupthink.\n* Quartely goals (maybe with OKRs).\n* Get customer feedback. The application should provide a simple way for users to provide feedback.\n\n### Instead of Microservices?\n\nI think a great web applications has three types of parts:\n\n* It uses a well-known open-soure web framework (for example Django).\n* The project is a monolith. In most cases this is closed-source software.\n* Many tiny libraries. Some are open-source, some are (inhouse) closed-source.\n\nThe last part is important. Imagine you want to run several projects, then putting\ngeneric code into the monolith makes it harder for you to create several projects.\n\nThis architecture works fine for 99% of all projects. If you reach the limit,\nthen you have enough money and development teams to refactor.\n\n### How to get to \"many tiny libraries\"?\n\nFirst: What is \"cookie-cutter\"?\n\n\u003e If you describe something as having a cookie-cutter approach or style, you mean that the same approach or style is always used and not enough attention is paid to individual differences.\n\nSource: [collinsdictionary.com](https://www.collinsdictionary.com/de/worterbuch/englisch/cookie-cutter)\n\nIn IT the term is often used for bootstrapping a new application. Such a tool helps you to set up your initial configuration and directory structure.\n\nThere are ready made cookie-cutter tools like [cookiecutter-django](https://cookiecutter-django.readthedocs.io/)\n\nI personally think the ready made cookie cutter tools are too bloated. \n\nIf you are new to a framework, then read their docs and follow the instructions. This should get you to your first running project.\n\nSooner or later you will create several projects, and you want to stream-line the development process (running tests, releasing new versions, uploading to a package repository, ...).\n\nThen it is time to write your own cookie-cutter. I think it is perfectly fine if every team has its own cookie-cutter scripts.\n\nWith the help of cookie-cutter scripts, you will be easily get to \"many tiny libaries\".\n\n[Anthony Sottile](https://github.com/asottile) is a good example. He has at the moment [74 projects at pypi](https://pypi.org/user/asottile/). Allthough I don't know if he uses cookie cutter scripts or not.\n\n### Dark side of Pull-Request Reviews\n\nPull-Request reviews have limitations\n\n* the person doing a PR review is not able to create new stuff during this time. He/she is blocked.\n* PR review can't detect code duplication. If you look at a PR, you have no clue if the lines which where added where written or copied.\n\nYou think PR-Review is important?\n\nPlease grab a pen and write down your reasons.\n\nNow have a look at your list of fear and concerns. What can be automated?\n\nI think PR-Review should follow these rules:\n\n* If a developer thinks a PR needs to review, then dev should be allowed to push without a review.\n* Usualy one review is enough. Two reviews should only be used for special cases.\n* Larger changes should be reviewed in a video call with two or three people. But not more than four people. These calls are usualy very productive.\n\nImagine you are developer and you spot a tiny typo. Will you change it or not? It depends on:\n\n* it depends on your emotions. Do you feel connected to the code and to the other team members?\n* the process: is it easy to fix this tiny thing?\n\nThe more fear people have that developers could break something, the more processes exist.\nThis means it is less likely is that the developer will fix this tiny thing.\nThis means in the long run the code will be dirty. This means people who care and are clever and creative will soon leave the company.\nOnly developers which don't care will stay.\n\n\n### S/N\n\n[S/N](https://en.wikipedia.org/wiki/Signal-to-noise_ratio) means \"Signal-to-noise ratio\". It is a measure used in science and engineering that compares the level of the desired signal to the level of background noise. But it can be used for reflecting on his input, too.\n\nHow many useless news do I receive daily? Is there a way to improve S/N?\n\nWhat is in my circle of influence, what not?\n\n### Sometimes you just have to speak out a wish\n\nAnecdote: In the year 2019 I loved to use the Stackoverflow Tag Trend. I wanted to\nleave my current context (Python, Django, PostgreSQL Development) and learn new stuff.\nIn the Javascript ecosystem, there were so many tools available and it was difficult\nto see which tool was great two years ago, but won't be used today if you can start from scratch. The Stackoverflow Tag Trend helped me to see what is hot and what was hot some years ago. Working 16 years for the same company I was blinded by routine and missed\na lot of changes outside my small IT context. \n\nA lot of things which I learned during studying information technology in Dresden from 1996 to\n2001 was outdated. I wrote down these things on the [Deadends of IT](https://github.com/guettli/deadends-of-it) page.\nThe  Stackoverflow Tag Trend worked well, but I had ideas to improve it:\n\n* [Tag aliases can be selected](https://github.com/robianmcd/tag-trends/issues/32)\n* I was missing [Link from tag to the description of tag](https://github.com/robianmcd/tag-trends/issues/34)\n* [URL could be simplified (no square brackets)](https://github.com/robianmcd/tag-trends/issues/33)\n* [It was down for some days](https://github.com/robianmcd/tag-trends/issues/31)\n\nAfter some weeks all my wishes were implemented by the author.\n\nWin-win: He could improve the usability and for me is more convenient now. I love it. Sometimes you just have to speak out a wish. \nYou just need to tell your wish to the right person in a friendly way.\n\n\n\n### Self-Service instead of Communication\n\nSetting up a simple workflow (maybe inside a chat tool) is easy. For example, the salesperson wants\na new demo system for a potential customer. He writes a message into a special chat channel and\nsomeone with more technical background creates the demo system. Fine, isn't it?\n\nYes, this chat-based workflow is doable and it is better than direct messages to individuals.\n\nNevertheless **self-service** is better. Wouldn't it be great if setting up a new demo system\nis simple and can be done by a salesperson who has no deep technical knowledge?\n\nCommunication is important, but daily tasks should get automated so that anyone who wants something\ncan help himself.\n\n### Architectural Decision Records\n\nArchitectural Decision Records (ADR) help to explain the \"why?\" if new team members don't understand the current state.\n\nTemplate:\n\n\u003e In the context of \u003cuse case/user story u\u003e, facing \u003cconcern c\u003e we decided for \u003coption o\u003e and neglected \u003cother options\u003e, to achieve \u003csystem qualities/desired consequences\u003e, accepting \u003cdownside d/undesired consequences\u003e, because \u003cadditional rationale\u003e.\n\nSource: [adr.github.io](https://adr.github.io/)\n\n### Missing Docs Process\n\nEvery company has processes. Only few companies have a clear process for missing or outdated docs.\n\nStart the discussion about it, and raise awareness. What should you do if you are missing docs?\n\nHow to find the owner of the current documentation you are looking at?\n\nHow to contact the owners to give them feedback?\n\n\n------------------------------------------------------------------------\n\n## 9. Epilog\n\n### It is always possible to make things more complicated\n\nIt is always possible to make things more complicated. The interesting\nadventure is to make things simpler, easier, and more obvious.\n\n### It helps to talk\n\nMost software developers do not talk much. Otherwise, they would not have\nchosen this job. If you think about something too long, then you get\nblind to the obvious and easy solution. It helps to talk.\n\nThere is something called [Rubber duck\ndebugging](https://en.wikipedia.org/wiki/Rubber_duck_debugging). This might help, but talking to humans helps much more. If you find no\nsolution in 30 minutes. Take a break. Do something different, talk to a\nteammate or friend, take a small walk outside.\n\n### Being kind\n\nImagine you ask a question in a forum and your question gets down-voted and this comment:\n\n\u003e Hardware recommendations are off-topic at this site. \n\nThat's not kind, not friendly.\n\nBut this is:\n\n\u003e You can may be try [Hardware Recommendations](https://hardwarerecs.stackexchange.com/)\n\nGuideline: If you say \"no\", then at least provide a hint where he/she could find help.\n\n### Be curious\n\nThere is always something you have not understood. Ask\nquestions, even if you think you know the answer. For one question,\nthere are always several answers. If you know one answer, then it is\nlikely that someone has a better answer.\n\nI like:\n\n-   \u003chttps://stackoverflow.com/\u003e\n-   \u003chttps://softwarerecs.stackexchange.com/\u003e\n-   \u003chttps://serverfault.com/\u003e\n-   And some mailing lists.\n\nOften I just write the question and don't write about the solution I\nhave on my mind. If you write about our solution, then the discussion is\nnarrowed to a simple pro/contra of your idea. Ask the question like a\nnewbie.\n\n### Sponsored Content in IT\n\nWhy does someone write an article on medium.com? Why does someone create content\nwhich is free?\n\nThere can be a thousand reasons.\n\nSome writers have good intentions. Some just do it to earn money by advertising something.\n\nThere are a lot of articles about AWS, Kubernetes, Azure, Docker, ElasticSearch, Graphana, Jamstack, Scrum, and other topics. Do you\nthink they all got written by volunteers who eat and breathe IT topics?\n\nSome authors share their knowledge since they like to share their knowledge.\n\nSponsored Content is everywhere: Web, Facebook, Instagram, Youtube, Medium ...\n\nMaybe it is even on Stackoverflow.\n\nI would like to pay for high-quality news and articles, but so far I have not found a channel that\n offers a rich and broad-spectrum.\n\n\n\n### Creativity Management\n\nA lot of ideas come to my mind if I am far away from a laptop or pc.\nFor example, if I cycle from home to the office or back.\n\nI started with this way of creativity management some years go: I write\na mail to myself.\n\nIf I cycle home on a Friday evening, I want to keep my mind relaxed and\nfocused on my family. All work-related thoughts should be far away. I\ndon't want to \"carry\" around work-related thoughts on the weekend. On\nthe road from office to home I might have an idea what to do (how to\nhunt a strange bug, how to implement a cool feature which needs only a\nvery little effort and time to implement, ...). I stop (that is the great\nadvantage of riding a bike - I can stop almost always immediately, and\ntake my mobile phone). Then I write a mail to my business address and now\nI am sure: This idea won't get lost. And I am free to have a nice\nweekend with my family.\n\nThe same happens when I drive from home to the office: I have an idea\nrelated to my personal life? I stop and write a mail to my account.\n\nThat's how most of this guide-line was created: Most items came to my\nmind during cycling, walking, listening to music, or laying in the bathtub.\nA short mail to myself, and some days later I opened the mail which contains\njust a handful of words that I have articulated.\n\n### Cut bigger problems into smaller ones\n\nA lot of newcomers have problems with this. Here is one example to\nillustrate the guideline \"Cut bigger problems into smaller ones\".\n\nImagine you are responsible for several servers and you should create\ngraphs of their disk/CPU usage.\n\nCut the bigger problem into smaller ones:\n\n-   How to collect the data on one host\n-   How to transport the data from the host to a central place?\n-   How to store the data in a central database?\n-   How to generate the graphs?\n\nBTW, why not use the PostgreSQL feature \"Logical Replication\"?\n\n### Go with the flow, not with the hype\n\nFlow: With \"flow\" I mean \"mainstream\". And the mainstream is according to\noxford dictionary: \"The ideas, attitudes, or activities that are shared\nby most people and regarded as normal or conventional.\"\n\nHype: According to Wikipedia: \"Hype (derived from hyperbole) is\npromotion, especially promotion consisting of exaggerated claims.\"\n\nBut how to distinguish between a flow and hype?\n\nMy answer: Stats or more verbose \"statistics\".\n\nHow to get stats?\n\nI like StackOverflow Tag-Trend. For example, you can compare \"python\"\nand \"java\". Maybe you have been coding Java for several years. You\nheard of python once or twice. But is it \"flow/mainstream\" or is it\n\"hype\"? Since you only know your context and not every developer and\nevery project in the world, you can't know the answer. Be upright to\nyourself: You are like a small ant. You walked several paths in the\npast, but you don't have the helicopter view.\n\nCheck this graph: \u003chttp://sotagtrends.com/?tags\u003e=\\[java,python\\] you\nwill see: Python is not just hyped it is the flow.\n\nDo not trust one source. Take a look at google trends:\n\u003chttps://trends.google.de/trends/explore?date=today%205-y\u0026q=%2Fm%2F05z1_,%2Fm%2F07sbkfb\u003e\n\nGo with the flow, not with the hype. Check the stats, not just our daily\ncontext.\n\n### Read the Release Notes\n\nRead the release notes and news of the tools you use daily. Looking there twice a year is enough.\n\nI like these release notes:\n\n- [PostgreSQL News](https://www.postgresql.org/about/newsarchive/)\n- [Django News](https://www.djangoproject.com/weblog/)\n- [What's new in Python](https://docs.python.org/3/whatsnew/index.html)\n- [CNCF Announcements](https://www.cncf.io/newsroom/announcements/)\n\n### Newsletters\n\n* [Google Webmaster Central Blog](https://webmasters.googleblog.com/)\n\n### Do you want your job to be exciting?\n\nIf your job is exciting, then it will be exhausting.\n\nThis is again the \"boring\" topic, which was one of the first topics of this text.\n\nUrgency gives you a feeling of importance. But that's a [bias](https://en.wikipedia.org/wiki/Bias). Your emotions\nare playing tricks on you. Important things are not urgent.\n\nIf you receive messages constantly (Mail, Chat, WhatsApp, Facebook, ...)\nthen you are not able to focus. My guide: switch off notifications and check\nfor messages only twice a day.\n\nFocus on your boring todo-list which will bring you to your fancy goal.\nMeet in small groups regularly. Before closing a meeting create\na small list of \"who does what until when\".\n\n### Three Mail Accounts\n\nI have three mail accounts:\n\n-   personal mails (family, friends, ...)\n-   work-related mails\n-   mailing lists\n\nMaybe you love your job. But your job is not your family. It is a kind of mental hygiene to\nkeep this separated.\n\n### Clean up your desk\n\nDon't forget to clean your desk. I didn't write this here because I do it\noften and with joy. No, exactly the opposite. I wrote it down since I want\nto push myself.\n\nDon't look at all these things on your desk at once. Start on the left\nside, take the first thing. Where is the best place for this single\nthing? Unsure? Why not throw it in the trash can? If you are unsure put\nit at least in a box behind a closed cabinet door. Some month later you\nmight be able to throw it in the garbage.\n\nThen wipe the dust.\n\nIf you never have time to do this, then there is something wrong. Slow\ndown.\n\n### Thrive for consistency\n\nImagine your software product is like a CMS. A user can edit pages. The product keeps \nversions of this page, so that the editor can revert to an older version.\n\nImagine the button to access history is labeled \"History\".\n\nTerminology is more important than you think.\n\nIf the internal docs have the heading \"Page versioning\", then you know\nthat these docs explain the button labeled \"History\".\n\nBut that's not consistent. \n\nImagine this story: A young and new developer wants to learn more about this feature. The\ndevelopers search in the internal docs with the keyword \"history\". This results\nin wasting some minutes of the developer's precious time. Sure, the developer will find\nthe docs sooner or later, but it is not clean. But this is just the first part of the story.\nThe second part is that the new developer will get demotivated in having clean docs.\n\nI don't want to call this \"Page History\" vs \"Page Versioning\" a problem. It is a spirit.\n\nUsually, there are \"thousand\" places where a term gets used. In the above example, there will\nbe files, classes, methods, data structures containing this name. Changing the label of\nthe button is easy. Changing files, classes, methods, data structures is hard.\n\nSometimes it is too much effort to reflect a new wording in the GUI in all places. With \"all\"\nI mean the places in the GUI and all the non-user visible places which are only visible for the\ndepartment developing the software.\n\nConsistency on the GUI is a must. Consistency in other places makes sense.\n\nSometimes it makes sense. It depends.\n\nThe spirit of the future lies in your hand. You are always able to influence the upcoming\nmonths.\n\n### Highlander, \"There can be only one\"\n\n\"Highlander\" is a 1986 British-American adventure action fantasy film\nwith the tagline \"There can be only one\". Thinking like this narrows your\nmind. There can be several thousand. Look at how successful ants and bees\nwork. If someone is better or faster, then smile. Give applaud and say\n\"wow\".\n\n[Don't be evil.](https://en.wikipedia.org/wiki/Don%27t_be_evil) Don't\nwaste time and mental energy. Applauding if the competitor is better,\nwas new to me in 2017. I was at Rothenbaum and attended the German Open\n(Tennis). The coach of one player was applauding every time the opponent\nmade a good shot. I was astonished. Why was the coach applauding the\nenemy? But this works. If you get angry, you waste energy and you start\nto think like a wild and stupid animal. Even if you have made a mistake\nor lost somehow, no reason not to walk upright.\n\nThere are some rumors about \"real programmers\" and what they do. I think \"real programmers\"\nuse vi and are terrible slow because of the tunnel vision created by too much testosterone.\nSmart developers have friends to talk to.\n\n### Don't waste your time with cheap hardware\n\nSome people love the [Raspberry\nPi](https://en.wikipedia.org/wiki/Raspberry_Pi). I don't like it. It\ndoes not have enough computing power for my use cases. Yes, the device\nis cheap, but I prefer to spend some more money to have more\nperformance. I don't like waiting.\n\n### Write a diary\n\nI think it helps to write a diary. Sitting down and writing about the\nlast days help you to reflect on the things you did. It helps you to focus\non your goals. Do you have goals? I found out that late (age of 40). A\ndiary is fun to read several months later. I try to do it at least once\na week. I have several of diaries.\n\nOne on Facebook readable for everyone. It contains things from my daily\nlife, written in german. \u003chttps://www.facebook.com/thomas.guttler.52\u003e\n\nThere is one on twitter which contains IT topics (open source,\nPython, Django, Linux, PostgreSQL), written in English and readable by everyone.\n[@guettli](https://twitter.com/guettli)\n    \nAnd there is a private which I maintain with Anki. Anki is a flashcard\napp. The front side is the question and the backside is the answer. I\nuse the first side for the date and one to three words, and the backside contains the text. This way I can ask myself what was on my mind\nthese days. But all this should be fun, not a burden.\n\nTo relax and enjoy my emotions at work I have a private document with the heading \"Grrr and Smile\". I \nuse it to write down things I like and dislike. This doc is handy for a feedback call with your manager.\nIf the same (negative) topic raises again and again, then it is time for a change.\n    \n    \n### The Bus factor\n\nFrom Wikipedia: The bus factor is a measurement of the risk resulting\nfrom information and capabilities not being shared among team members\n\n[Bus factor](https://en.wikipedia.org/wiki/Bus_factor)\n\nAvoid creating secret knowledge that is only available to you. Share\nknowledge.\n\nAvoid overspecialization of yourself. It will have drawbacks. Imagine\nthere are some things which only you know. Sooner or later you want to\ngo on holiday and you want a relaxed holiday. You don't want to be\ncalled on your mobile phone by your boss or a teammate. You want two\nweeks off without a single interruption that is related to your work.\n\nI guess all people love it if they are important. Everybody loves it\nif someone needs them. But you will get burnout if no one else can do\nthe things you do.\n\nAvoid overspecialization of a teammate, too. If a teammate has secret\nknowledge and there is no one else who has a clue: Talk. Try to reveal\nthe things which only one person knows. Tell him about your concerns\n(Bus factor). Maybe talk to his boss.\n\nImagine there is an action that needs to be done roughly twice a year.\nFor example, setting up a new server. Up until now, Bob did this every time.\nTalk to your teammates. Explain that every action should be known to at\nleast two people. In practice, this means the next time Bob won't do it.\nIt needs to be done by someone else.\n\nIf you read the above sentences and think \"that's not my job, that's the job\nof the team leader\", then I think it is time to stop acting like a dumb\nsleeping sheep. Get responsible. React relaxed if nobody is listening or\nunderstanding your concerns. \"The Best Path to Long-Term Change Is Slow,\nSimple and Boring.\"\n\n### Related things I wrote\n\nSee [Thomas doing working out loud](https://github.com/guettli/wol)\n\n### Related\n\n- [Hacker-Laws](https://github.com/dwmkerr/hacker-laws)\n\n\n### I need your feedback\n\nIf you have a general question, please start a [new discussion](https://github.com/guettli/programming-guidelines/discussions/new).\n\nIf you think something is wrong or missing, feel free to open an issue or pull request.\n\n### Thank you\n\n-   Robert C. Martin for the book \"Clean Coder\"\n-   Malcolm Tredinnick. Only a few people listened as he did. With\n    \"listen\" I mean \"trying to understand the conversation partner\".\n-   Linus Torvalds for the quote \"Bad programmers worry about the code.\n    Good programmers worry about data structures and their relationships.\".\n-   Bill Gates for the quote \"I choose a lazy person to do a hard job.\n    Because a lazy person will find an easy way to do it.\"\n-   All people who contribute to open-source software (Linux, Python,\n    PostgreSQL, ...)\n-   All people who ask questions and/or answers them at places like\n    StackOverflow.\n-   People I met during study at HTW-Dresden\n-   My teammates at [tbz-pariv](http://www.tbz-pariv.de/).\n-   \u003chttps://chemnitzer.linux-tage.de/\u003e All people involved in this\n    great yearly event.\n-   Ionel Cristian Mărieș for the link to bash pitfalls.\n-   Audience at my presentation at [Python User Group Leipzig 2019](https://www.meetup.com/de-DE/Leipzig-Python-User-Group/events/rbwmtpyzmbnb/)\n-   Marco Bakera for hints (mailing-list python-de 2019)\n","funding_links":[],"categories":["Others"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguettli%2Fprogramming-guidelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguettli%2Fprogramming-guidelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguettli%2Fprogramming-guidelines/lists"}