{"id":21182752,"url":"https://github.com/jszitas/what_python_programmers_dont_get","last_synced_at":"2025-10-15T09:12:42.237Z","repository":{"id":220431578,"uuid":"751628967","full_name":"JSzitas/what_python_programmers_dont_get","owner":"JSzitas","description":"A simple exploration into why cache and vectorisation matter (more than just big O)","archived":false,"fork":false,"pushed_at":"2024-02-12T18:49:59.000Z","size":23,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-21T12:34:00.503Z","etag":null,"topics":["cache","cpp","cpp17","decomposition-algorithm","performance-analysis","qr"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JSzitas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-02T01:30:46.000Z","updated_at":"2024-02-03T16:41:25.000Z","dependencies_parsed_at":"2024-02-12T19:50:07.677Z","dependency_job_id":"9ac06a84-e8a9-44de-8990-3142ff34d1cd","html_url":"https://github.com/JSzitas/what_python_programmers_dont_get","commit_stats":null,"previous_names":["jszitas/what_python_programmers_dont_get"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JSzitas%2Fwhat_python_programmers_dont_get","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JSzitas%2Fwhat_python_programmers_dont_get/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JSzitas%2Fwhat_python_programmers_dont_get/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JSzitas%2Fwhat_python_programmers_dont_get/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JSzitas","download_url":"https://codeload.github.com/JSzitas/what_python_programmers_dont_get/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243639392,"owners_count":20323505,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","cpp","cpp17","decomposition-algorithm","performance-analysis","qr"],"created_at":"2024-11-20T17:57:51.244Z","updated_at":"2025-10-15T09:12:37.196Z","avatar_url":"https://github.com/JSzitas.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# What Python programmers do not get\n\nThis project, named to be slightly inflammatory in the programming language flame wars, \nis an example of speeding a program up purely by optimizing the memory layout. This features \ntwo versions of otherwise identical code, with algorithms with identical big **O** complexity.\n\nArguably, the faster algorithm even has a slightly worse constant; what we are comparing here are two \nimplementations of QR decomposition. This is based on [Givens rotations](https://en.wikipedia.org/wiki/Givens_rotation), and both versions are logically \nequivalent; they both perform rotations in the same way. There is no evil witchery. \n\nBoth implementations use the same implementation of Given's rotations, and both replace full matrix multiplies\nby very similar hand-crafted block matrix multiplications. \n\n**There are no tricks. The only difference is the memory layout.**\n\nBoth algorithms take an input matrix to decompose as an array of numbers (to be precise,\na std::vector). Given that 2d objects are just a convenient\nfiction (in programming), what we need to decide is how exactly we will lay out the entries in the matrix in memory:\n\n* **column wise**, also known as column major\n* or **row wise**, also known as row major\n\nSuppose we have a matrix:\n\n| 1 | 2 |\n|-|-|\n| 3 | 4 |\n\nIn the first case, our 1d array is equal to [1, 3, 2, 4]. In the second, it is [1, 2, 3, 4].\n\nAs I show, **this has significant performance implications.** \n\n### The implementation\n\nBoth algorithms take, as an input, a matrix **X** in column major format. This is crucial for the \nrest of the algorithm. Read on to understand why. \n\nThe first implementation implements the algorithm naively; you pass an input matrix of size **n x p**\n(rows x columns) and this is processed, where rotations are applied, row by row. \n\nTo shed some light on this, some simple pseudocode of what we are trying to accomplish\n(actual implementation is slightly harder to read, go to the tinyqr.h header file if you would \nlike to see it):\n\n```pseudocode\nfunction QR inputs: matrix X, integer columns, integer rows \n  initialize R of size rows x columns with X, \n  inialize Q of size rows x rows \n\n  for j in columns:\n    for i in reverse(rows-1): // reverse means loop runs in reverse, i.e. down from rows, rather than up to rows\n      c,s = get_givens_rotation(R[j, i-1], R[j, i])\n      // rotation is just a multiplication with a 2x2 block\n      for k in columns:\n        tmp1 = R[k, i-1]\n        tmp2 = R[k, i]\n        R[k, i-1] = c * temp_1 + s * temp_2;\n        R[k, i] = -s * temp_1 + c * temp_2;\n    // analogously for matrix Q; not shown for sake of brevity\n    //...\nreturn Q, R  \n```\n\nThe second implementation is almost identical, but it transposes the (column major) input matrix \nwhen initializing R:\n```pseudocode\nfunction QR inputs: matrix X, integer columns, integer rows \n  initialize R of size columns x rows with t(X), \n  inialize Q of size rows x rows \n```\nand correspondingly changes all indices to match:\n```\nfor j in columns:\n  for i in reverse(rows-1): // reverse means loop runs in reverse, i.e. down from rows, rather than up to rows\n    c,s = get_givens_rotation(R[i-1, j], R[i, j])\n    for k in columns:\n      tmp1 = R[i, k]\n      tmp2 = R[i-1, k]\n      R[i, k] = c * temp_1 + s * temp_2;\n      R[i-1, k] = -s * temp_1 + c * temp_2;\n      //...\nreturn Q, R\n```\n\n### The benchmark\n\nI run three benchmarks;\n\n* **Non-Square** matrices of size n \u003e= p (General tall matrix) - what I deem the most common use case\n* **Square** (n == p, where n is a multiple of 2) - the second most common use case; you would have this \n  if you were implementing the [QR algorithm](https://en.wikipedia.org/wiki/QR_algorithm)\n* **Unaligned Square** (n == p, where n is a multiple of 2 offset by 1) - purely to show what impact\n  some small amount of cache misalignment can have\n\nacross four different optimization levels:\n\n* **unoptimized** (-O0) \n  - as a baseline, disabling all possible optimization; this is the only case where we observe both \n  versions to be near equivalent; in this case the compiler likely does not take full advantage of \n  the much better cache locality of the second version\n* **optimized** (-03)\n  - as what is more realistic to observe in production systems\n* **optimized natively** (-03 -march=-native) \n  - to see whether optimizing to the CPU architecture \n  actually leads to a significant difference (in principle this should improve things like auto vectorization)\n* **optimized with vectorisation disabled** \n  (-03 -fno-slp-vectorize -fno-tree-vectorize) \n  - since the second version is much easier\n  optimize via loop unrolling (i.e. loops can actually be unrolled and vectorised) and I want to\n  discuss primarily the cache benefits, this shows that we still get large benefits out of making \n  cache happy \n\n### Running the benchmark \n\nYou can easily build the whole suite via make:\n```bash\nmake all\n```\nAnd run it via:\n```bash\nmake run_all\n```\nThis will dump the results into subdirectory 'benchmarks'. \n\n### The results\n\nAfter running the previously mentioned tests, we get the following results. I show median as well as average speedups since averages can be \ndeceiving (i.e. if you had mostly the same performance and then a few large outliers): \n\n| Test                   | Average speedup | Median speedup | Total time | Worst speedup | Best speedup | Average speed v1 | Average speed v2 |\n|------------------------| --------------: | -------------: | ---------: | ------------: | -----------: | ---------------: | --------------: |\n| optimized_sqr.csv      | 14.95           | 12.28          | 45,798.3   | 0.43          | 33.02        | 1,060,330        | 45,663           |\n| unvec_unal.csv         | 1.55            | 1.51           | 40,392.2   | 0.02          | 18.68        | 11,625.1         | 5,736.51         |\n| optimizedv_nonsqr.csv  | 23.25           | 16.62          | 210,265    | 0.02          | 34.09        | 82,779.1         | 2,573.85         |\n| unvec_nonsqr.csv       | 7.39            | 6.02           | 88,387.5   | 0.14          | 23.9         | 64,341.3         | 5,834.42         |\n| optimizedv_sqr.csv     | 26.84           | 21.07          | 79,301.9   | 0.15          | 42.56        | 1,309,850        | 31,978.4         |\n| optimizedv_unal.csv    | 3.69            | 2.93           | 69,509.7   | 0.32          | 45.23        | 11,523           | 2,352.79         |\n| optimized_unal.csv     | 2.50            | 2.19           | 53,312.5   | 0.42          | 9.09         | 12,059           | 3,606.53         |\n| unoptimized_unal.csv   | 1.07            | 1.00           | 35,109.9   | 0.18          | 3.29         | 89,008.4         | 82,174.8         |\n| optimized_nonsqr.csv   | 12.41           | 9.59           | 130,495    | 0.07          | 19.36        | 67,966.5         | 3,809.31         |\n| unoptimized_sqr.csv    | 1.00            | 0.99           | 8,992.35   | 0.34          | 3.14         | 1,182,540        | 1,158,360        |\n| unoptimized_nonsqr.csv | 1.00            | 0.99           | 34,637.2   | 0.05          | 6.19         | 76,459.4         | 75,971.5         |\n| unvec_sqr.csv          | 8.60            | 7.37           | 28,020.9   | 0.04          | 18.02        | 1,014,400        | 75,699.5         |\n\nUnder the unoptimized setting, we see what we would expect; that doing 'more work' equates to longer runtimes; \ntransposing the matrix has a small but non-zero cost, and so our second version is seemingly a bit slower. \n\nIn every other case, i.e. if you do not forget to turn on optimization, the second version is drastically better. \nUnder every optimized setting, we see at least a 50% speedup; for the unaligned, unvectorised case, as should be expected. \nThe best case is clearly the optimized + vectorised case with square matrices, but even aside from that, \nwe see very healthy speedups across the board. We see that even in the unaligned cases, with optimization\nwe are looking at a 2x speedup. Vectorization easily makes this 3x.\n\n**But for aligned data, we are easily over 10x better.** \n\nKeep in mind that all we did was 'more work' and a change of data layout. There was no manual vectorization, \nno work to make sure the data are aligned (to fit neatly into vector registers), etc. These things would likely improve \nperformance even further. \n\n### Okay, this is cool, but why the name?\n\nI named this repository the way it is for two reasons. Reason number one is, obviously, attention; it is easier to grab some \nif people are getting at least a bit upset. \n\nThe other less silly reason is this; I keep hearing, quite often, from people\nwho work in Python / Javascript / **insert your favourite high level language** about how you do not need to think about \nperformance. \n\nHow you will only gain a bit of performance by switching over to a different language / framework / ... (usually whatever the speaker deems \nan acceptable 'loss of performance', quite often a figure like 10-20%).\n\nHow you do not need to worry about 'low level'\ndetails of your program, as they cannot possibly beat big **O** complexity and other concerns, and besides, \nthey are surely not worth the added code complexity.\n\n**This repo shows why I consider those arguments misguided.**\n\nI have added minimal code complexity, if any. I have not written thousands of lines of code, or fine tuned something in assembly. \nI changed the data layout, in a fairly intuitive way. Big **O** complexity is important. But so are data layouts, and if you work in a language where you cannot control them, \nyou might get lucky, or you might get unlucky. It sucks to get unlucky.\n\nAt the end of the day, we all need to think of data layouts if we want performance, regardless of language.\n\n#### Additional reading/talks\n\nI was originally inspired by numerous talks to investigate this topic. \nI strongly recommend watching:\n\n[Mike Acton's legendary talk at CppCon 2014](https://www.youtube.com/watch?v=rX0ItVEVjHc\u0026t=51s)\n\n[A follow-up on Data Oriented Programming](https://www.youtube.com/watch?v=yy8jQgmhbAU)\n\n[A very good talk by Andrei Alexandrescu from CppCon 2019](https://www.youtube.com/watch?v=FJJTYQYB1JQ)\n\n[A really nice hash table implementation](https://www.youtube.com/watch?v=DMQ_HcNSOAI\u0026t=1s)\n\n[Explanation of how much performance can be lost via virtual calls](https://www.youtube.com/watch?v=tD5NrevFtbU\u0026t=1023s)\n\n[Why performance matters](https://www.youtube.com/watch?v=x2EOOJg8FkA\u0026t=203s)\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjszitas%2Fwhat_python_programmers_dont_get","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjszitas%2Fwhat_python_programmers_dont_get","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjszitas%2Fwhat_python_programmers_dont_get/lists"}