{"id":24545522,"url":"https://github.com/sueszli/nogil","last_synced_at":"2025-08-21T14:05:20.922Z","repository":{"id":267944101,"uuid":"890489506","full_name":"sueszli/nogil","owner":"sueszli","description":"performance optimizations with GIL free cpython","archived":false,"fork":false,"pushed_at":"2024-12-17T07:03:12.000Z","size":8162,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-22T21:32:47.878Z","etag":null,"topics":["global-interpreter-lock","pep-703","performance-engineering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sueszli.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-18T16:51:23.000Z","updated_at":"2024-12-17T07:03:15.000Z","dependencies_parsed_at":"2024-12-13T12:36:47.603Z","dependency_job_id":null,"html_url":"https://github.com/sueszli/nogil","commit_stats":null,"previous_names":["sueszli/nogil"],"tags_count":0,"template":false,"template_full_name":"sueszli/python-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sueszli%2Fnogil","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sueszli%2Fnogil/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sueszli%2Fnogil/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sueszli%2Fnogil/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sueszli","download_url":"https://codeload.github.com/sueszli/nogil/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243849043,"owners_count":20357696,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["global-interpreter-lock","pep-703","performance-engineering"],"created_at":"2025-01-22T21:29:04.169Z","updated_at":"2025-03-16T09:25:43.120Z","avatar_url":"https://github.com/sueszli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Check out the:\n\n- report: https://sueszli.github.io/nogil/docs/report.pdf\n- slides: https://sueszli.github.io/nogil/docs/slides.pdf\n\n    Y\n  .-^-.\n /     \\      .- ~ ~ -.    pssssss...\n()     ()    /   _ _   `.                     _ _ _\n \\_   _/    /  /     \\   \\                . ~  _ _  ~ .\n   | |     /  /       \\   \\             .' .~       ~-. `.\n   | |    /  /         )   )           /  /             `.`.\n   \\ \\_ _/  /         /   /           /  /                `'\n    \\_ _ _.'         /   /           (  (\n                    /   /             \\  \\\n                   /   /               \\  \\    psssss.....\n                  /   /                 )  )\n                 (   (                 /  /\n                  `.  `.             .'  /\n                    `.   ~ - - - - ~   .'\n                       ~ . _ _ _ _ . ~\n\nFile tree:\n\n- `src/0_plain`: plain python implementations of hashcat\n- `src/1_multiprocessing`: multiprocessing implementations\n- `src/2_multithreading`: multithreading implementations with the GIL enabled/disabled\n- `src/3_ctypes`: ctypes implementations, some using OpenMP\n- `src/4_cpython`: cpython implementations, based on the C API of CPython\n- `benchmark.sh`: non-portable benchmarking script using `perf`\n\nEvaluation:\n\n```bash\n# \n# prepare environment\n# \n\n# compile cpython\nmake docker-up # takes 20min\ndocker compose exec main python3 -c 'import sys; assert sys.version_info \u003e= (3, 13); assert not sys._is_gil_enabled(); print(\"it works!\")'\n\n# get dependencies for hyperfine, openssl\ndocker compose exec main apt update\ndocker compose exec main apt install -y build-essential apt-utils curl libssl-dev openssl\ndocker compose exec main sh -c 'curl --proto \"=https\" --tlsv1.2 -sSf https://sh.rustup.rs | sh'\ndocker compose exec main /root/.cargo/bin/cargo --version\ndocker compose exec main /root/.cargo/bin/cargo install --locked hyperfine\ndocker compose exec main /root/.cargo/bin/hyperfine --version\n\n# get target hash to crack\npython -c \"import hashlib; print(hashlib.sha1('aaa'.encode()).hexdigest())\"\n\n# \n# runtime benchmark\n# \n\n# plain\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/0_plain/itertools.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +1 tmp.csv \u003e\u003e results.csv\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/0_plain/lib.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/0_plain/plain.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\n\n# multiprocessing\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/1_multiprocessing/imap_unordered.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/1_multiprocessing/imap.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/1_multiprocessing/map_async.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/1_multiprocessing/map.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\n\n# multithreading\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"PYTHON_GIL=1 python ./src/2_multithreading/executor.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"PYTHON_GIL=0 python ./src/2_multithreading/executor.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"PYTHON_GIL=1 python ./src/2_multithreading/workers.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"PYTHON_GIL=0 python ./src/2_multithreading/workers.py 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\n\n# ctypes\ndocker compose exec main gcc -fopenmp -fPIC -shared -o ./src/3_ctypes/libhashcat.so ./src/3_ctypes/hashcat.c -lcrypto -lssl\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/3_ctypes/invoke_hashcat.py ./src/3_ctypes/libhashcat.so 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\n\ndocker compose exec main gcc -fopenmp -fPIC -shared -o ./src/3_ctypes/libhashcat_openmp.so ./src/3_ctypes/hashcat_openmp.c -lcrypto -lssl\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/3_ctypes/invoke_hashcat.py ./src/3_ctypes/libhashcat_openmp.so 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\n\n# cpython\ndocker compose exec main gcc -shared -fopenmp -o ./src/4_cpython/hashcatmodule.so -fPIC -I/usr/local/include/python3.13t ./src/4_cpython/hashcat.c -lcrypto -lssl\ndocker compose exec main /root/.cargo/bin/hyperfine --warmup 3 --export-csv tmp.csv \"python ./src/4_cpython/invoke_hashcat.py ./src/3_ctypes/libhashcat_openmp.so 7e240de74fb1ed08fa08d38063f6a6a91462a815\" \u0026\u0026 tail -n +2 tmp.csv \u003e\u003e results.csv\n```\n\n# Advanced Evaluation\n\nWe also measure cycles, instructions and runtime in the `benchmark.sh` script. However, it isn't portable and requires an amd64 architecture for execution.\n\n- 1) update the python binary path in the script at the variable `PYTHON_BIN`. for example: `PYTHON_BIN=\"python3.13-nogil\"`\n- 2) compile the cpython extension binaries:\n\n   ```bash\n   docker compose exec main gcc -shared -fopenmp -o ./src/4_cpython/hashcatmodule.so -fPIC -I/usr/local/include/python3.13t ./src/4_cpython/hashcat.c -lcrypto -lssl\n   docker compose exec main gcc -fopenmp -fPIC -shared -o ./src/3_ctypes/libhashcat_openmp.so ./src/3_ctypes/hashcat_openmp.c -lcrypto -lssl\n   docker compose exec main gcc -fopenmp -fPIC -shared -o ./src/3_ctypes/libhashcat.so ./src/3_ctypes/hashcat.c -lcrypto -lssl\n   ```\n\n- 3) run `./benchmark.sh`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsueszli%2Fnogil","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsueszli%2Fnogil","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsueszli%2Fnogil/lists"}