{"id":19402825,"url":"https://github.com/cscfi/scalable-python","last_synced_at":"2025-02-25T00:24:26.275Z","repository":{"id":71819026,"uuid":"43138984","full_name":"CSCfi/scalable-python","owner":"CSCfi","description":"Python interpreter for massively parallel HPC systems","archived":false,"fork":false,"pushed_at":"2017-10-04T10:00:13.000Z","size":22180,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-01-07T12:25:52.778Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-09-25T12:20:02.000Z","updated_at":"2017-12-28T05:26:29.000Z","dependencies_parsed_at":"2023-03-11T11:27:33.343Z","dependency_job_id":null,"html_url":"https://github.com/CSCfi/scalable-python","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fscalable-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fscalable-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fscalable-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fscalable-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/scalable-python/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240577869,"owners_count":19823533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T11:25:57.965Z","updated_at":"2025-02-25T00:24:26.253Z","avatar_url":"https://github.com/CSCfi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scalable Python 1.2\n\n\u003e based on Python 2.7.13 (see original [README](README.orig))\n\u003e\n\u003e \u0026copy; 2011-2017\n\u003e  [CSC - IT Center for Science Ltd.](http://github.com/CSCfi)   \n\u003e authors:\n\u003e  [Jussi Enkovaara](http://github.com/jussienko) \u0026\n\u003e  [Martti Louhivuori](http://github.com/mlouhivu)\n\n***Special Python interpreter intended for massively parallel HPC systems.***\n\nWhen using Python for parallel applications, standard Python puts a lot of\npressure on the filesystem with multiple processes accessing several small\nfiles during import statements. With thousands of processes the startup time\nof a Python application can be more than 30 minutes.\n\n**Scalable Python** performs the I/O operations used e.g. by import statements\nin a single process and uses MPI to transmit data to/from all other processes.\nPlease see the [inherent limitation](#limitations) of this approach.\n\n## Short build instructions\n\nUse `--enable-mpi` option for configure, e.g.\n\n    ./configure --enable-mpi\n\nBuild and install normal Python interpreter (`python`) as well as standard\nmodules etc.\n\n    make\n    make install\n\nBuild and install the special Python interpreter (`python_mpi`)\n\n    make mpi\n    make install-mpi\n\nBy default, the MPI related parts are build using `mpicc`, but an alternative\nMPI compiler can be specified using the environment variable MPICC (e.g.\n`export MPICC=cc`).\n\n## Short usage instructions\n\nInstallation creates two executables:\n\n- **python** is the standard Python interpreter, which can be used for normal\n  serial applications and e.g. for software installations.\n- **python_mpi** is the special Python interpreter that wraps I/O calls and\n  uses MPI for communication, and thus has to be started with `mpirun` (or\n  an equivalent MPI launch command), e.g.\n\n```\nmpirun -np 16384 python_mpi my_application.py\n```\n\n## Disabling I/O wrappers\n\nIt is possible to disable the special I/O wrappers on-the-fly and revert to\nthe standard POSIX calls by using the built-in function `wrapoff()` in a\nPython script. For example:\n\n    wrapoff()\n    from numpy import array\n\nSimilarly, they can be turned back on with the built-in function `wrapon()`.\n\n## Limitations\n\nAll MPI processes have to perform the same I/O operations (e.g. import\nstatements). For example, a code like this\n\n    if rank == 0:\n        import mymodule\n\nis NOT allowed. If you need to do it, please use the `wrapoff()` and\n`wrapon()` functions to disable the special I/O wrappers temporarily.\n\nIf you experience an unexpected deadlock while using Scalable Python, it is\nmost likely due to triggering this limitation. Unfortunately, even some\nstandard modules use in-function import statements, so the root cause may turn\nout to be an innocent looking function call within an if statement. If needed,\nyou can try to turn off the special I/O wrappers e.g. after the initial module\nimports.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fscalable-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fscalable-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fscalable-python/lists"}