{"id":17816602,"url":"https://github.com/frobnitzem/mpi_list","last_synced_at":"2025-03-18T03:31:00.224Z","repository":{"id":57443427,"uuid":"354154854","full_name":"frobnitzem/mpi_list","owner":"frobnitzem","description":"A package for working with lists distributed over MPI","archived":false,"fork":false,"pushed_at":"2023-09-08T17:48:35.000Z","size":47,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-16T16:39:59.033Z","etag":null,"topics":["data-science","hpc","map-reduce","mpi4py"],"latest_commit_sha":null,"homepage":"https://mpi-list.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/frobnitzem.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.rst","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-02T23:16:25.000Z","updated_at":"2024-01-11T11:49:28.000Z","dependencies_parsed_at":"2024-10-27T17:15:14.098Z","dependency_job_id":"2701064b-c406-4e32-99a3-f385e21a6f90","html_url":"https://github.com/frobnitzem/mpi_list","commit_stats":{"total_commits":12,"total_committers":1,"mean_commits":12.0,"dds":0.0,"last_synced_commit":"c4c754f027402c8e3df15c1d59f811d12ab148df"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frobnitzem%2Fmpi_list","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frobnitzem%2Fmpi_list/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frobnitzem%2Fmpi_list/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frobnitzem%2Fmpi_list/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/frobnitzem","download_url":"https://codeload.github.com/frobnitzem/mpi_list/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244150274,"owners_count":20406552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","hpc","map-reduce","mpi4py"],"created_at":"2024-10-27T16:38:38.591Z","updated_at":"2025-03-18T03:30:59.923Z","avatar_url":"https://github.com/frobnitzem.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"========\nmpi list\n========\n\nThis package implements the `DFM` class.\n\nThe `DFM` is a useful abstraction for working with\nlists distributed over a set of MPI ranks.\nThe acronym stands for distributed free monoid,\nwhich is just a fancy way to say it's a list.\n\nIf you're familiar with spark, it's like an RDD,\nbut only holds a list.\n\nQuick Start\n===========\n\n.. code-block::\n\n    from mpi_list import Context, DFM\n\n    C = Context() # calls MPI_Init via mpi4py\n\n    # After each of the three lines below:\n    #  1. each rank now has 1000//C.procs consecutive numbers\n    #  2. each rank now has a list of strings\n    #  3. only numbers containing a '2' remain\n    dfm = C . iterates(1000) \\\n            . map(lambda i: f\"String {i}\") \\\n            . filter(lambda s: '2' in s)\n\n    if C.rank == 0:\n        # Caution! Uncommenting this will deadlock your program.\n        # Collective calls must be called by all ranks!\n        #print( dfm . head(10) )\n        pass\n\n    # This is OK, since all ranks now have 'ans'\n    ans = dfm.head(10)\n    if C.rank == 0:\n        print( ans )\n\n    ans = dfm . filter(lambda s: len(s) \u003c= len(\"String nn\")) \\\n              . collect()\n    if ans is not None: # only rank 0 gets \"collect\"\n        print( ans )\n\nLaunch your program with `mpirun python my_prog.py`.\n\nIf you're using a supercomputer, consider installing\n`spindle \u003chttps://computing.llnl.gov/projects/spindle/software\u003e`_,\nand then use `spindle mpirun python my_prog.py`.\n\n.. _pyscaffold-notes:\n\nNote\n====\n\nThis project has been set up using PyScaffold 4.0.1. For details and usage\ninformation on PyScaffold see https://pyscaffold.org/.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrobnitzem%2Fmpi_list","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrobnitzem%2Fmpi_list","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrobnitzem%2Fmpi_list/lists"}