{"id":20688074,"url":"https://github.com/neutrinoceros/gpgi","last_synced_at":"2025-04-22T15:10:13.597Z","repository":{"id":60838006,"uuid":"543586119","full_name":"neutrinoceros/gpgi","owner":"neutrinoceros","description":"A lightweight Python library for efficient in-RAM particle deposition on rectilinear, unrefined grids.","archived":false,"fork":false,"pushed_at":"2025-04-12T10:42:53.000Z","size":1110,"stargazers_count":4,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-22T15:09:47.146Z","etag":null,"topics":["data-analysis","grid","particles","performance"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neutrinoceros.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-30T12:17:29.000Z","updated_at":"2025-04-12T10:42:54.000Z","dependencies_parsed_at":"2023-10-02T07:42:46.418Z","dependency_job_id":"fb0c1373-0bef-4dec-9093-d28602c23558","html_url":"https://github.com/neutrinoceros/gpgi","commit_stats":{"total_commits":141,"total_committers":3,"mean_commits":47.0,"dds":"0.049645390070921946","last_synced_commit":"c07a5dc730b613da59849e1b183223b4e8dfc77e"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neutrinoceros%2Fgpgi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neutrinoceros%2Fgpgi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neutrinoceros%2Fgpgi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neutrinoceros%2Fgpgi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neutrinoceros","download_url":"https://codeload.github.com/neutrinoceros/gpgi/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250264909,"owners_count":21402004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","grid","particles","performance"],"created_at":"2024-11-16T22:59:08.027Z","updated_at":"2025-04-22T15:10:13.585Z","avatar_url":"https://github.com/neutrinoceros.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GPGI\n[![PyPI](https://img.shields.io/pypi/v/gpgi.svg?logo=pypi\u0026logoColor=white\u0026label=PyPI)](https://pypi.org/project/gpgi/)\n[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)\n\n***Fast particle deposition at post-processing time***\n\nThis small Python library implements fundamental grid deposition algorithms to\nanalyse (rectilinear) grid + particle datasets, with an emphasize on\nperformance. Core algorithms are implemented as Cython extensions.\n\nGPGI stands for **G**eneric **P**article + **G**rid data **I**nterface\n\n## Table of Contents\n\n\u003c!-- toc --\u003e\n\n- [Installation](#installation)\n- [Supported applications](#supported-applications)\n  * [Builtin deposition methods](#builtin-deposition-methods)\n  * [Supported geometries](#supported-geometries)\n- [Time complexity](#time-complexity)\n- [Usage](#usage)\n  * [Supplying arbitrary metadata](#supplying-arbitrary-metadata)\n  * [Boundary conditions](#boundary-conditions)\n    + [Builtin recipes](#builtin-recipes)\n    + [Define custom recipes](#define-custom-recipes)\n  * [Weight fields (Depositing intensive quantities)](#weight-fields-depositing-intensive-quantities)\n  * [Count Sorting](#count-sorting)\n- [Deposition algorithm](#deposition-algorithm)\n- [Thread safety](#thread-safety)\n\n\u003c!-- tocstop --\u003e\n\n## Installation\n\n```shell\npython -m pip install --upgrade pip\npython -m pip install gpgi\n```\n\n## Supported applications\n\nA rectilinear grid is defined as 1D arrays representing cell left edges in each directions. Note that the last point of such an array is interpreted as the right edge of the rightmost cell, so for instance, a 1D grid containing 100 cells is defined by 101 edges.\n\nParticles are defined as points that live within the grid's bounds.\n\nDeposition is the action of going from particle description\nto a grid description of a field.\nIt is useful to analyze, compare and combine simulation data that exists in a combination of the two formalisms.\nThis process is not reversible as it degrades information.\n\nFor instance, here's a simple overlay of a particle set (red dots)\nagainst a background that represents the deposited particle count.\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/neutrinoceros/gpgi/main/tests/pytest_mpl_baseline/test_2D_deposit_ngp.png\" width=\"600\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nThis example illustrates the simplest possible deposition method, \"Nearest Grid Point\" (NGP), in which each particle contributes only to the cell\nthat contains it.\n\nMore refined methods are also available.\n\n### Builtin deposition methods\n| method name             | abbreviated name | order |\n|-------------------------|:----------------:|:-----:|\n| Nearest Grid Point      | NGP              | 0     |\n| Cloud in Cell           | CIC              | 1     |\n| Triangular Shaped Cloud | TSC              | 2     |\n\n*new in gpgi 0.12.0*\nUser-defined alternative methods may be provided to `Dataset.deposit` as `method=my_func`.\nTheir signature need to be compatible with `gpgi.typing.DepositionMethodT` or `gpgi.typing.DepositionMethodWithMetadataT` .\n\n### Supported geometries\n| geometry name | axes order                  |\n|---------------|-----------------------------|\n| cartesian     | x, y, z                     |\n| polar         | radius, z, azimuth          |\n| cylindrical   | radius, azimuth, z          |\n| spherical     | radius, colatitude, azimuth |\n| equatorial    | radius, azimuth, latitude   |\n\n## Time complexity\n\nAn important step in performing deposition is to associate particle indices to cell indices. This step is called \"particle indexing\".\nIn directions where the grid is uniformly stepped (if any), indexing a particle is an O(1) operation.\nIn the more general case, indexing is performed by bisection, which is a O(log(nx)) operation (where nx represents the number of cells in the direction of interest).\n\n\n## Usage\n\nThe API consists in a `load` function, which returns a `Dataset` object.\n\n**Load** data\n\n```python\nimport numpy as np\nimport gpgi\n\nnx = ny = 64\nnparticles = 600_000\n\nprng = np.random.RandomState(0)\nds = gpgi.load(\n    geometry=\"cartesian\",\n    grid={\n        \"cell_edges\": {\n            \"x\": np.linspace(-1, 1, nx),\n            \"y\": np.linspace(-1, 1, ny),\n        },\n    },\n    particles={\n        \"coordinates\": {\n            \"x\": 2 * (prng.normal(0.5, 0.25, nparticles) % 1 - 0.5),\n            \"y\": 2 * (prng.normal(0.5, 0.25, nparticles) % 1 - 0.5),\n        },\n        \"fields\": {\n            \"mass\": np.ones(nparticles),\n        },\n    },\n)\n```\nThe `Dataset` object holds a `grid` and a `particles` attribute,\nwhich both hold a `fields` attribute for accessing their data.\nBut more importantly, the `Dataset` has a `deposit` method to\ntranslate particle fields to the grid formalism.\n\n**Deposit Particle fields on the grid**\n\n```python\nparticle_mass = ds.deposit(\"mass\", method=\"nearest_grid_point\")  # or \"ngp\" for shorts\n```\n\n**Visualize**\nIn this example we'll use `matplotlib` for rendering, but note that `matplotlib` is not a dependency to `gpgi`\n```python\nimport matplotlib.pyplot as plt\n\nfig, ax = plt.subplots()\nax.set(aspect=1, xlabel=\"x\", ylabel=\"y\")\n\nim = ax.pcolormesh(\n    \"x\",\n    \"y\",\n    particle_mass.T,\n    data=ds.grid.cell_edges,\n    cmap=\"viridis\",\n)\nfig.colorbar(im, ax=ax)\n```\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/neutrinoceros/gpgi/main/tests/pytest_mpl_baseline/test_readme_example.png\" width=\"600\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nThe example script given here takes about a second (top to bottom).\n\n\n\n### Supplying arbitrary metadata\n*new in gpgi 0.4.0*\n\n`Dataset` objects have a special attribute `metadata` which is a dictionary with string keys.\nThis attribute is meant to hold any special metadata that may be relevant for labelling or processing (e.g. simulation time, author, ...).\nMetadata can be supplied at load time as\n```python\nds = gpgi.load(\n    geometry=\"cartesian\",\n    grid=...,\n    particles=...,\n    metadata={\"simulation_time\": 12.5, \"author\": \"Clément Robert\"}\n)\n```\n\n\n### Boundary conditions\n*new in gpgi 0.5.0*\n\nWith CIC and TSC deposition, particles contribute to cells neighbouring the one\nthat contains them. For particles that live in the outermost layer of the\ndomain, this means some of their contribution is lost. This behavior\ncorresponds to the default `'open'` boundary condition, but `gpgi` has builtin\nsupport for more conservative boundary conditions.\n\nBoundary conditions can be selected per field, per axis and per side. Builtin\nrecipes all perform linear combinations of ghost layers (same-side and opposite\nside) and active domain layers (same-side and opposite side), and replace the\nsame-side active layer with the result.\n\nUser-selected boundary conditions take the form of an optional argument to\n`Dataset.deposit`, as a dictionary with keys being axes names, and values being\n2-tuples of boundary condition names (left and right side respectively).\nFor instance, here's how one would require periodic boundary conditions on all axes:\n\n```python\nds.deposit(\n    \"mass\",\n    method=\"cic\",\n    boundaries={\n        \"x\": (\"periodic\", \"periodic\"),\n        \"y\": (\"periodic\", \"periodic\"),\n    }\n)\n```\nUnspecified axes will use the default `'open'` boundary.\n\n\n#### Builtin recipes\n\n| boundary conditions     | description                                           | conservative ? |\n|-------------------------|-------------------------------------------------------|:--------------:|\n| open (default)          | no special treatment                                  | no             |\n| periodic                | add opposite ghost layer to the active domain         | yes            |\n| wall                    | add same-side ghost layer to the active domain        | yes            |\n| antisymmetric           | subtract same-side ghost layer from the active domain | no             |\n\n\n#### Define custom recipes\n\n`gpgi`'s boundary recipes can be customized. Let's illustrate this feature with a simple example.\nSay we want to fix the value of the deposited field in some outer layer.\nThis is done by defining a new function on the user side:\n\n```python\ndef ones(\n    same_side_active_layer,\n    same_side_ghost_layer,\n    opposite_side_active_layer,\n    opposite_side_ghost_layer,\n    weight_same_side_active_layer,\n    weight_same_side_ghost_layer,\n    weight_opposite_side_active_layer,\n    weight_opposite_side_ghost_layer,\n    side,\n    metadata,\n):\n   return 1.0\n```\nwhere all first eight arguments are `numpy.ndarray` objects with the same shape (which includes ghost padding !),\nto which the return value must be broadcastable, `side` can only be either\n`\"left\"` or `\"right\"`, and `metadata` is the special `Dataset.metadata`\nattribute. Not all arguments need be used in the body of the function, but this\nsignature is required.\n\nThe method must then be registered as a boundary condition recipe as\n```python\nds.boundary_recipes.register(\"ones\", ones)\n```\nwhere the associated key (here `\"ones\"`) is arbitrary. The recipe can now be\nused exactly as builtin ones, and all of them can be mixed arbitrarily.\n```python\nds.deposit(\n    \"mass\",\n    method=\"cic\",\n    boundaries={\n        \"x\": (\"ones\", \"wall\"),\n        \"y\": (\"periodic\", \"periodic\"),\n    }\n)\n```\n\nNote that all first eight arguments in a boundary recipe function should\nrepresent an *extensive* physical quantity (as opposed to *intensive*). When\ndepositing an *intensive* quantity `u`, a weight field `w` should be supplied\n(see next section), in which case, the first four arguments represent `u*w` and\nthe following four represent *w*, so that *u* can still be obtained within the\nfunction as a ratio if needed.\n\n\n### Weight fields (Depositing intensive quantities)\n*new in gpgi 0.7.0*\n\nFundamentally, deposition algorithms construct on-grid fields by performing\n*summations*. An implication is that the physical quantities being deposited are\nrequired to be *extensive* (like mass or momentum). *Intensive* quantities (like\nvelocity or temperature) require additional operations, and necessitate the use\nof an additional *weight field*.\n\nThis section provides showcases their *usage*. For a detailed explanation of the deposition algorithm for intensive quantities, see [Deposition algorithm](#deposition-algorithm).\n\nIn order to deposit an *intensive* field (e.g., `vx`), an additional `weight_field` argument must be provided as\n```python\nds.deposit(\n    \"vx\",\n    method=\"cic\",\n    boundaries={\n        \"y\": (\"periodic\", \"periodic\"),\n        \"x\": (\"antisymmetric\", \"antisymmetric\"),\n    },\n    weight_field=\"mass\",\n    weight_field_boundaries={\n        \"y\": (\"periodic\", \"periodic\"),\n        \"x\": (\"open\", \"open\"),\n    },\n)\n```\n\nBoundary recipes may be also associated to the weight field with the\n`weight_field_boundaries` argument. This arguments becomes *required* if\n`boundaries` and `weight_field` are both provided.\n\nCall `help(ds.deposit)` for more detail.\n\n\n### Count Sorting\n*new in gpgi 0.14.0*\n\ngpgi can load arbitrarily ordered particle sets, though deposition algorithms\nperform better when the in-memory position of particles correlates with their physical positions relative to the grid, since such a state minimizes the number of cache misses.\n\nParticles may be sorted by a [counting sort algorithm](https://en.wikipedia.org/wiki/Counting_sort),\nas\n\n```python\nds = ds.sorted()\n```\nNote that this method returns a *copy* of the dataset, so it will best perform for datasets that, at most, fit in half your RAM.\n\nThis operation is costly in itself, so there may be a trade-off depending on how many\ndepositions one needs to perform on a given dataset before tossing it out.\n\nBy default, axes are weighted in the order that's optimal for gpgi's deposition routines,\nbut arbitrary priority order may be specified as, for instance\n```python\nds = ds.sorted(axes=(1, 0))\n```\n\nUse the `Dataset.is_sorted` method to check whether particles are already sorted without\nperforming the sort. `Dataset.is_sorted` accepts an `axes` argument just like `Dataset.sorted`. This is useful for testing and comparative purposes.\n\n\n## Deposition algorithm\n\nThis section provides details on the general deposition algorithm, as\nimplemented in `gpgi`.\n\nWithout loss of generality, we will illustrate how  an *intensive* field (`v`)\nis deposited, since this case requires the most computational steps. As it\nhappens, depositing an *extensive* field (`w`) separately is actually part of\nthe algorithm.\n\n**Definitions**\n\n- `v` is an intensive field that we want to deposit on the grid\n- `w` is an extensive field that will be used as weights\n- `u = v * w` is an extensive equivalent to `v` (conceptually, if `v` is a velocity and `w` is a mass, `u` corresponds to a momentum)\n\n`u(i)`, `v(i)` and `w(i)` are defined for each particle `i`.\n\nWe note `U(x)`, `V(x)` and `W(x)` the corresponding on-grid fields, where `V(x)`\nis the final output of the algorithm. These are defined at grid cell centers\n`x`, within the *active domain*.\n\nLast, we note `U'(x)`, `V'(x)` and `W'(x)` the *raw* deposited fields, meaning\nno special treatment is applied to the outermost layers (boundary conditions).\nThese are defined at grid cell centers, *including one ghost layer* that will be\nused to apply boundary conditions.\n\n**Algorithm**\n\n1) `W'` and `U'` are computed as\n```\nW'(x) = Σ c(i,x) w(i)\nU'(x) = Σ c(i,x) w(i) v(i)\n```\nwhere `c(i,x)` are geometric coefficients associated with the deposition method. Taking the nearest grid point (NGP) method for illustration, `c(i,x) = 1` if particle `i` is contained in the cell whose center is `x`, and `c(i,x) = 0` elsewhere.\n\n2) boundary conditions are applied\n```\nW(x) = W_BCO(W', 1, metadata)\nU(x) = U_BCO(U', W', metadata)\n```\nwhere `W_BCO` and `U_BCO` denote arbitrary boundary condition operators\nassociated with `W` and `U` respectively, and which take 3 arguments,\nrepresenting the field to be transformed, its associated weight field and a\nwildcard `metadata` argument which may contain any additional data relevant to\nthe operator.\n\nNote `1` is used a placeholder \"weight\" for `W`, for symmetry reasons: all boundary condition operators must expose a similar interface, as explained in [Define custom recipes](#define-custom-recipes).\n\n3) Finally, `V(x)` is obtained as\n```\nV(x) = (U/W)(x)\n```\n\n\n## Thread safety\n\nStarting in gpgi 2.0.0, thread safety is guaranteed in `Dataset.host_cell_index`\ncomputation and `Dataset.deposit`, and both operations release the\nGIL (Global Interpreter Lock) around their respective hotloops. Thread safety is\nalso tested against the experimental free-threaded build of Python 3.13.\n\nNote that, by default, `Dataset.deposit` still uses a lock per `Dataset`\ninstance, which in the most general case is preferable since concurrently\ndepositing many fields can cause catastrophic degradations of performances as\nit encourages cache misses. Optimal performance is however application-specific,\nso this strategy can be overridden using the `lock` parameter:\n- using `lock=None` will not use any lock, which in restricted conditions\n  leads to better walltime performances\n- alternatively, an externally managed `threading.Lock` instance may be supplied\n\n`Dataset.boundary_recipes.register` is also thread-safe: registering a shared\nfunction multiple times is supported, but an error is raised in case one\nattempts registering a different function under an existing key.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneutrinoceros%2Fgpgi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneutrinoceros%2Fgpgi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneutrinoceros%2Fgpgi/lists"}