{"id":19578647,"url":"https://github.com/materialsproject/maggma","last_synced_at":"2025-04-09T11:12:27.632Z","repository":{"id":21861077,"uuid":"94256653","full_name":"materialsproject/maggma","owner":"materialsproject","description":"MongoDB aggregation machine","archived":false,"fork":false,"pushed_at":"2024-04-15T22:28:46.000Z","size":15608,"stargazers_count":35,"open_issues_count":53,"forks_count":30,"subscribers_count":13,"default_branch":"main","last_synced_at":"2024-04-16T01:28:39.129Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://materialsproject.github.io/maggma/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/materialsproject.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-06-13T20:50:20.000Z","updated_at":"2024-08-23T18:40:19.064Z","dependencies_parsed_at":"2023-10-26T21:34:25.028Z","dependency_job_id":"d2d33460-f89b-4b77-bc76-e5e7b6986ae9","html_url":"https://github.com/materialsproject/maggma","commit_stats":{"total_commits":2100,"total_committers":28,"mean_commits":75.0,"dds":0.6309523809523809,"last_synced_commit":"5f0edb599f38da14ca4dab36bd95197fbed54b5f"},"previous_names":[],"tags_count":199,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/materialsproject%2Fmaggma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/materialsproject%2Fmaggma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/materialsproject%2Fmaggma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/materialsproject%2Fmaggma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/materialsproject","download_url":"https://codeload.github.com/materialsproject/maggma/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248027411,"owners_count":21035594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T07:12:28.470Z","updated_at":"2025-04-09T11:12:27.597Z","avatar_url":"https://github.com/materialsproject.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# ![Maggma](docs/logo_w_text.svg)\n\n[![Static Badge](https://img.shields.io/badge/documentation-blue?logo=github)](https://materialsproject.github.io/maggma) [![testing](https://github.com/materialsproject/maggma/workflows/testing/badge.svg)](https://github.com/materialsproject/maggma/actions?query=workflow%3Atesting) [![codecov](https://codecov.io/gh/materialsproject/maggma/branch/main/graph/badge.svg)](https://codecov.io/gh/materialsproject/maggma) [![python](https://img.shields.io/badge/Python-3.9+-blue.svg?logo=python\u0026amp;logoColor=white)]()\n\n## What is Maggma\n\nMaggma is a framework to build scientific data processing pipelines from data stored in\na variety of formats -- databases, Azure Blobs, files on disk, etc., all the way to a\nREST API. The rest of this README contains a brief, high-level overview of what `maggma` can do.\nFor more, please refer to [the documentation](https://materialsproject.github.io/maggma).\n\n\n## Installation from PyPI\n\nMaggma is published on the [Python Package Index](https://pypi.org/project/maggma/).  The preferred tool for installing\npackages from *PyPi* is **pip**.  This tool is provided with all modern\nversions of Python.\n\nOpen your terminal and run the following command.\n\n``` shell\npip install --upgrade maggma\n```\n\n## Basic Concepts\n\n`maggma`'s core classes -- [`Store`](#store) and [`Builder`](#builder) -- provide building blocks for\nmodular data pipelines. Data resides in one or more `Store` and is processed by a\n`Builder`. The results of the processing are saved in another `Store`, and so on:\n\n```mermaid\nflowchart LR\n    s1(Store 1) --Builder 1--\u003e s2(Store 2) --Builder 2--\u003e s3(Store 3)\ns2 -- Builder 3--\u003es4(Store 4)\n```\n\n### Store\n\nA major challenge in building scalable data pipelines is dealing with all the different types of data sources out there. Maggma's `Store` class provides a consistent, unified interface for querying data from arbitrary data sources. It was originally built around MongoDB, so it's interface closely resembles `PyMongo` syntax. However, Maggma makes it possible to use that same syntax to query other types of databases, such as Amazon S3, GridFS, or files on disk, [and many others](https://materialsproject.github.io/maggma/getting_started/stores/#list-of-stores). Stores implement methods to `connect`, `query`, find `distinct` values, `groupby` fields, `update` documents, and `remove` documents.\n\nThe example below demonstrates inserting 4 documents (python `dicts`) into a `MongoStore` with `update`, then\naccessing the data using `count`, `query`, and `distinct`.\n\n```python\n\u003e\u003e\u003e turtles = [{\"name\": \"Leonardo\", \"color\": \"blue\", \"tool\": \"sword\"},\n               {\"name\": \"Donatello\",\"color\": \"purple\", \"tool\": \"staff\"},\n               {\"name\": \"Michelangelo\", \"color\": \"orange\", \"tool\": \"nunchuks\"},\n               {\"name\":\"Raphael\", \"color\": \"red\", \"tool\": \"sai\"}\n            ]\n\u003e\u003e\u003e store = MongoStore(database=\"my_db_name\",\n                       collection_name=\"my_collection_name\",\n                       username=\"my_username\",\n                       password=\"my_password\",\n                       host=\"my_hostname\",\n                       port=27017,\n                       key=\"name\",\n                    )\n\u003e\u003e\u003e with store:\n        store.update(turtles)\n\u003e\u003e\u003e store.count()\n4\n\u003e\u003e\u003e store.query_one({})\n{'_id': ObjectId('66746d29a78e8431daa3463a'), 'name': 'Leonardo', 'color': 'blue', 'tool': 'sword'}\n\u003e\u003e\u003e store.distinct('color')\n['purple', 'orange', 'blue', 'red']\n```\n\n### Builder\n\nBuilders represent a data processing step, analogous to an extract-transform-load (ETL) operation in a data\nwarehouse model. Much like `Store` provides a consistent interface for accessing data, the `Builder` classes\nprovide a consistent interface for transforming it. `Builder` transformation are each broken into 3 phases: `get_items`, `process_item`, and `update_targets`:\n\n1. `get_items`: Retrieve items from the source Store(s) for processing by the next phase\n2. `process_item`: Manipulate the input item and create an output document that is sent to the next phase for storage.\n3. `update_target`: Add the processed item to the target Store(s).\n\nBoth `get_items` and `update_targets` can perform IO (input/output) to the data stores. `process_item` is expected to not perform any IO so that it can be parallelized by Maggma. Builders can be chained together into an array and then saved as a JSON file to be run on a production system.\n\n## Origin and Maintainers\n\nMaggma has been developed and is maintained by the [Materials Project](https://materialsproject.org/) team at Lawrence Berkeley National Laboratory and the [Materials Project Software Foundation](https://github.com/materialsproject/foundation).\n\nMaggma is written in [Python](http://docs.python-guide.org/en/latest/) and supports Python 3.9+.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaterialsproject%2Fmaggma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaterialsproject%2Fmaggma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaterialsproject%2Fmaggma/lists"}