{"id":14959457,"url":"https://github.com/nodestream-proj/nodestream","last_synced_at":"2025-04-05T13:04:07.627Z","repository":{"id":175616812,"uuid":"593268221","full_name":"nodestream-proj/nodestream","owner":"nodestream-proj","description":"A Declarative framework for Building, Maintaining, and Analyzing Graph Data","archived":false,"fork":false,"pushed_at":"2025-03-27T17:42:30.000Z","size":3721,"stargazers_count":41,"open_issues_count":14,"forks_count":13,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-27T18:34:54.349Z","etag":null,"topics":["api","athena","aws","cli","data-engineering","data-lake","data-science","declarative","etl","framework","graph","graphql","kafka","knowledge-graph","neo4j","python","s3","security","visualization","yaml"],"latest_commit_sha":null,"homepage":"https://nodestream-proj.github.io/docs/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nodestream-proj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-25T16:15:25.000Z","updated_at":"2025-03-27T17:42:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"448fd1a4-b7ba-4574-ad64-031f9bec4ab9","html_url":"https://github.com/nodestream-proj/nodestream","commit_stats":{"total_commits":912,"total_committers":14,"mean_commits":65.14285714285714,"dds":0.4144736842105263,"last_synced_commit":"448906e4e19e438978fde17a8a3f87cbb2ffcc73"},"previous_names":["nodestream-proj/nodestream"],"tags_count":95,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodestream-proj%2Fnodestream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodestream-proj%2Fnodestream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodestream-proj%2Fnodestream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodestream-proj%2Fnodestream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nodestream-proj","download_url":"https://codeload.github.com/nodestream-proj/nodestream/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247339153,"owners_count":20923014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","athena","aws","cli","data-engineering","data-lake","data-science","declarative","etl","framework","graph","graphql","kafka","knowledge-graph","neo4j","python","s3","security","visualization","yaml"],"created_at":"2024-09-24T13:19:46.877Z","updated_at":"2025-04-05T13:04:07.594Z","avatar_url":"https://github.com/nodestream-proj.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nodestream \n\n\u003cpicture \u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"./docs/img/NodeSteamLogo_Small_Light.png\"\u003e\n  \u003cimg alt=\"Nodestream Logo\" src=\"./docs/img/NodeSteamLogo_Small_Dark.png\" width=\"300\" align=\"right\"\u003e\n\u003c/picture\u003e\n\n\n\u003e A Declarative framework for Building, Maintaining, and Analyzing Graph Data 🚀\n\n[![Continuous Integration](https://github.com/nodestream-proj/nodestream/actions/workflows/ci.yaml/badge.svg)](https://github.com/nodestream-proj/nodestream/actions/workflows/ci.yaml)\n[![codecov](https://codecov.io/gh/nodestream-proj/nodestream/branch/main/graph/badge.svg?token=HAPEVKQ6OQ)](https://codecov.io/gh/nodestream-proj/nodestream)\n[![ApacheV2 License](https://img.shields.io/badge/License-Apache%202.0-yellow.svg)](https://opensource.org/license/apache-2-0/)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-14-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n\n**Nodestream allows you to work with graphs declaratively.** With nodestream, you unlock a bounty of features purpose built for working with graphs. \nSemantically model your graph and map labels and properties directly to your data. \nBetter yet, you are not locked into your choices. \nNodestream works with you as you evolve your application by providing migration utilities to change your data schema. \nNodestream even decouples you from the underyling database technology so you can even change databases.\n\n#### Highlights\n\n- Connect to data sources like [Kafka](https://nodestream-proj.github.io/docs/docs/reference/extractors/#streamextractor), [files](https://nodestream-proj.github.io/docs/docs/reference/extractors/#unifiedfileextractor), [apis](https://nodestream-proj.github.io/docs/docs/reference/extractors/#simpleapiextractor), and [more](https://nodestream-proj.github.io/docs/docs/reference/extractors/)!\n- Evolve your application over time with database migrations ([Docs](https://nodestream-proj.github.io/docs/docs/tutorials-intermediate/working-with-migrations/))\n- Use your favorite Graph Database to fit any tech stack ([Docs](https://nodestream-proj.github.io/docs/docs/category/database-support/))\n- Clean up your own data with TTLs ([Docs](https://nodestream-proj.github.io/docs/docs/tutorials-intermediate/removing-data/#implementing-a-ttl-pipeline))\n- Infinite Customizability Since Nearly Everything is Pluggable!\n\n\u003e [Website](https://nodestream-proj.github.io/docs/) • [Blog](https://nodestream-proj.github.io/docs/blog/) • [Discussions](https://github.com/orgs/nodestream-proj/discussions)  • [Contributing](#contributing) • [Contributing Developer Guides](https://nodestream-proj.github.io/docs/docs/category/developer-reference/) • [Talks from Maintainers](https://www.youtube.com/watch?v=2F-xx4LcTng\u0026list=PLUiAbWRQecSOorv_V6TzfUBoIZyf-6r6R\u0026pp=gAQBiAQB)\n\n## Features \n\nNodestream has a pleasant CLI interface to get new projects up and running fast. \n\n![Demo](https://raw.githubusercontent.com/nodestream-proj/nodestream/e94d0faa024c0f8da1e83a4ff6d83746504d197e/docs/img/demo.gif)\n\nNot a fan of the defaults? You can change out databases very easily\n\n![Using Another Database](https://nodestream-proj.github.io/docs/assets/images/neptune-2c1c78b173e824fc1e824f54287e467f.gif)\n\nThen you can start to model your data and nodestream will evolve your database for you. No more messing with constraints or writing database queries. \n\n![Running Migrations](https://nodestream-proj.github.io/docs/assets/images/migrations-1ede1ab3d5438cdca24d66cfa6d66231.gif)\n\n## Getting Started\n\nConviced? Install nodestream with `pip` to get started. \n\n```bash\n  pip install nodestream\n  nodestream new --database neo4j my_project \u0026\u0026 cd my_project\n  nodestream run sample -v\n```\n\nWe highly recommend following our tutorials [here](https://nodestream-proj.github.io/docs/docs/category/tutorial---basics/)\n\n## Packages \n\nNodestream is built on a Highly Pluggable and Modular Architecture. Thus... we have a lot of packages to keep track of. \n\n| Package                      \t| Description                                                                   \t| Version                                                                                                                        \t|\n|------------------------------\t|-------------------------------------------------------------------------------\t|--------------------------------------------------------------------------------------------------------------------------------\t|\n| `nodestream`                 \t| The core library. Declarative ingestion.                                      \t| ![PyPI Version](https://badge.fury.io/py/nodestream.svg)                                 \t|\n| `nodestream-plugin-neo4j`     \t| Neo4j database connector.                                                     \t| ![PyPI Version](https://badge.fury.io/py/nodestream-plugin-neo4j.svg)         \t|\n| `nodestream-plugin-neptune`  \t| AWS Neptune database connector.                                               \t| ![PyPI Version](https://badge.fury.io/py/nodestream-plugin-neptune.svg)   \t|\n| `nodestream-plugin-dotenv`   \t| Adds DotEnv integration.                                                      \t| ![PyPI Version](https://badge.fury.io/py/nodestream-plugin-dotenv.svg)     \t|\n| `nodestream-plugin-pedantic` \t| A series of lints to enforce reasonable naming standards, etc.                \t| ![PyPI Version](https://badge.fury.io/py/nodestream-plugin-pedantic.svg) \t|\n| `nodestream-plugin-shell`    \t| An integration with nodestream to run shell commands.                         \t| ![PyPI Version](https://badge.fury.io/py/nodestream-plugin-shell.svg)       \t|\n| `nodestream-plugin-sbom`     \t| Import SBOM files in CycloneDX and SPDX into an opinionated graph data model. \t| ![PyPI Version](https://badge.fury.io/py/nodestream-plugin-sbom.svg)         \t|\n| `nodestream-plugin-akamai`   \t| Parse Akamai properties, redirect configs, and much more and ingests them.    \t| ![PyPI Version](https://badge.fury.io/py/nodestream-plugin-akamai.svg)     \t|\n| `nodestream-plugin-k8s`      \t| In incubation. A plugin that orchestrates Nodestream on k8s.                  \t| ![PyPI Version](https://badge.fury.io/py/nodestream-plugin-k8s.svg)           \t|\n\n\n## Contributors\n\nNodestream is a community project. We welcome all contributions. \nBe sure to checkout or [Contributing Docs](https://nodestream-proj.github.io/docs/docs/category/developer-reference/) and our [Code of Conduct](./CODE_OF_CONDUCT.md) before contributing. \n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/zprobst\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/22159145?v=4?s=100\" width=\"100px;\" alt=\"Zach Probst\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eZach Probst\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=zprobst\" title=\"Code\"\u003e💻\u003c/a\u003e \u003ca href=\"https://github.com/nodestream-proj/nodestream/pulls?q=is%3Apr+reviewed-by%3Azprobst\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e \u003ca href=\"#maintenance-zprobst\" title=\"Maintenance\"\u003e🚧\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/ccloes\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/1000538?v=4?s=100\" width=\"100px;\" alt=\"Chad Cloes\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eChad Cloes\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=ccloes\" title=\"Code\"\u003e💻\u003c/a\u003e \u003ca href=\"https://github.com/nodestream-proj/nodestream/pulls?q=is%3Apr+reviewed-by%3Accloes\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e \u003ca href=\"#maintenance-ccloes\" title=\"Maintenance\"\u003e🚧\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/angelosantos4\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/142852840?v=4?s=100\" width=\"100px;\" alt=\"asantos4\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003easantos4\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=angelosantos4\" title=\"Code\"\u003e💻\u003c/a\u003e \u003ca href=\"https://github.com/nodestream-proj/nodestream/pulls?q=is%3Apr+reviewed-by%3Aangelosantos4\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e \u003ca href=\"#maintenance-angelosantos4\" title=\"Maintenance\"\u003e🚧\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/grantleehoffman\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/603848?v=4?s=100\" width=\"100px;\" alt=\"Grant Hoffman\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eGrant Hoffman\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=grantleehoffman\" title=\"Code\"\u003e💻\u003c/a\u003e \u003ca href=\"https://github.com/nodestream-proj/nodestream/pulls?q=is%3Apr+reviewed-by%3Agrantleehoffman\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/khneal\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/40273388?v=4?s=100\" width=\"100px;\" alt=\"khneal\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ekhneal\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=khneal\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/orozen\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/62594754?v=4?s=100\" width=\"100px;\" alt=\"orozen\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eorozen\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=orozen\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://sites.google.com/view/ai4allrobotics\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/66497192?v=4?s=100\" width=\"100px;\" alt=\"Sophia Don Tranho\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSophia Don Tranho\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=sophiadt\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/bechbd\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/6898505?v=4?s=100\" width=\"100px;\" alt=\"bechbd\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ebechbd\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=bechbd\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/yasonk\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/6750414?v=4?s=100\" width=\"100px;\" alt=\"yasonk\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eyasonk\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=yasonk\" title=\"Code\"\u003e💻\u003c/a\u003e \u003ca href=\"https://github.com/nodestream-proj/nodestream/pulls?q=is%3Apr+reviewed-by%3Ayasonk\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/stuartio\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/22449467?v=4?s=100\" width=\"100px;\" alt=\"Stuart Macleod\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eStuart Macleod\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=stuartio\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Cole-Greer\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/112986082?v=4?s=100\" width=\"100px;\" alt=\"Cole Greer\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eCole Greer\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=Cole-Greer\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Aclucas1\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/21301692?v=4?s=100\" width=\"100px;\" alt=\"Austin Lucas\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAustin Lucas\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=Aclucas1\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/rreddy2\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/18334818?v=4?s=100\" width=\"100px;\" alt=\"rreddy2\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003erreddy2\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=rreddy2\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://jondotcomdotorg.net/\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/314260?v=4?s=100\" width=\"100px;\" alt=\"Jon Bristow\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJon Bristow\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/nodestream-proj/nodestream/commits?author=jbristow\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n  \u003ctfoot\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" size=\"13px\" colspan=\"7\"\u003e\n        \u003cimg src=\"https://raw.githubusercontent.com/all-contributors/all-contributors-cli/1b8533af435da9854653492b1327a23a4dbd0a10/assets/logo-small.svg\"\u003e\n          \u003ca href=\"https://all-contributors.js.org/docs/en/bot/usage\"\u003eAdd your contributions\u003c/a\u003e\n        \u003c/img\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tfoot\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\n### Contributing \n\nNeed a quick reference guide on how to contribute? Here you go!\n\n#### Getting Setup \n\nTo get started you'll need to install poery.\n\n```bash\ncurl -sSL https://install.python-poetry.org | python3 -\n```\n\nYou then can install the project dependencies with the following command:\n\n```bash\npoetry install\n```\n\nNo need to active a virtual environment. Poetry handles that for you with `poetry run` and `poetry shell`.\n\n#### Running Tests\n\nTo run tests for the entire project, run the following command:\n\n```bash\npoetry run pytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnodestream-proj%2Fnodestream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnodestream-proj%2Fnodestream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnodestream-proj%2Fnodestream/lists"}