Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/nodestream-proj/nodestream
A Declarative framework for Building, Maintaining, and Analyzing Graph Data
https://github.com/nodestream-proj/nodestream
api athena aws cli data-engineering data-lake data-science declarative etl framework graph graphql kafka knowledge-graph neo4j python s3 security visualization yaml
Last synced: about 1 month ago
JSON representation
A Declarative framework for Building, Maintaining, and Analyzing Graph Data
- Host: GitHub
- URL: https://github.com/nodestream-proj/nodestream
- Owner: nodestream-proj
- License: apache-2.0
- Created: 2023-01-25T16:15:25.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-09-19T16:36:29.000Z (about 2 months ago)
- Last Synced: 2024-09-28T12:15:15.841Z (about 2 months ago)
- Topics: api, athena, aws, cli, data-engineering, data-lake, data-science, declarative, etl, framework, graph, graphql, kafka, knowledge-graph, neo4j, python, s3, security, visualization, yaml
- Language: Python
- Homepage: https://nodestream-proj.github.io/docs/
- Size: 3.24 MB
- Stars: 36
- Watchers: 2
- Forks: 11
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
README
# Nodestream
> A Declarative framework for Building, Maintaining, and Analyzing Graph Data 🚀
[](https://github.com/nodestream-proj/nodestream/actions/workflows/ci.yaml)
[](https://codecov.io/gh/nodestream-proj/nodestream)
[](https://opensource.org/license/apache-2-0/)
[](#contributors-)
**Nodestream allows you to work with graphs declaratively.** With nodestream, you unlock a bounty of features purpose built for working with graphs.
Semantically model your graph and map labels and properties directly to your data.
Better yet, you are not locked into your choices.
Nodestream works with you as you evolve your application by providing migration utilities to change your data schema.
Nodestream even decouples you from the underyling database technology so you can even change databases.
#### Highlights
- Connect to data sources like [Kafka](https://nodestream-proj.github.io/docs/docs/reference/extractors/#streamextractor), [files](https://nodestream-proj.github.io/docs/docs/reference/extractors/#unifiedfileextractor), [apis](https://nodestream-proj.github.io/docs/docs/reference/extractors/#simpleapiextractor), and [more](https://nodestream-proj.github.io/docs/docs/reference/extractors/)!
- Evolve your application over time with database migrations ([Docs](https://nodestream-proj.github.io/docs/docs/tutorials-intermediate/working-with-migrations/))
- Use your favorite Graph Database to fit any tech stack ([Docs](https://nodestream-proj.github.io/docs/docs/category/database-support/))
- Clean up your own data with TTLs ([Docs](https://nodestream-proj.github.io/docs/docs/tutorials-intermediate/removing-data/#implementing-a-ttl-pipeline))
- Infinite Customizability Since Nearly Everything is Pluggable!
> [Website](https://nodestream-proj.github.io/docs/) • [Blog](https://nodestream-proj.github.io/docs/blog/) • [Discussions](https://github.com/orgs/nodestream-proj/discussions) • [Contributing](#contributing) • [Contributing Developer Guides](https://nodestream-proj.github.io/docs/docs/category/developer-reference/) • [Talks from Maintainers](https://www.youtube.com/watch?v=2F-xx4LcTng&list=PLUiAbWRQecSOorv_V6TzfUBoIZyf-6r6R&pp=gAQBiAQB)
## Features
Nodestream has a pleasant CLI interface to get new projects up and running fast.
Not a fan of the defaults? You can change out databases very easily
Then you can start to model your data and nodestream will evolve your database for you. No more messing with constraints or writing database queries.
## Getting Started
Conviced? Install nodestream with `pip` to get started.
```bash
pip install nodestream
nodestream new --database neo4j my_project && cd my_project
nodestream run sample -v
```
We highly recommend following our tutorials [here](https://nodestream-proj.github.io/docs/docs/category/tutorial---basics/)
## Packages
Nodestream is built on a Highly Pluggable and Modular Architecture. Thus... we have a lot of packages to keep track of.
| Package | Description | Version |
|------------------------------ |------------------------------------------------------------------------------- |-------------------------------------------------------------------------------------------------------------------------------- |
| `nodestream` | The core library. Declarative ingestion. |  |
| `nodestream-plugin-neo4j` | Neo4j database connector. |  |
| `nodestream-plugin-neptune` | AWS Neptune database connector. |  |
| `nodestream-plugin-dotenv` | Adds DotEnv integration. |  |
| `nodestream-plugin-pedantic` | A series of lints to enforce reasonable naming standards, etc. |  |
| `nodestream-plugin-shell` | An integration with nodestream to run shell commands. |  |
| `nodestream-plugin-sbom` | Import SBOM files in CycloneDX and SPDX into an opinionated graph data model. |  |
| `nodestream-plugin-akamai` | Parse Akamai properties, redirect configs, and much more and ingests them. |  |
| `nodestream-plugin-k8s` | In incubation. A plugin that orchestrates Nodestream on k8s. |  |
## Contributors
Nodestream is a community project. We welcome all contributions.
Be sure to checkout or [Contributing Docs](https://nodestream-proj.github.io/docs/docs/category/developer-reference/) and our [Code of Conduct](./CODE_OF_CONDUCT.md) before contributing.
Zach Probst
💻 👀 🚧
Chad Cloes
💻 👀 🚧
asantos4
💻 👀 🚧
Grant Hoffman
💻 👀
khneal
💻
orozen
💻
Sophia Don Tranho
💻
bechbd
💻
yasonk
💻 👀
Stuart Macleod
💻
Cole Greer
💻
Add your contributions
### Contributing
Need a quick reference guide on how to contribute? Here you go!
#### Getting Setup
To get started you'll need to install poery.
```bash
curl -sSL https://install.python-poetry.org | python3 -
```
You then can install the project dependencies with the following command:
```bash
poetry install
```
No need to active a virtual environment. Poetry handles that for you with `poetry run` and `poetry shell`.
#### Running Tests
To run tests for the entire project, run the following command:
```bash
poetry run pytest
```