Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/wey-gu/nebulagraph-ai

(Pre Alpha)NebulaGraph AI High-Level API, do Graph Algo, Analytics in 4 lines of code.
https://github.com/wey-gu/nebulagraph-ai

graph graph-algorithms hacktoberfest nebulagraph networkx spark

Last synced: 4 days ago
JSON representation

(Pre Alpha)NebulaGraph AI High-Level API, do Graph Algo, Analytics in 4 lines of code.

Awesome Lists containing this project

README

        

NebulaGraph AI Suite(ng_ai)


NebulaGraph AI Suite with 4 line code to run Graph Algo on NebulaGraph


for NebulaGraph


Docker Extension


GitHub release (latest by date)


License


PyPI version


pdm-managed


Tests


Docker Image


Docker Image


Docker Image

Python 3.9

---

**Documentation**: https://github.com/wey-gu/nebulagraph-ai#documentation

**Source Code**: https://github.com/wey-gu/nebulagraph-ai

---

NebulaGraph AI Suite for Python (ng_ai) is a powerful Python library that offers APIs for data scientists to effectively read, write, analyze, and compute data in NebulaGraph.

With the support of a single-machine engine(NetworkX), or distributed computing environment using Spark, we could perform Graph Analysis and Algorithms on top of NebulaGraph in less than 10 lines of code, in unified and intuitive API.

## Quick Start in 5 Minutes

**Option 1**: Try on your Desktop. Go with [NebulaGraph Docker Extension](https://hub.docker.com/extensions/weygu/nebulagraph-dd-ext)!

**Option 2**: On Linux Server? Go with Nebula-Up πŸ‘‡πŸ»

- Set up env with Nebula-Up following [this guide](https://github.com/wey-gu/nebulagraph-ai/blob/main/docs/Environment_Setup.md).
- Install ng_ai with pip from the Jupyter Notebook with http://localhost:8888 (password: `nebula`).
- Open the demo notebook and run cells one by one.
- Check the [API Reference](https://github.com/wey-gu/nebulagraph-ai/blob/main/docs/API.md)

## Installation

```bash
pip install ng_ai
```

## Usage

### Call from nGQL

See more details in the [docs](https://github.com/wey-gu/nebulagraph-ai/blob/main/docs/ng_ai_API_Gateway.md)

```cypher
RETURN ng_ai("pagerank", ["follow"], ["degree"], "spark",
{space: "basketballplayer", max_iter: 10}, {write_mode: "insert"})
```

### Spark Engine Examples

See also: [examples/spark_engine.ipynb](https://github.com/wey-gu/nebulagraph-ai/blob/main/examples/spark_engine.ipynb)

Run Algorithm on top of NebulaGraph:

> Note, there is also a query mode, refer to [examples](https://github.com/wey-gu/nebulagraph-ai/blob/main/examples/spark_engine.ipynb) or [docs](https://github.com/wey-gu/nebulagraph-ai/blob/main/docs/API.md) for more details.

```python
from ng_ai import NebulaReader

# read data with spark engine, scan mode
reader = NebulaReader(engine="spark")
reader.scan(edge="follow", props="degree")
df = reader.read()

# run pagerank algorithm
pr_result = df.algo.pagerank(reset_prob=0.15, max_iter=10)
```

Write back to NebulaGraph:

```python
from ng_ai import NebulaWriter
from ng_ai.config import NebulaGraphConfig

config = NebulaGraphConfig()

properties = {"louvain": "cluster_id"}

writer = NebulaWriter(
data=df_result, sink="nebulagraph_vertex", config=config, engine="spark")
writer.set_options(
tag="louvain", vid_field="_id", properties=properties,
batch_size=256, write_mode="insert",)
writer.write()
```

Then we could query the result in NebulaGraph:

```cypher
MATCH (v:louvain)
RETURN id(v), v.louvain.cluster_id LIMIT 10;
```

### NebulaGraph Engine Examples

Basically the same as Spark Engine, but with `engine="nebula"`, refer to [examples](https://github.com/wey-gu/nebulagraph-ai/blob/main/examples/networkx_engine.ipynb) or [docs](https://github.com/wey-gu/nebulagraph-ai/blob/main/docs/API.md) for more details.

```diff
- reader = NebulaReader(engine="spark")
+ reader = NebulaReader(engine="nebula")
```

## Documentation

[Environment Setup](https://github.com/wey-gu/nebulagraph-ai/blob/main/docs/Environment_Setup.md)

[API Reference](https://github.com/wey-gu/nebulagraph-ai/blob/main/docs/API.md)

## How it works

ng_ai is a unified abstraction layer for different engines, the current implementation is based on Spark, NetworkX, DGL, and NebulaGraph, but it's easy to extend to other engines like Flink, GraphScope, PyG, etc.

```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Spark Cluster β”‚
β”‚ .─────. .─────. .─────. .─────. β”‚
β”‚ ; : ; : ; : ; : β”‚
β”Œβ”€β–Άβ”‚ : ; : ; : ; : ; β”‚
β”‚ β”‚ β•² β•± β•² β•± β•² β•± β•² β•± β”‚
β”‚ β”‚ `───' `───' `───' `───' β”‚
Algo Spark β”‚
Engineβ””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
└─── β”‚ β”‚
β”‚ NebulaGraph AI Suite(ngai) β”‚ ngai-api │◀─┐
β”‚ β”‚ β”‚ β”‚
β”‚ └─────────── β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ Reader β”‚ β”‚ Algo β”‚ β”‚ Writer β”‚ β”‚ GNN β”‚ β”‚ β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β–Άβ”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β” └──────┐ β”‚ β”‚
β”‚ β”‚ β–Ό β–Ό β–Ό β–Ό β”‚ β”‚
β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”Œβ”€β”€β”€ β”‚ SparkEngine β”‚ β”‚ NebulaEngine β”‚ β”‚ NetworkX β”‚β”‚ DGLEngineβ”‚ β”‚ β”‚
β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚ β”‚ Spark β”‚
β”‚ β”‚ └────────Reader ────────────┐ β”‚
β”‚ Spark Query Mode β”‚ β”‚
β”‚ Reader β”‚ β”‚
β”‚Scan Mode β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ ngai-udf│◀─────────────┐
β”‚ β”‚ β”‚ β”‚ └────────── β”‚
β”‚ β”‚ β”‚ NebulaGraph Graph Engine Nebula-GraphD β”‚ ngai-GraphD β”‚ β”‚
β”‚ β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚ β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ NebulaGraph Storage Engine β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ β”‚ β”‚ β”‚
β”‚ └─▢│ Nebula-StorageD β”‚ Nebula-Metad β”‚ β”‚
β”‚ β”‚ β”‚ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ RETURN ng_ai("pagerank", ["follow"], ["degree"], "spark", {space:"basketballplayer"}) β”‚β”€β”€β”˜
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ β”‚ from ng_ai import NebulaReader β”‚
β”‚ β”‚ β”‚
β”‚ β”‚ # read data with spark engine, scan mode β”‚
β”‚ β”‚ reader = NebulaReader(engine="spark") β”‚
β”‚ β”‚ reader.scan(edge="follow", props="degree") β”‚
└──│ df = reader.read() β”‚
β”‚ β”‚
β”‚ # run pagerank algorithm β”‚
β”‚ pr_result = df.algo.pagerank(reset_prob=0.15, max_iter=10) β”‚
β”‚ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

### Spark Engine Prerequisites
- Spark 2.4, 3.0(not yet tested)
- [NebulaGraph 3.4+](https://github.com/vesoft-inc/nebula)
- [NebulaGraph Spark Connector 3.4+](https://repo1.maven.org/maven2/com/vesoft/nebula-spark-connector/)
- [NebulaGraph Algorithm 3.1+](https://repo1.maven.org/maven2/com/vesoft/nebula-algorithm/)

### NebulaGraph Engine Prerequisites
- [NebulaGraph 3.4+](https://github.com/vesoft-inc/nebula)
- [NebulaGraph Python Client 3.4+](https://github.com/vesoft-inc/nebula-python)
- [NetworkX](https://networkx.org/)

## Contributing

See [HACKING.md](https://github.com/wey-gu/nebulagraph-ai/blob/main/HACKING.md) for details.

## License

This project is licensed under the terms of the Apache License 2.0.