{"id":20749105,"url":"https://github.com/g-research/spark-dgraph-connector","last_synced_at":"2026-01-11T17:57:30.267Z","repository":{"id":36984164,"uuid":"267683863","full_name":"G-Research/spark-dgraph-connector","owner":"G-Research","description":"A connector for Apache Spark and PySpark to Dgraph databases.","archived":false,"fork":false,"pushed_at":"2024-09-24T12:21:43.000Z","size":1468,"stargazers_count":43,"open_issues_count":50,"forks_count":12,"subscribers_count":16,"default_branch":"main","last_synced_at":"2024-12-13T01:41:54.809Z","etag":null,"topics":["dgraph","gr-oss","pyspark","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/G-Research.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-28T19:58:11.000Z","updated_at":"2024-09-24T12:21:44.000Z","dependencies_parsed_at":"2023-01-17T11:32:38.997Z","dependency_job_id":"6559899b-af8d-42e4-af7b-0173bdf3fe24","html_url":"https://github.com/G-Research/spark-dgraph-connector","commit_stats":null,"previous_names":[],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Research%2Fspark-dgraph-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Research%2Fspark-dgraph-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Research%2Fspark-dgraph-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/G-Research%2Fspark-dgraph-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/G-Research","download_url":"https://codeload.github.com/G-Research/spark-dgraph-connector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230523755,"owners_count":18239444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dgraph","gr-oss","pyspark","spark"],"created_at":"2024-11-17T08:21:03.931Z","updated_at":"2026-01-11T17:57:30.194Z","avatar_url":"https://github.com/G-Research.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark Dgraph Connector\n\nThis project provides an [Apache Spark](https://spark.apache.org/) connector\nfor [Dgraph databases](https://dgraph.io/) in Scala and Python.\nIt comes with a [Spark Data Source](https://spark.apache.org/docs/latest/sql-data-sources.html)\nto read graphs from a Dgraph cluster directly into\n[DataFrames](https://spark.apache.org/docs/latest/sql-programming-guide.html),\n[GraphX](https://spark.apache.org/docs/latest/graphx-programming-guide.html) or\n[GraphFrames](https://graphframes.github.io/graphframes/docs/_site/index.html).\nThe connector supports [filter pushdown](https://github.com/apache/spark/blob/v3.0.0/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.java#L30),\n[projection pushdown](https://github.com/apache/spark/blob/v3.0.0/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownRequiredColumns.java#L31)\nand partitioning by orthogonal dimensions [predicates](#partitioning-by-predicates) and [nodes](#partitioning-by-uids).\n\nExample Scala code:\n\n```scala\nimport org.apache.spark.sql.DataFrame\n\nval target = \"localhost:9080\"\n\nimport org.apache.spark.graphx._\nimport uk.co.gresearch.spark.dgraph.graphx._\nval graph: Graph[VertexProperty, EdgeProperty] = spark.read.dgraph.graphx(target)\nval edges: RDD[Edge[EdgeProperty]] = spark.read.dgraph.edges(target)\nval vertices: RDD[(VertexId, VertexProperty)] = spark.read.dgraph.vertices(target)\n\nimport org.graphframes.GraphFrame\nimport uk.co.gresearch.spark.dgraph.graphframes._\nval graph: GraphFrame = spark.read.dgraph.graphframes(target)\nval edges: DataFrame = spark.read.dgraph.edges(target)\nval vertices: DataFrame = spark.read.dgraph.vertices(target)\n\nimport org.apache.spark.sql.DataFrame\nimport uk.co.gresearch.spark.dgraph.connector._\nval triples: DataFrame = spark.read.dgraph.triples(target)\nval edges: DataFrame = spark.read.dgraph.edges(target)\nval nodes: DataFrame = spark.read.dgraph.nodes(target)\n```\n\nExample Python code (pyspark ≥3.0, see [PySpark Shell and Python script](#pyspark-shell-and-python-script)):\n\n```python\nfrom pyspark.sql import DataFrame\nfrom gresearch.spark.dgraph.connector import *\n\ntriples: DataFrame = spark.read.dgraph.triples(\"localhost:9080\")\nedges: DataFrame = spark.read.dgraph.edges(\"localhost:9080\")\nnodes: DataFrame = spark.read.dgraph.nodes(\"localhost:9080\")\n```\n\n## Features\n\nThe connector provides the following features:\n\n- **Scala, Java, and Python API**: Supports Spark's Scala, Java, and Python Dataset API.\n- **Spark Graph API support**: Supports Spark Graph APIs [GraphX](#graphx) and [GraphFrames](#graphframes).\n- **Various graph representations**: Graph data are available as [Spark Datasets in various formats](#dataframe):\n  Triples, Nodes and Edge Datasets, fully typed, wide or long format.\n- **Filter Pushdown**: Many filters applied to the Graph Datasets will be [pushed down to Dgraph](#filter-pushdown)\n  so that only the relevant sub-graph is read from the cluster.\n- **Projection Pushdown**: Only [actually used columns of the Graph Datasets](#projection-pushdown) will be read from the Dgraph cluster.\n- **Multi-language strings**: Strings values of the same predicate can be stored in [multiple languages](#language-strings).\n- **Spark metrics**: The connector collects [Spark metrics](#metrics) per partition providing insights in throughout\n  and timing of the communication to the Dgraph cluster.\n- **Graph Partitioning and Streaming**: Graph data are partitioned and streamed in small chunks from Dgraph into Spark.\n  This guarantees that graphs of any size can be read into Spark.\n\n## Limitations\n\nThe connector has the following known limitations:\n\n- **Read-only**: The connector does not support mutating the graph ([issue #8](https://github.com/G-Research/spark-dgraph-connector/issues/8)).\n- **Namespaces**: The connector can only read the default namespace ([issue #148](https://github.com/G-Research/spark-dgraph-connector/issues/148)).\n- **Authorization**: No authorization against Dgraph supported ([issue #149](https://github.com/G-Research/spark-dgraph-connector/issues/149)).\n- **Limited Lifetime of Transactions**: The connector optionally reads all partitions within the same transaction, but concurrent mutations [reduce the lifetime of that transaction](#transactions).\n- **Language tags**: The node source in wide mode cannot read string values with [language tags](https://dgraph.io/docs/tutorial-4/#strings-and-languages). All other sources and modes can read language strings.\n- **Filtering on language string**: The connector does not support filtering predicates with [Dgraph `@lang` directives](https://dgraph.io/docs/tutorial-4/#strings-and-languages).\n- **Facets**: The connector cannot read [facets](https://dgraph.io/docs/query-language/facets/).\n\n## Using Spark Dgraph Connector\n\nThe Spark Dgraph Connector is available for Spark 3.0, 3.1, 3.2, 3.3, 3.4 and 3.5, with Scala 2.12 and 2.13.\nUse Maven artifact ID `spark-dgraph-connector_2.12` or `spark-dgraph-connector_2.12`. The Spark version is part of the package version,\ni.e. 0.12.0-3.0, 0.12.0-3.1, 0.12.0-3.2, 0.12.0-3.3, 0.12.0-3.4 and 0.12.0-3.5, respectively.\n\n### SBT\n\nAdd this line to your `build.sbt` file to use the latest version for Spark 3.5:\n\n```sbt\nlibraryDependencies += \"uk.co.gresearch.spark\" %% \"spark-dgraph-connector\" % \"0.12.0-3.5\"\n```\n\n### Maven\n\nAdd this dependency to your `pom.xml` file to use the latest version:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003euk.co.gresearch.spark\u003c/groupId\u003e\n  \u003cartifactId\u003espark-dgraph-connector_2.13\u003c/artifactId\u003e\n  \u003cversion\u003e0.12.0-3.5\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### Spark Shell\n\nLaunch the Scala Spark REPL (Spark ≥3.0.0) with the Spark Dgraph Connector dependency (version ≥0.5.0) as follows:\n\n```shell script\nspark-shell --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.12.0-3.5\n```\n\n### PySpark Shell and Python script\n\nLaunch the Python Spark REPL (pyspark ≥3.0.0) with the Spark Dgraph Connector dependency (version ≥0.5.0) as follows:\n\n```shell script\npyspark --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.12.0-3.5\n```\n\nRun your Python scripts that use PySpark (pyspark ≥3.0.0) and the Spark Dgraph Connector (version ≥0.5.0) via `spark-submit`:\n\n```shell script\nspark-submit --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.12.0-3.5 [script.py]\n```\n\n## Examples\n\nThe following examples use a local Dgraph (≥20.03.3) instance setup as described in the\n[Dgraph Quickstart Guide](https://dgraph.io/docs/get-started).\nRun [Step 1](https://dgraph.io/docs/get-started/#step-1-run-dgraph) to start an instance,\na `DROP_ALL` for Dgraph ≥20.07.0 only,\n[Step 2](https://dgraph.io/docs/get-started/#step-2-run-mutation) to load example graph data, and\n[Step 3](https://dgraph.io/docs/get-started/#step-3-alter-schema) to add a schema. These steps are\nprovided in the following scripts:\n\n```shell script\n./dgraph-instance.start.sh\n./dgraph-instance.drop-all.sh  # for Dgraph ≥20.07.0 only\n./dgraph-instance.schema.sh\n./dgraph-instance.insert.sh\n```\n\nThe Dgraph version can optionally be set via `DGRAPH_TEST_CLUSTER_VERSION` environment variable.\n\nThe connection to Dgraph can be established via a `target`, which is the [hostname and gRPC port of a\nDgraph Alpha node](https://dgraph.io/docs/deploy/#cluster-setup) in the form `\u003chostname\u003e:\u003cport\u003e`.\nWith our example instance started above, we can use `localhost:9080` as the target.\n\n### Dgraph UI\n\nThe Dgraph UI Ratel can be used to query your local Dgraph instance. Open a browser and\ngot to https://play.dgraph.io/?latest. Connect to your local Dgraph via http://localhost:8080.\n\n### GraphX\n\nYou can load the entire Dgraph database into an\n[Apache Spark GraphX](https://spark.apache.org/docs/latest/graphx-programming-guide.html)\ngraph. For example:\n\n```scala\nimport uk.co.gresearch.spark.dgraph.graphx._\n\nval graph = spark.read.dgraph.graphx(\"localhost:9080\")\n```\n\nExample code to perform a [PageRank](https://spark.apache.org/docs/latest/graphx-programming-guide.html#pagerank)\ncomputation on this graph to test that the connector is working:\n\n```scala\nval pageRank = graph.pageRank(0.0001)\npageRank.vertices.foreach(println)\n```\n\n### GraphFrames\n\nYou can load the entire Dgraph database into a\n[GraphFrames](https://graphframes.github.io/graphframes/docs/_site/index.html) graph. For example:\n\n```scala\nimport uk.co.gresearch.spark.dgraph.graphframes._\n\nval graph: GraphFrame = spark.read.dgraph.graphframes(\"localhost:9080\")\n```\n\nExample code to perform a [PageRank](https://graphframes.github.io/graphframes/docs/_site/user-guide.html#pagerank)\ncomputation on this graph to test that the connector is working:\n\n```scala\nval pageRank = graph.pageRank.maxIter(10)\npageRank.run().triplets.show(false)\n```\n\nNote: Predicates get renamed when they are loaded from the Dgraph database. Any `.` (dot) in the name\nis replaced by a `_` (underscore). To guarantee uniqueness of names, underscores in the original predicate\nnames are replaced by two underscores. For instance, predicates `dgraph.type` and `release_date`\nbecome `dgraph_type` and `release__date`, respectively.\n\n### DataFrame\n\nDgraph data can be loaded into Spark DataFrames in various forms:\n\n- Triples\n  - fully typed values\n  - string values\n- Nodes\n  - fully typed properties\n  - wide schema\n- Edges\n\n#### Typed Triples\n\nYou can load the entire Dgraph database as triples into an [Apache Spark DataFrame](https://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes). For example:\n\n```scala\nimport uk.co.gresearch.spark.dgraph.connector._\n\nval triples = spark.read.dgraph.triples(\"localhost:9080\")\n```\n\nThe returned `DataFrame` has the following schema:\n\n    root\n     |-- subject: long (nullable = false)\n     |-- predicate: string (nullable = true)\n     |-- objectUid: long (nullable = true)\n     |-- objectString: string (nullable = true)\n     |-- objectLong: long (nullable = true)\n     |-- objectDouble: double (nullable = true)\n     |-- objectTimestamp: timestamp (nullable = true)\n     |-- objectBoolean: boolean (nullable = true)\n     |-- objectGeo: string (nullable = true)\n     |-- objectPassword: string (nullable = true)\n     |-- objectType: string (nullable = true)\n\nThe object value gets stored in exactly one of the `object*` (except `objectType`) columns, depending on the type of the value.\nThe `objectType` column provides the type of the object. Here is an example:\n\n|subject|predicate   |objectString                                  |objectLong|objectDouble|objectTimestamp    |objectBoolean|objectGeo|objectPassword|objectType|\n|:-----:|:----------:|:--------------------------------------------:|:--------:|:----------:|:-----------------:|:-----------:|:-------:|:------------:|:--------:|\n|1      |dgraph.type |Person                                        |null      |null        |null               |null         |null     |null          |string    |\n|1      |name        |Luke Skywalker                                |null      |null        |null               |null         |null     |null          |string    |\n|2      |dgraph.type |Person                                        |null      |null        |null               |null         |null     |null          |string    |\n|2      |name        |Princess Leia                                 |null      |null        |null               |null         |null     |null          |string    |\n|3      |dgraph.type |Film                                          |null      |null        |null               |null         |null     |null          |string    |\n|3      |name        |Star Wars: Episode IV - A New Hope            |null      |null        |null               |null         |null     |null          |string    |\n|3      |release_date|null                                          |null      |null        |1977-05-25 00:00:00|null         |null     |null          |timestamp |\n|3      |revenue     |null                                          |null      |7.75E8      |null               |null         |null     |null          |double    |\n|3      |running_time|null                                          |121       |null        |null               |null         |null     |null          |long      |\n\nThis model allows you to store the fully-typed triples in a `DataFrame`.\n\n#### String Triples\n\nThe triples can also be loaded in an un-typed, narrow form:\n\n```scala\nimport uk.co.gresearch.spark.dgraph.connector._\n\nspark\n  .read\n  .option(TriplesModeOption, TriplesModeStringOption)\n  .dgraph.triples(\"localhost:9080\")\n  .show\n```\n\nThe resulting `DataFrame` has the following schema:\n\n    root\n     |-- subject: long (nullable = false)\n     |-- predicate: string (nullable = true)\n     |-- objectString: string (nullable = true)\n     |-- objectType: string (nullable = true)\n\nThe object value gets stored as a string in `objectString`, and `objectType` provides you\nwith the actual type of the object. Here is an example:\n\n|subject|predicate   |objectString                                  |objectType|\n|:-----:|:----------:|:--------------------------------------------:|:--------:|\n|1      |dgraph.type |Person                                        |string    |\n|1      |name        |Luke Skywalker                                |string    |\n|2      |dgraph.type |Person                                        |string    |\n|2      |name        |Princess Leia                                 |string    |\n|3      |dgraph.type |Film                                          |string    |\n|3      |revenue     |7.75E8                                        |double    |\n|3      |running_time|121                                           |long      |\n|3      |starring    |1                                             |uid       |\n|3      |starring    |2                                             |uid       |\n|3      |starring    |6                                             |uid       |\n|3      |director    |7                                             |uid       |\n|3      |name        |Star Wars: Episode IV - A New Hope            |string    |\n|3      |release_date|1977-05-25 00:00:00.0                         |timestamp |\n\n#### Typed Nodes\n\nYou can load all nodes into a `DataFrame` in a fully-typed form. This contains all the nodes' properties but no edges to other nodes:\n\n```scala\nimport uk.co.gresearch.spark.dgraph.connector._\n\nspark.read.dgraph.nodes(\"localhost:9080\")\n```\n\nThe returned `DataFrame` has the following schema:\n\n    root\n     |-- subject: long (nullable = false)\n     |-- predicate: string (nullable = true)\n     |-- objectString: string (nullable = true)\n     |-- objectLong: long (nullable = true)\n     |-- objectDouble: double (nullable = true)\n     |-- objectTimestamp: timestamp (nullable = true)\n     |-- objectBoolean: boolean (nullable = true)\n     |-- objectGeo: string (nullable = true)\n     |-- objectPassword: string (nullable = true)\n     |-- objectType: string (nullable = true)\n\nThe schema of the returned `DataFrame` is very similar to the typed triples schema, except that there is no `objectUid` column linking to other nodes. Here is an example:\n\n|subject|predicate   |objectString                                  |objectLong|objectDouble|objectTimestamp    |objectBoolean|objectGeo|objectPassword|objectType|\n|:-----:|:----------:|:--------------------------------------------:|:--------:|:----------:|:-----------------:|:-----------:|:-------:|:------------:|:--------:|\n|1      |dgraph.type |Person                                        |null      |null        |null               |null         |null     |null          |string    |\n|1      |name        |Luke Skywalker                                |null      |null        |null               |null         |null     |null          |string    |\n|2      |dgraph.type |Person                                        |null      |null        |null               |null         |null     |null          |string    |\n|2      |name        |Princess Leia                                 |null      |null        |null               |null         |null     |null          |string    |\n|3      |dgraph.type |Film                                          |null      |null        |null               |null         |null     |null          |string    |\n|3      |revenue     |null                                          |null      |7.75E8      |null               |null         |null     |null          |double    |\n|3      |running_time|null                                          |121       |null        |null               |null         |null     |null          |long      |\n|3      |name        |Star Wars: Episode IV - A New Hope            |null      |null        |null               |null         |null     |null          |string    |\n|3      |release_date|null                                          |null      |null        |1977-05-25 00:00:00|null         |null     |null          |timestamp |\n\n#### Wide Nodes\n\nNodes can also be loaded in a wide, fully-typed format:\n\n```scala\nimport uk.co.gresearch.spark.dgraph.connector._\n\nspark\n  .read\n  .option(NodesModeOption, NodesModeWideOption)\n  .dgraph.nodes(\"localhost:9080\")\n```\n\nThe returned `DataFrame` has the following schema format, which is dependent on the schema of the underlying Dgraph database.\nNode properties are stored in typed columns and are ordered alphabetically (property columns start after the `subject` column):\n\n    root\n     |-- subject: long (nullable = false)\n     |-- dgraph.graphql.schema: string (nullable = true)\n     |-- dgraph.type: string (nullable = true)\n     |-- name: string (nullable = true)\n     |-- release_date: timestamp (nullable = true)\n     |-- revenue: double (nullable = true)\n     |-- running_time: long (nullable = true)\n\nNote: The graph schema could become very large and therefore the `DataFrame` could become prohibitively wide.\n\n|subject|dgraph.graphql.schema|dgraph.type|name                                          |release_date       |revenue|running_time|\n|:-----:|:-------------------:|:---------:|:--------------------------------------------:|:-----------------:|:-----:|:----------:|\n|1      |null                 |Person     |Luke Skywalker                                |null               |null   |null        |\n|2      |null                 |Person     |Princess Leia                                 |null               |null   |null        |\n|3      |null                 |Film       |Star Wars: Episode IV - A New Hope            |1977-05-25 00:00:00|7.75E8 |121         |\n|4      |null                 |Film       |Star Wars: Episode VI - Return of the Jedi    |1983-05-25 00:00:00|5.72E8 |131         |\n|5      |null                 |Film       |Star Trek: The Motion Picture                 |1979-12-07 00:00:00|1.39E8 |132         |\n|6      |null                 |Person     |Han Solo                                      |null               |null   |null        |\n|7      |null                 |Person     |George Lucas                                  |null               |null   |null        |\n|8      |null                 |Person     |Irvin Kernshner                               |null               |null   |null        |\n|9      |null                 |Person     |Richard Marquand                              |null               |null   |null        |\n|10     |null                 |Film       |Star Wars: Episode V - The Empire Strikes Back|1980-05-21 00:00:00|5.34E8 |124         |\n\nNote: The Wide Nodes source enforces the [predicate partitioner](#partitioning-by-predicates) to produce a single partition.\n\n#### Edges\n\nEdges can be loaded as follows:\n\n```scala\nimport uk.co.gresearch.spark.dgraph.connector._\n\nspark.read.dgraph.edges(\"localhost:9080\")\n```\n\nThe returned `DataFrame` has the following simple schema:\n\n    root\n     |-- subject: long (nullable = false)\n     |-- predicate: string (nullable = true)\n     |-- objectUid: long (nullable = false)\n\nThough there is only a single `object` column for the destination node, it is called `objectUid` to align with the `DataFrame` schemata above.\n\n|subject|predicate|objectUid|\n|:-----:|:-------:|:-------:|\n|3      |starring |1        |\n|3      |starring |2        |\n|3      |starring |6        |\n|3      |director |7        |\n|4      |starring |1        |\n|4      |starring |2        |\n|4      |starring |6        |\n|4      |director |9        |\n|10     |starring |1        |\n|10     |starring |2        |\n|10     |starring |6        |\n|10     |director |8        |\n\n## Language Strings\n\nPredicates marked in the Dgraph schema with the `@lang` directive can store string values in\n[multiple languages](https://dgraph.io/docs/tutorial-4/#strings-and-languages) at a time:\n\n```\n{\n  set {\n   _:sw3 \u003ctitle\u003e \"Star Wars: Episode VI - Return of the Jedi\" .\n   _:sw3 \u003ctitle@en\u003e \"Star Wars: Episode VI - Return of the Jedi\" .\n   _:sw3 \u003ctitle@zh\u003e \"星際大戰六部曲：絕地大反攻\" .\n   _:sw3 \u003ctitle@th\u003e \"สตาร์ วอร์ส เอพพิโซด 6: การกลับมาของเจได\" .\n   _:sw3 \u003ctitle@de\u003e \"Die Rückkehr der Jedi-Ritter\" .\n  }\n}\n```\n\nThe connector reads all these languages. Each of the predicate name contains the language\nin the form `predicate@language`:\n\n|subject|predicate            |objectString                                            |objectType|\n|:-----:|:-------------------:|:-------------------------------------------------------|:--------:|\n|6      |dgraph.type          |Film                                                    |string    |\n|6      |title                |Star Wars: Episode VI - Return of the Jedi              |string    |\n|6      |title@en             |Star Wars: Episode VI - Return of the Jedi              |string    |\n|6      |title@zh             |星際大戰六部曲：絕地大反攻                              |string    |\n|6      |title@th             |สตาร์ วอร์ส เอพพิโซด 6: การกลับมาของเจได                    |string    |\n|6      |title@de             |Die Rückkehr der Jedi-Ritter                            |string    |\n\n\n## Transactions\n\nDgraph isolates reads from writes through transactions. Since the connector initiates multiple reads\nwhile fetching the entire graph (partitioning), writes called [mutations](https://dgraph.io/docs/mutations/)\nshould be isolated in order to get a consistent snapshot of the graph.\n\nSetting the `dgraph.transaction.mode` option to `\"read\"` will cause the connector to read all partitions\nwithin the same transaction. However, this will cause an exception on the Dgraph cluster when too many\nmutations occur while reading partitions. With that option set to `\"none\"`, no such exception will\noccur but reads are not isolated from writes.\n\n## Filter Pushdown\n\nThe connector supports filter pushdown to improve efficiency when reading only sub-graphs.\nThis is supported only in conjunction with the [predicate partitioner](#partitioning-by-predicates).\nSpark filters can only be pushed for some column and data source because columns may have different meaning.\nColumns can be of the following types:\n\n|Column Type|Description|Type |Columns|Sources|\n|:---------:|-----------|:---:|:-----:|-------|\n|subject column|the subject of the row|`long`|`subject`|all [DataFrame sources](#dataframe)|\n|predicate column|the predicate of the row|`string`|`predicate`|all but [Wide Nodes source](#wide-nodes)|\n|predicate value column|the value of a specific predicate, column name is predicate name|*any*|one column for each predicate in the schema, e.g. `dgraph.type`|[Wide Nodes source](#wide-nodes)|\n|object value columns|object value of the row|`long`\u003cbr/\u003e`string`\u003cbr/\u003e`long`\u003cbr/\u003e`double`\u003cbr/\u003e`timestamp`\u003cbr/\u003e`boolean`\u003cbr/\u003e`geo`\u003cbr/\u003e`password`|`objectUid`\u003cbr\u003e`objectString`\u003cbr\u003e`objectLong`\u003cbr\u003e`objectDouble`\u003cbr\u003e`objectTimestamp`\u003cbr\u003e`objectBoolean`\u003cbr\u003e`objectGeo`\u003cbr\u003e`objectPassword`|all but [Wide Nodes source](#Wide-nodes)\u003cbr\u003ethe [String Triples source](#string-triples) has only `objectString`\u003cbr\u003ethe [Typed Nodes source](#typed-nodes) lacks the `objectUid`\u003cbr\u003ethe [Edges source](#edges) has only `objectUid`|\n|object type column|the type of the object|`string`|`objectType`|[String Triples source](#string-triples), [Typed Triples source](#typed-triples) and [Typed Nodes source](#typed-nodes)|\n\nThe following table lists all supported Spark filters:\n\n|Spark Filter|Supported Columns|Example|\n|:----------:|-------|-------|\n|`EqualTo`   |\u003cul\u003e\u003cli\u003esubject column\u003c/li\u003e\u003cli\u003epredicate column\u003c/li\u003e\u003cli\u003epredicate value column\u003c/li\u003e\u003cli\u003eobject type column\u003c/li\u003e\u003cli\u003eobject value columns\u003cul\u003e\u003cli\u003efor [Typed Triples source](#typed-triples) and [Edge source](#edges) only with single predicate per partition\u003c/li\u003e\u003cli\u003enot for [String Triples source](#string-triples)\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e|\u003cul\u003e\u003cli\u003e`.where($\"subject\" === 1L)`\u003c/li\u003e\u003cli\u003e`.where($\"predicate\" === \"dgraph.type\")`\u003c/li\u003e\u003cli\u003e`.where($\"dgraph.type\" === \"Person\")`\u003c/li\u003e\u003cli\u003e`.where($\"objectType\" === \"string\")`\u003c/li\u003e\u003cli\u003e`.where($\"objectLong\" === 123)`\u003c/li\u003e\u003c/ul\u003e|\n|`In`        |\u003cul\u003e\u003cli\u003esubject column\u003c/li\u003e\u003cli\u003epredicate column\u003c/li\u003e\u003cli\u003epredicate value column\u003c/li\u003e\u003cli\u003eobject type column\u003c/li\u003e\u003cli\u003eobject value columns\u003cul\u003e\u003cli\u003efor [Typed Triples source](#typed-triples) and [Edge source](#edges) only with single predicate per partition\u003c/li\u003e\u003cli\u003enot for [String Triples source](#string-triples)\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e|\u003cul\u003e\u003cli\u003e`.where($\"subject\".isin(1L,2L))`\u003c/li\u003e\u003cli\u003e`.where($\"predicate\".isin(\"release_date\", \"revenue\"))`\u003c/li\u003e\u003cli\u003e`.where($\"dgraph.type\".isin(\"Person\",\"Film\"))`\u003c/li\u003e\u003cli\u003e`.where($\"objectType\".isin(\"string\",\"long\"))`\u003c/li\u003e\u003cli\u003e`.where($\"objectLong\".isin(123,456))`\u003c/li\u003e\u003c/ul\u003e|\n|`IsNotNull` |\u003cul\u003e\u003cli\u003epredicate value column\u003c/li\u003e\u003cli\u003eobject value columns (not for [String Triples source](#string-triples))\u003c/li\u003e\u003c/ul\u003e|\u003cul\u003e\u003cli\u003e`.where($\"dgraph.type\".isNotNull)`\u003c/li\u003e\u003cli\u003e`.where($\"objectLong\".isNotNull)`\u003c/li\u003e\u003c/ul\u003e|\n\n## Projection Pushdown\n\nThe connector supports projection pushdown to improve efficiency when reading only sub-graphs.\nA projection in Spark terms is a `select` operation that selects only a subset of a DataFrame's columns.\nThe [Wide Nodes source](#wide-nodes) supports projection pushdown on all [predicate value columns](#filter-pushdown).\n\n## Filter and Projection Pushdown Example\n\nThe following query uses filter and projection pushdown. First we define a wide node `DataFrame`:\n\n```scala\nval df =\n  spark.read\n    .options(Map(\n      NodesModeOption -\u003e NodesModeWideOption,\n      PartitionerOption -\u003e PredicatePartitionerOption\n    ))\n    .dgraph.nodes(\"localhost:9080\")\n```\n\nThen we select some columns (projection) and rows (filter):\n\n```scala\ndf\n  .select($\"subject\", $\"`dgraph.type`\", $\"revenue\")  // projection\n  .where($\"revenue\".isNotNull)                       // filter\n  .show()\n```\n\nThis selects the columns `subject`, `dgraph.type` and `revenue` for only those rows that actually have a value for `revenue`.\nThe underlying query to Dgraph simplifies from (the full graph):\n\n    {\n      pred1 as var(func: has(\u003cdgraph.graphql.schema\u003e))\n      pred2 as var(func: has(\u003cdgraph.graphql.xid\u003e))\n      pred3 as var(func: has(\u003cdgraph.type\u003e))\n      pred4 as var(func: has(\u003cname\u003e))\n      pred5 as var(func: has(\u003crelease_date\u003e))\n      pred6 as var(func: has(\u003crevenue\u003e))\n      pred7 as var(func: has(\u003crunning_time\u003e))\n\n      result (func: uid(pred1,pred2,pred3,pred4,pred5,pred6,pred7)) {\n        uid\n        \u003cdgraph.graphql.schema\u003e\n        \u003cdgraph.graphql.xid\u003e\n        \u003cdgraph.type\u003e\n        \u003cname\u003e\n        \u003crelease_date\u003e\n        \u003crevenue\u003e\n        \u003crunning_time\u003e\n      }\n    }\n\nto (selected predicates and nodes only):\n\n    {\n      pred1 as var(func: has(\u003crevenue\u003e))\n\n      result (func: uid(pred1)) {\n        uid\n        \u003cdgraph.type\u003e\n        \u003crevenue\u003e\n      }\n    }\n\nThe response is faster as only relevant data are transferred between Dgraph and Spark.\n\n|subject|dgraph.type|revenue|\n|:-----:|:---------:|:-----:|\n|      4|       Film| 7.75E8|\n|      5|       Film| 5.34E8|\n|      6|       Film| 5.72E8|\n|      9|       Film| 1.39E8|\n\n## Metrics\n\nThe connector (Spark ≥3.0 only) collects metrics per partition that provide insights in throughout and timing of the communication\nto the Dgraph cluster. For each request to Dgraph (a chunk), the number of received bytes, uids and retrieval time are recorded and\nsummed per partition. The values can be seen on the Spark UI for the respective stages that performs the read:\n\n![Dgraph metrics as shown on Spark UI Stages page](static/accumulators.png \"Dgraph metrics as shown on Spark UI Stages page\")\n\nThe connector uses [Spark Accumulators](http://spark.apache.org/docs/1.6.2/api/java/org/apache/spark/Accumulator.html)\nto collect these metrics. They can be accessed by the Spark driver via a `SparkListener`:\n\n```scala\nval handler = new SparkListener {\n  override def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit =\n    stageCompleted.stageInfo.accumulables.values.foreach(println)\n}\n\nspark.sparkContext.addSparkListener(handler)\nspark.read.dgraph.triples(\"localhost:9080\").count()\n```\n\n\nThe following metrics are available:\n\n|Metric|Description|\n|------|-----------|\n|`Dgraph Bytes`|Size of JSON responses from the Dgraph cluster in Byte.|\n|`Dgraph Chunks`|Number of requests sent to the Dgraph cluster.|\n|`Dgraph Time`|Time waited for Dgraph to respond in Seconds.|\n|`Dgraph Uids`|Number of Uids read.|\n\n\n## Partitioning\n\nPartitioning your Dgraph graph is essential to be able to load large quantities of graph data into Spark.\nSpark splits data into partitions, where ideally all partitions have the same size and are of decent size.\nPartitions that are too large will kill your Spark executor as they won't fit into memory. When partitions\nare too small your Spark job becomes inefficient and slow, but will not fail.\n\nEach partition connects to the Dgraph cluster and reads a specific sub-graph. Partitions are non-overlapping.\n\nThis connector provides various ways to partition your graph. When the default partitioning does not work for your\nspecific use case, try a more appropriate partitioning scheme.\n\n### Partitioner\n\nThe following `Partitioner` implementations are available:\n\n| Partitioner                       | partition by              | Description | Use Case |\n|:---------------------------------:|:-------------------------:|-------------|----------|\n| Singleton                         | _nothing_                 | Provides a single partition for the entire graph. | Unit Tests and small graphs that fit into a single partition. Can be used for large graphs if combined with a \"by uid\" partitioner. |\n| Predicate                         | predicate                 | Provides multiple partitions with at most `P` predicates per partition where `P` defaults to `1000`. Picks multiple predicates from the same Dgraph group. | Large graphs where each predicate fits into a partition, otherwise combine with Uid Range partitioner. Skewness of predicates reflects skewness of partitions. |\n| Uid Range                         | uids                      | Each partition has at most `N` uids where `N` defaults to `1000000`. | Large graphs where single `uid`s fit into a partition. Can be combined with any predicate partitioner, otherwise induces internal Dgraph cluster communication across groups. |\n| Predicate + Uid Range _(default)_ | predicates + uids         | Partitions by predicate first (see Predicate Partitioner), then each partition gets partitioned by uid (see Uid Partitioner) | Graphs of any size. |\n\n\n#### Partitioning by Predicates\n\nThe Dgraph data can be partitioned by predicates. Each partition then contains a distinct set of predicates.\nThe number of predicates per partition can be configured via `dgraph.partitioner.predicate.predicatesPerPartition`,\nwhich defaults to `1000`.\n\nPredicate partitions connect only to alpha nodes that contain those predicates. Hence, these reads are all\nlocally to the alpha nodes and induce no Dgraph cluster internal communication.\n\n#### Partitioning by Uids\n\nA `uid` represents a node or vertex in Dgraph terminology. An \"Uid Range\" partitioning splits\nthe graph by the subject of the graph triples. This can be combined with predicate partitioning,\nwhich serves as an orthogonal partitioning. Without predicate partitioning, `uid` partitioning\ninduces internal Dgraph cluster communication across the groups.\n\nThe uid partitioning always works on top of a predicate partitioner. If none is defined a singleton partitioner is used.\nThe number of uids of each underlying partition has to be estimated. Once the number of uids is estimated,\nthe partition is further split into ranges of that uid space.\n\nThe space of existing `uids` is split into ranges of `N` `uids` per partition. The `N` defaults to `1000000`\nand can be configured via `dgraph.partitioner.uidRange.uidsPerPartition`. The `uid`s are allocated to partitions in ascending order.\nSuch a split will not be done if more than `dgraph.partitioner.uidRange.maxPartitions` partitions would be created. This defaults to `10000`.\nIf vertex size is skewed and a function of `uid`, then partitions will be skewed as well.\n\nNote: With uid partitioning, the chunk size configured via `dgraph.chunkSize` should be at least a 10th of\nthe number of uids per partition configured via `dgraph.partitioner.uidRange.uidsPerPartition` to avoid\ninefficiency due to chunks overlapping with partition borders. When your result is sparse w.r.t. the uid space\nset the chunk size to 100th or less.\n\n\u003c!-- there is only one estimator left, no need to mention this until we have another\nThe estimator can be selected with the `dgraph.partitioner.uidRange.estimator` option. These estimators are available:\n\n##### Cluster maxUid (formerly maxLeaseId)\n\nThe Dgraph cluster [maintains a maxUid](https://dgraph.io/docs/deploy/dgraph-zero/#more-about-the-state-endpoint), which is the largest existing uid.\nIt grows as new uids are added to the cluster, so it serves as an upper estimate of the actual largest uid.\nCompared to the count estimator it is very cheap to retrieve this value.\nThis estimator can be selected with the `maxUid` value.\n--\u003e\n\n### Streamed Partitions\n\nThe connector reads each partition from Dgraph in a streamed fashion. It splits up a partition into smaller chunks,\nwhere each chunk contains `100000` uids. This chunk size can be configured via `dgraph.chunkSize`.\nEach chunk sends a single query to Dgraph. The chunk size limits the size of the result.\nDue to the low memory footprint of the connector, Spark could read your entire graph via a single partition\n(you would have to `repartition` the read DataFrame to make Spark shuffle the data properly).\nHowever, this would be slow, but it proves the connector can handle any size of graph with fixed executor memory requirement.\n\n## Logging\n\nThe connector uses Spark's Log4j standard logging framework. Add the following line to your `log4j.properties` to set\nthe log level of the connector specifically:\n\n    log4j.logger.uk.co.gresearch.spark.dgraph.connector=DEBUG\n\nSee `SPARK_HOME/conf/log4j.properties.template` for a template file.\n\n## Testing\n\nSome unit tests require a Dgraph (≥20.03.3) cluster running at `localhost:9080`. It has to be set up as\ndescribed in the [Examples](#examples) section. If that cluster is not running, the unit tests will\nlaunch and set up such a cluster for you. This requires `docker` to be installed on your machine\nand will make the tests take longer. If you run those tests frequently it is recommended you run\nthe cluster setup yourself.\n\nYou can set the Dgraph version that is started automatically\nby setting environment variable `DGRAPH_TEST_CLUSTER_VERSION`.\nThe default version is defined in `uk.co.gresearch.spark.dgraph.DgraphTestCluster.DgraphDefaultVersion`.\n\nThe Python code can be tested with `pytest`:\n\n```shell script\nPYTHONPATH=\"python:python/test\" python -m pytest python/test\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg-research%2Fspark-dgraph-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fg-research%2Fspark-dgraph-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg-research%2Fspark-dgraph-connector/lists"}