{"id":13667525,"url":"https://github.com/ldbc/ldbc_snb_datagen_spark","last_synced_at":"2025-12-17T21:15:13.356Z","repository":{"id":25474998,"uuid":"28905625","full_name":"ldbc/ldbc_snb_datagen_spark","owner":"ldbc","description":"Synthetic graph generator for the LDBC Social Network Benchmark, running on Spark","archived":false,"fork":false,"pushed_at":"2025-01-03T19:34:09.000Z","size":303939,"stargazers_count":169,"open_issues_count":10,"forks_count":58,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-03-28T14:06:14.681Z","etag":null,"topics":["snb"],"latest_commit_sha":null,"homepage":"https://ldbcouncil.org/benchmarks/snb","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ldbc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"contributing.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"code_of_conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-01-07T08:28:41.000Z","updated_at":"2025-01-28T17:06:52.000Z","dependencies_parsed_at":"2023-01-14T07:00:58.724Z","dependency_job_id":"4ea63d3c-4a6a-47cb-b577-b5950bb4c3b5","html_url":"https://github.com/ldbc/ldbc_snb_datagen_spark","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldbc%2Fldbc_snb_datagen_spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldbc%2Fldbc_snb_datagen_spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldbc%2Fldbc_snb_datagen_spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ldbc%2Fldbc_snb_datagen_spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ldbc","download_url":"https://codeload.github.com/ldbc/ldbc_snb_datagen_spark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247198450,"owners_count":20900080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["snb"],"created_at":"2024-08-02T07:00:39.626Z","updated_at":"2025-12-17T21:15:13.290Z","avatar_url":"https://github.com/ldbc.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"![LDBC_LOGO](https://raw.githubusercontent.com/wiki/ldbc/ldbc_snb_datagen_spark/images/ldbc-logo.png)\n\n# LDBC SNB Datagen (Spark-based)\n\n[![Build Status](https://circleci.com/gh/ldbc/ldbc_snb_datagen_spark.svg?style=svg)](https://circleci.com/gh/ldbc/ldbc_snb_datagen_spark)\n\nThe LDBC SNB Data Generator (Datagen) produces the datasets for the [LDBC Social Network Benchmark's workloads](https://ldbcouncil.org/benchmarks/snb/). The generator is designed to produce directed labelled graphs that mimic the characteristics of those graphs of real data. A detailed description of the schema produced by Datagen, as well as the format of the output files, can be found in the latest version of official [LDBC SNB specification document](https://github.com/ldbc/ldbc_snb_docs).\n\n:scroll: If you wish to cite the LDBC SNB, please refer to the [documentation repository](https://github.com/ldbc/ldbc_snb_docs#how-to-cite-ldbc-benchmarks).\n\n:warning: There are two different versions of the Datagen:\n\n* The [Hadoop-based Datagen](https://github.com/ldbc/ldbc_snb_datagen_hadoop/) generates the Interactive workload's SF1-1000 data sets.\n* For the BI workload, use the Spark-based Datagen (in this repository).\n* For the Interactive workloads's larger data sets, see the [conversion script in the driver repository](https://github.com/ldbc/ldbc_snb_interactive_driver/blob/main/scripts/README.md).\n\nFor each commit on the `main` branch, the CI deploys [freshly generated small data sets](https://ldbcouncil.org/ldbc_snb_datagen_spark/).\n\n## Quick start\n\n### Build the JAR\n\nTo assemble the JAR file with SBT, run:\n\n```bash\nsbt assembly\n```\n\n### Install Python tools\n\nSome of the build utilities are written in Python. To use them, you have to create a Python virtual environment\nand install the dependencies.\n\nE.g. with [pyenv](https://github.com/pyenv/pyenv) and [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv):\n\n```bash\npyenv install 3.7.13\npyenv virtualenv 3.7.13 ldbc_datagen_tools\npyenv local ldbc_datagen_tools\npip install -U pip\npip install ./tools\n```\n\nIf the environment already exists, activate it with:\n\n```bash\npyenv activate\n```\n\n### Running locally\n\nThe `./tools/run.py` script is intended for **local runs**. To use it, download and extract Spark as follows.\n\n#### Spark 3.2.x\n\nSpark 3.2.x is the recommended runtime to use. The rest of the instructions are provided assuming Spark 3.2.x.\n\nTo place Spark under `/opt/`:\n\n```bash\nscripts/get-spark-to-opt.sh\nexport SPARK_HOME=\"/opt/spark-3.2.2-bin-hadoop3.2\"\nexport PATH=\"${SPARK_HOME}/bin\":\"${PATH}\"\n```\n\nTo place it under `${HOME}/`:\n\n```bash\nscripts/get-spark-to-home.sh\nexport SPARK_HOME=\"${HOME}/spark-3.2.2-bin-hadoop3.2\"\nexport PATH=\"${SPARK_HOME}/bin\":\"${PATH}\"\n```\n\nBoth Java 8 and Java 11 are supported, but Java 17 is not (Spark 3.2.2 will fail, since it uses internal Java APIs and does not set the permissions appropriately).\n\n#### Building the project\n\nRun:\n\n```bash\nscripts/build.sh\n```\n\n#### Running the generator\n\nOnce you have Spark in place and built the JAR file, run the generator as follows:\n\n```bash\nexport PLATFORM_VERSION=$(sbt -batch -error 'print platformVersion')\nexport DATAGEN_VERSION=$(sbt -batch -error 'print version')\nexport LDBC_SNB_DATAGEN_JAR=$(sbt -batch -error 'print assembly / assemblyOutputPath')\n./tools/run.py \u003cruntime configuration arguments\u003e -- \u003cgenerator configuration arguments\u003e\n```\n\n#### Runtime configuration arguments\n\nThe runtime configuration arguments determine the amount of memory, number of threads, degree of parallelism. For a list of arguments, see:\n\n```bash\n./tools/run.py --help\n```\n\nTo generate a single `part-*` file, reduce the parallelism (number of Spark partitions) to 1.\n\n```bash\n./tools/run.py --parallelism 1 -- --format csv --scale-factor 0.003 --mode bi\n```\n#### Generator configuration arguments\n\nThe generator configuration arguments allow the configuration of the output directory, output format, layout, etc.\n\nTo get a complete list of the arguments, pass `--help` to the JAR file:\n\n```bash\n./tools/run.py -- --help\n```\n\n* Generating `csv-composite-merged-fk` files in **BI mode** resulting in compressed `.csv.gz` files:\n\n  ```bash\n  ./tools/run.py -- --format csv --scale-factor 0.003 --mode bi --format-options compression=gzip\n  ```\n\n* Generating `csv-composite-merged-fk` files in **BI mode** and generating factors:\n\n  ```bash\n  ./tools/run.py -- --format csv --scale-factor 0.003 --mode bi --generate-factors\n  ```\n\n* Generating CSVs in **raw mode**:\n\n  ```bash\n  ./tools/run.py -- --format csv --scale-factor 0.003 --mode raw --output-dir sf0.003-raw\n  ```\n\n* Generating Parquet files in **BI mode**:\n\n  ```bash\n  ./tools/run.py -- --format parquet --scale-factor 0.003 --mode bi\n  ```\n\n* Use epoch milliseconds encoded as longs for serializing date and datetime values in **BI mode** (this is equivalent to using the [`LongDateFormatter` in the Hadoop Datagen](https://github.com/ldbc/ldbc_snb_datagen_hadoop/blob/v0.3.8/src/main/java/ldbc/snb/datagen/util/formatter/LongDateFormatter.java)):\n\n  ```bash\n  ./tools/run.py -- --format csv --scale-factor 0.003 --mode bi --epoch-millis\n  ```\n\n* For the **BI mode**, the `--format-options` argument allows passing formatting options such as timestamp/date formats, the presence/abscence of headers (see the [Spark formatting options](https://spark.apache.org/docs/2.4.8/api/scala/index.html#org.apache.spark.sql.DataFrameWriter) for details), and whether quoting the fields in the CSV required:\n\n  ```bash\n  ./tools/run.py -- --format csv --scale-factor 0.003 --mode bi --format-options timestampFormat=MM/dd/y\\ HH:mm:ss,dateFormat=MM/dd/y,header=false,quoteAll=true\n  ```\n\n* The `--explode-attrs` argument implies one of the `csv-singular-{projected-fk,merged-fk}` formats, which has separate files to store multi-valued attributes (`email`, `speaks`).\n\n  ```bash\n  ./tools/run.py -- --format csv --scale-factor 0.003 --mode bi --explode-attrs\n  ```\n\n* The `--explode-edges` argument implies one of the `csv-{composite,singular}-projected-fk` formats, which has separate files to store many-to-one edges (e.g. `Person_isLocatedIn_City`, `Tag_hasType_TagClass`, etc.).\n\n  ```bash\n  ./tools/run.py -- --format csv --scale-factor 0.003 --mode bi --explode-edges\n  ```\n\n* The `--explode-attrs`  and `--explode-edges` arguments together imply the `csv-singular-projected-fk` format:\n\n  ```bash\n  ./tools/run.py -- --format csv --scale-factor 0.003 --mode bi --explode-attrs --explode-edges\n  ```\n\nTo change the Spark configuration directory, adjust the `SPARK_CONF_DIR` environment variable.\n\nA complex example:\n\n```bash\nexport SPARK_CONF_DIR=./conf\n./tools/run.py --parallelism 4 --memory 8G -- --format csv --format-options timestampFormat=MM/dd/y\\ HH:mm:ss,dateFormat=MM/dd/y --explode-edges --explode-attrs --mode bi --scale-factor 0.003\n```\n\nIt is also possible to pass a parameter file:\n\n```bash\n./tools/run.py -- --format csv --param-file params.ini\n```\n\n### Docker images\n\nSNB Datagen images are available via [Docker Hub](https://hub.docker.com/orgs/ldbc/repositories).\nThe image tags follow the pattern `${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION}`, e.g `ldbc/datagen-standalone:0.5.0-2.12_spark3.2`.\n\nWhen building images ensure that you [use BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/#to-enable-buildkit-builds).\n\n#### Standalone Docker image\n\nThe standalone image bundles Spark with the JAR and Python helpers, so you can run a workload in a container similarly to a local run, as you can\nsee in this example:\n\n```bash\nexport SF=0.003\nmkdir -p out_sf${SF}_bi   # create output directory\ndocker run \\\n    --mount type=bind,source=\"$(pwd)\"/out_sf${SF}_bi,target=/out \\\n    --mount type=bind,source=\"$(pwd)\"/conf,target=/conf,readonly \\\n    -e SPARK_CONF_DIR=/conf \\\n    ldbc/datagen-standalone:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION} \\\n    --parallelism 1 \\\n    -- \\\n    --format csv \\\n    --scale-factor ${SF} \\\n    --mode bi \\\n    --generate-factors\n```\n\nThe standalone Docker image can be built with the provided Dockerfile. To build, execute the following command from the repository directory:\n\n```bash\nexport PLATFORM_VERSION=$(sbt -batch -error 'print platformVersion')\nexport DATAGEN_VERSION=$(sbt -batch -error 'print version')\nexport DOCKER_BUILDKIT=1\ndocker build . --target=standalone -t ldbc/datagen-standalone:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION}\n```\n\n#### JAR-only image\n\nThe `ldbc/datagen-jar` image contains the assembly JAR, so it can bundled in your custom container:\n\n```docker\nFROM my-spark-image\nARG VERSION\nCOPY --from=ldbc/datagen-jar:${VERSION} /jar /lib/ldbc-datagen.jar\n```\n\nThe JAR-only Docker image can be built with the provided Dockerfile. To build, execute the following command from the repository directory:\n\n```bash\ndocker build . --target=jar -t ldbc/datagen-jar:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION}\n```\n\n#### Pushing to Docker Hub\n\nTo release a new snapshot version on Docker Hub, run:\n\n```bash\ndocker tag ldbc/datagen-jar:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION} ldbc/datagen-jar:latest\ndocker push ldbc/datagen-jar:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION}\ndocker push ldbc/datagen-jar:latest\ndocker tag ldbc/datagen-standalone:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION} ldbc/datagen-standalone:latest\ndocker push ldbc/datagen-standalone:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION}\ndocker push ldbc/datagen-standalone:latest\n```\n\nTo release a new stable version, create a new Git tag (e.g. by creating a new release on GitHub), then build the Docker image and push it.\n\n### Elastic MapReduce\n\nWe provide scripts to run Datagen on AWS EMR. See the README in the [`./tools/emr`](tools/emr) directory for details.\n\n## Graph schema\n\nThe graph schema is as follows:\n\n![](https://raw.githubusercontent.com/ldbc/ldbc_snb_docs/dev/figures/schema-comfortable.png)\n\n## Troubleshooting\n\n* When running the tests, they might throw a `java.net.UnknownHostException: your_hostname: your_hostname: Name or service not known` coming from `org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal`. The solution is to add an entry of your machine's hostname to the `/etc/hosts` file: `127.0.1.1 your_hostname`.\n* If you are using Docker and Spark runs out of space, make sure that Docker has enough space to store its containers. To move the location of the Docker containers to a larger disk, stop Docker, edit (or create) the `/etc/docker/daemon.json` file and add `{ \"data-root\": \"/path/to/new/docker/data/dir\" }`, then sync the old folder if needed, and restart Docker. (See [more detailed instructions](https://www.guguweb.com/2019/02/07/how-to-move-docker-data-directory-to-another-location-on-ubuntu/)).\n* If you are using a local Spark installation and run out of space in `/tmp` (`java.io.IOException: No space left on device`), set the `SPARK_LOCAL_DIRS` to point to a directory with enough free space.\n* The Docker image may throw the following error when generating factors `java.io.FileNotFoundException: /tmp/blockmgr-.../.../temp_shuffle_... (No file descriptors available)`. This error occurs on Fedora 36 host machines. Changing to an Ubuntu 22.04 host machine resolves the problem. Related issue: [#420](https://github.com/ldbc/ldbc_snb_datagen_spark/issues/420).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fldbc%2Fldbc_snb_datagen_spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fldbc%2Fldbc_snb_datagen_spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fldbc%2Fldbc_snb_datagen_spark/lists"}