{"id":23396509,"url":"https://github.com/mneedham/neo4j-spark-chicago","last_synced_at":"2025-04-08T17:32:40.516Z","repository":{"id":30357716,"uuid":"33910190","full_name":"mneedham/neo4j-spark-chicago","owner":"mneedham","description":null,"archived":false,"fork":false,"pushed_at":"2015-08-08T09:03:11.000Z","size":2629,"stargazers_count":30,"open_issues_count":3,"forks_count":18,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-02-14T13:50:22.625Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mneedham.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-14T04:10:48.000Z","updated_at":"2020-03-04T09:19:01.000Z","dependencies_parsed_at":"2022-07-31T09:48:08.058Z","dependency_job_id":null,"html_url":"https://github.com/mneedham/neo4j-spark-chicago","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fneo4j-spark-chicago","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fneo4j-spark-chicago/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fneo4j-spark-chicago/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mneedham%2Fneo4j-spark-chicago/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mneedham","download_url":"https://codeload.github.com/mneedham/neo4j-spark-chicago/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247892554,"owners_count":21013734,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-22T07:36:02.752Z","updated_at":"2025-04-08T17:32:40.483Z","avatar_url":"https://github.com/mneedham.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Importing Chicago Crime Dataset into Neo4j\n\nThe following are instructions for importing the open Chicago crime data set into Neo4j.\n\n* Download the link:https://data.cityofchicago.org/Public-Safety/Crimes-2001-to-present/ijzp-q8t2[Chicago crime data set]\n* Set a system variable containing the path to the data set. e.g.\n\n```\nexport CSV_FILE=Crimes_-_2001_to_present.csv\n```\n\n* Download a link:https://spark.apache.org/downloads.html[pre-installed Spark distribution] into this folder.\nThe `create_files.sh` script assumes the use of the link:http://apache.mirrors.spacedump.net/spark/spark-1.3.0/spark-1.3.0-bin-hadoop1.tgz[1.3.0 release pre built for Hadoop 1.X]\n* Unpack Spark\n\n```\ntar -xvf spark-1.3.0-bin-hadoop1.tgz\n```\n\n* Generate the CSV files that Neo4j import will use:\n\n```\n./create-files\n```\n\nYou should see the following at the beginning of the output:\n\n```\n$ ./create_files.sh\nSpark assembly has been built with Hive, including Datanucleus jars on classpath\nUsing /Users/markneedham/projects/neo4j-spark-chicago/Crimes_-_2001_to_present.csv\n...\n```\n\nThe following files will be generated:\n\n```\n$ ls -alh tmp/*.csv\n-rwxrwxrwx  1 markneedham  wheel   3.0K 14 Apr 06:48 /tmp/beats.csv\n-rwxrwxrwx  1 markneedham  wheel   217M 14 Apr 06:48 /tmp/crimes.csv\n-rwxrwxrwx  1 markneedham  wheel    84M 14 Apr 06:49 /tmp/crimesBeats.csv\n-rwxrwxrwx  1 markneedham  wheel   120M 14 Apr 06:49 /tmp/crimesPrimaryTypes.csv\n-rwxrwxrwx  1 markneedham  wheel   912B 14 Apr 06:48 /tmp/primaryTypes.csv\n```\n\nNow let's clean up the CSV files to get rid of empty columns:\n\n```\n./clean.sh\n```\n\n* link:http://neo4j.com/download/[Download Neo4j]\n\n* Unpack Neo4j\n\n```\ntar -xvf neo4j-enterprise-2.2.3-unix.tar.gz\n```\n\n* Create a Neo4j graph from the CSV files:\n\n\n```\n./import.sh [/path/to/csv/files] [/path/to/neo4j]\n```\n\n== FAQS\n\n===  Using a different version of Spark\n\nIf you want to use a different version of Spark you'll need to update the appropriate references in `create_files.sh` and `build.sbt`\n\n=== I can't generate the CSV files\n\nIf you forget to set the `CSV_FILE` environment variable or set it to a non-existent file you'll see the following error message:\n\n```\n$ ./create_files.sh\nSpark assembly has been built with Hive, including Datanucleus jars on classpath\nException in thread \"main\" java.lang.RuntimeException: Cannot find CSV file [null]\n...\n```\n\nSet the `CSV_FILE` environment variable and you're good to go:\n\n```\nexport CSV_FILE=Crimes_-_2001_to_present.csv\n```\n\n=== I made some changes to the project and want to build it\n\nIf you haven't made any changes to the source code of the project there's no need to build it - a JAR is checked in and referenced in `create_files.sh`.\nYou can, however, rebuild the JAR with the following command:\n\n```\nsbt clean package\n```\n\n=== I can't build this project\n\nsbt doesn't seem to honour `java_home` so if you're seeing the following error:\n\n```\n2015-04-14 13:57:50,116 INFO  [main] spark.SparkContext (Logging.scala:logInfo(59)) - Job finished: saveAsTextFile at GenerateCSVFiles.scala:51, took 8.292283862 s\nException in thread \"main\" java.lang.UnsupportedClassVersionError: MyFileUtil : Unsupported major.minor version 52.0\n        at java.lang.ClassLoader.defineClass1(Native Method)\n        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)\n        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)\n```\n\nYou should set the following environment variable:\n\n```\nexport JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/\n```\n\nAnd then retry:\n\n```\nPATH=$JAVA_HOME/bin:$PATH sbt clean package\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmneedham%2Fneo4j-spark-chicago","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmneedham%2Fneo4j-spark-chicago","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmneedham%2Fneo4j-spark-chicago/lists"}