{"id":13399220,"url":"https://github.com/GraphChi/graphchiDB-scala","last_synced_at":"2025-03-14T03:31:47.284Z","repository":{"id":16072010,"uuid":"18816418","full_name":"GraphChi/graphchiDB-scala","owner":"GraphChi","description":"*Experimental* GraphChi-DB graph database with computational capabilities","archived":false,"fork":false,"pushed_at":"2015-10-07T22:44:02.000Z","size":2338,"stargazers_count":79,"open_issues_count":0,"forks_count":24,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-07-31T19:18:38.596Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GraphChi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-15T21:22:25.000Z","updated_at":"2023-04-12T13:50:42.000Z","dependencies_parsed_at":"2022-09-24T11:02:05.190Z","dependency_job_id":null,"html_url":"https://github.com/GraphChi/graphchiDB-scala","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GraphChi%2FgraphchiDB-scala","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GraphChi%2FgraphchiDB-scala/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GraphChi%2FgraphchiDB-scala/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GraphChi%2FgraphchiDB-scala/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GraphChi","download_url":"https://codeload.github.com/GraphChi/graphchiDB-scala/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243520603,"owners_count":20304173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T19:00:35.411Z","updated_at":"2025-03-14T03:31:47.272Z","avatar_url":"https://github.com/GraphChi.png","language":"Scala","funding_links":[],"categories":["数据库","Scala"],"sub_categories":["Spring Cloud框架"],"readme":"# GraphChi-DB\n\nGraphChi-DB is a scalable, embedded, single-computer online graph database that can also execute similar large-scale graph computation as [GraphChi](https://github.com/graphchi).\nit has been developed by [Aapo Kyrola](http://www.cs.cmu.edu/~akyrola) as part of his Ph.D. thesis.\n\nIt can handle graphs with billions of edges on just a laptop or PC, fast!\n\nGraphChi-DB is written in Scala, with some Java code. Generally, you need to know Scala quite well to be able to use it.\n\n**IMPORTANT: GraphChi-DB is early release, research code. It is buggy, it has awful API, and it is provided with no guarantees.\nDO NOT USE IT FOR ANYTHING IMPORTANT.**\n\n\n## License\n\nGraphChi-DB is licensed under the Apache License, Version 2.0.\nSee any source code file for full license information.\n\n## Requirements\n\n* Solid-state Drive (SSD)  (for good query performance; hard drive might work well if enough RAM)\n* 8 GB of RAM is recommended. Might work with less.\n* Scala 2.9\n\n ## Discussion group\n\n Please join the discussion group (shared with GraphChi users).\n\n http://groups.google.com/group/graphchi-discuss\n\n\n ## Publication\n\nYou can read about GraphChi-DB's design, evaluation and motivation from pre-print [GraphChi-DB: Simple Design For a Scalable Graph Database System - on Just a PC](http://arxiv.org/abs/1403.0701).\n\n# Documentation\n\nNot provided. We suggest you look through the examples (see below), to get the basic idea.\n\n# Examples\n\n**In general, you must be very familiar with Scala and Java development to be able to use GraphChi-DB.** \n\n## To run\n\nBest way to explore GraphChi-DB is to load the project into an IDE (such as Eclipse or IntelliJ IDEA), and use the Scala Console. This will allow you to interactively explore the data.\nYou can also include GraphChi-DB easily in your Scala project.\n\n\nYou can also launch Scala console directly from Maven, but remember to specify the proper JVM parameters, as follows:\n\n```\nexport MAVEN_OPTS=\"-Xmx6G -Djava.library.path=lib\" \u0026\u0026 mvn compile scala:console\n```\n\nFollowing JVM parameters are recommended:\n```\n   -Xmx6G -ea\n```\n\n**Note:** With less than 5 gigabyte of RAM, the database may crash (silently) in an out-of-memory exception. This is because its buffers overflow, and the database cannot yet manage its own memory usage.\nHowever, do NOT add more memory to the JVM, because GraphChi-DB uses memory mapping of the operating system to manage the data. It is better to leave as much memory for the OS to use for memory mapping.\n\n\n## Example: Social Network\n\nSource: https://github.com/GraphChi/graphchiDB-scala/blob/master/src/main/scala/edu/cmu/graphchidb/examples/SocialNetworkExample.scala\n\nThis example creates a graph from a social network data. For each edge, we add a timestamp and weight column. In this example, this values\nare populated with random values, so they are just provided as an example.\n\nIn addition, the example computes continuously Pagerank for each vertex. You can also invoke a connected components computation. Note that these computations operate in the background and if the graph changes, also their results can become incorrect. \n\nThis example shows also how to compute simple social network recommendations based on friends-of-friends. \n\n### Input data\n\nTo run this example, you need to have some input graph. You can try these:\n* Live Journal: http://snap.stanford.edu/data/soc-LiveJournal1.html\n* Twitter graph (2010): http://an.kaist.ac.kr/traces/WWW2010.html\n\nRemember to configure the proper filenames in the code. Look for variable \"sourceFile\".\nAlso set the \"numShards\" parameter properly. Based on the expected number of edges, use one shard / 5 million edges. That is, for 1 billion edgers, use 200 shards. \n\n\n### Example session\n\n```scala\n   import  edu.cmu.graphchidb.examples.SocialNetworkExample._\n\n   // To initialize DB (you need to do this only on your first session)\n   startIngest\n\n   // Some testing\n   recommendFriends(8737)\n   recommendFriends(2419)\n   recommendFriendsLimited(2419)\n   \n   // In and out neighbors\n   DB.queryIn(DB.originalToInternalId(2409), 0)\n   DB.queryOut(DB.originalToInternalId(8737), 0)\n\n   // To run connected components\n   connectedComponents()\n\n   // After a while, you can ask\n   ccAlgo.printStats\n\n   // To get a vertex component label (which might not be yet the final one)\n   ccAlgo.vertexDataColumn.get.get(DB.originalToInternalId(8737)).get\n\n   // To get pagerank of a vertex (note, that it is being continuously updated), so this\n   // just looks up the value.\n   pagerankCol.get(DB.originalToInternalId(8737)).get\n```\n\n### Advanced Social Network Analysis\n\nSource: https://github.com/GraphChi/graphchiDB-scala/blob/master/src/main/scala/edu/cmu/graphchidb/examples/SubgraphFrequencies.scala\n\nApplication __SubgraphFrequencies__ allows reproducing some of the results in paper [Ugander, Backstrom, Kleinberg: \"Subgraph frequencies: Mapping the empirical and extremal geography of large graph collections\"] (http://www2013.org/proceedings/p1307.pdf).\n\nGiven a graph (social network), it will sample a set of vertices, and for each of the vertices load the induced subgraph of the vertex's neighbors (excluding itself).\nFrom each induced neighborhood graph, we sample thousands of times three vertices and record the type of subgraph induced by the three vertices.\nThe frequencies are output to a file that can be plotted using R or Matlab to see the subgraph frequency distribution of the graph.\n\nNote: you need to first create the graph (see previous example).\n\nExample session:\n```scala\n   // Usage:\n       import edu.cmu.graphchidb.examples.SubgraphFrequencies._\n\n       // To compute subgraph freqs of a given vertex neighborhood\n       computeThreeVertexSubgraphFrequencies(inducedNeighborhoodGraph(DB.originalToInternalId(2419)))\n\n\n      // To produce data similar to used in Figure 1 of Ugander et. al.:\n       computeDistribution(500)\n   )\n\n```\n\nThe source code contains a sample R script to plot the distribution.\n\nTo read more about statistical graph analysis using GraphChi-DB, check the wiki page: [Tutorial-Statistical-Graph-Analysis](https://github.com/GraphChi/graphchiDB-scala/wiki/Tutorial-Statistical-Graph-Analysis).\n\n\n\n## Example: Wikipedia Graph\n\nSource: https://github.com/GraphChi/graphchiDB-scala/blob/master/src/main/scala/edu/cmu/graphchidb/examples/WikipediaGraph.scala\n\nThis example application reads Wikipedia's SQL dumps and creates a graph of the wikipedia pages. The process takes a while (a couple of hours)\nbecause the program needs to resolve page names to page IDs.\n\nIt allows you to then find shortest paths between pages.\nYou can consider extending it with Pagerank, Connected Components etc. by using techniques from the previous example.\n\n### Input data\n\nWikipedia dumps are available here: http://dumps.wikimedia.org/enwiki/latest/\n\nYou need **two** files: \n* Pagelinks: for example http://dumps.wikimedia.org/enwiki/20140402/enwiki-20140402-pagelinks.sql.gz\n* Per-page data: http://dumps.wikimedia.org/enwiki/20140402/enwiki-20140402-page.sql.gz\n\nThe example application includes a very hacky, but very fast, parser for the data. Note that it has been tested only with the English wikipedia data.\n\n\n### Example session: Ingest\n\nYou need to run this only on your first session:\n```scala\n  import  edu.cmu.graphchidb.examples.WikipediaGraph._\n\n  // If first time, populate the DB (takes 4 hours on SSD, MacBook Pro)\n  populate()\n```\n\n### Example session: Play\n\nNote, that the first query will take a long time as the application needs to compute the page title index. Also, it takes time before the graph is fully cached (using memory mapping).\n\n```scala\n  import  edu.cmu.graphchidb.examples.WikipediaGraph._\n  shortestPath(\"Barack_Obama\", \"Sauli_Niinisto\")\n  shortestPath(\"Helsinki\", \"Pittsburgh\")\n  shortestPath(\"Carnegie_Mellon_University\", \"Graph\")\n  shortestPath(\"Rabbit\", \"Empire_State_Building\")\n```\n\n## Example: Movie Database and Recommender\n\nSource: https://github.com/GraphChi/graphchiDB-scala/blob/master/src/main/scala/edu/cmu/graphchidb/examples/MovieDatabase.scala\n\nThis example creates a database of movies and a graph of user - movie ratings. After the database has been populated, you can run matrix factorization using the Alternating Least Squares (ALS) algorithm and use it to recommend movies to a user. Note, that this is overly simple recommender algorithm and unlikely to produce very good results.\n\n### Data\n\nYou can get Netflix Prize movie rating database from here:  http://www.select.cs.cmu.edu/code/graphlab/datasets/\n\nDownload the \"netflix_mm\" file.\n\n\n### Example session\n\nIngest:\n```scala\n import edu.cmu.graphchidb.examples.MovieDatabase._\n  startIngest\n```\n\nRecommend:\n```scala\n  import edu.cmu.graphchidb.examples.MovieDatabase._\n  recommendForUser(X)\n```\n\n### Notes\nThis example is a bit awkward: GraphChi-DB does not support typed vertices, so we need to use different range of IDs for movie and user vertices. Variable \"userIdOffset\" is used for this. The example functions in the app include the translation.\n\n\n# Notes\n\n## Online\n\nAn added edge is immediately visible to queries and subsequent computation. That is, you can query the database while inserting new data.\nThere is currently no batch mode to insert edges. Still, even in the online mode you can expect to be able to insert over 100K edges per second.\n\n## ID-mapping\n\nGraphChi-DB maps original vertex IDs to an internal ID space. Functions that ask for \"origId\" expect you to provide the original vertex ID (from the source data).\nTypically you need to map IDs returned by the database into original IDs. See the example applications for examples. The GraphChiDB class has two functions to map between the ID spaces:\n\n```scala\nDB.originalToInternalId(origId)\nDB.internalToOriginalId(internalId)\n```\n\nThis mapping is awkward, but crucial for the performance of the database. See the publication for more information.\n\n\n## Crash?\n\nSometimes GraphChi-DB seems to get silently stuck. This is caused by an Out-of-Memory error that is not being, for some reason, thrown and logged.\nYou should allocate 5 gigabytes of RAM to the JVM: -Xmx5G\n\n## Durability\n\nNew edges are first added to in-memory buffers. When the buffers are full, they are flushed to disk. If the computer crashes when the edges are in buffers, they are lost!\nGraphChi-DB has also a durable mode which logs all edges to a file before adding them to buffers (see the configuration file). However, there is currently no method to recover after a crash.\n\nGraphChi-DB uses a shutdown hook to flush the buffered edges to disk. It is important not to force-kill the JVM! You can also manually call\n\n```java\nDB.flushAllBuffers()\n\n```\n\nAgain, see the example applications...\n\n\n## Number of shards\n\nWhen creating a database, you need to specify the number of shards. A rule of thumb: if you expect N edges, use N / 5 million shards. So for 1 billion edges, use 200 shards, for 5 billion, maybe 500 is fine. \n\n## Debug log\n\nInside the database directory, there is a debug log-file that is written by the database.\n\n```\n   tail -f *debug.txt*\n```\n\n## Deleting database\n\nTo delete your database, just remove the directory containing the database. You need to restart your application as well.\n\n## Native Library for Sorting\n\nTo get better performance, you can enable a native library for sort operations. It is provided in binary under lib/, and can be compiled from src/jni/ (although I have had trouble compiling it on Linux).\n\nTo enable, add this to the command line:\n```\n  ... -Djava.library.path=lib/\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGraphChi%2FgraphchiDB-scala","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FGraphChi%2FgraphchiDB-scala","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGraphChi%2FgraphchiDB-scala/lists"}