{"id":17255798,"url":"https://github.com/lintool/bespin","last_synced_at":"2025-10-16T07:51:31.731Z","repository":{"id":57731483,"uuid":"46937749","full_name":"lintool/bespin","owner":"lintool","description":"Reference implementations of data-intensive algorithms in MapReduce and Spark","archived":false,"fork":false,"pushed_at":"2018-09-03T12:45:41.000Z","size":209,"stargazers_count":82,"open_issues_count":7,"forks_count":94,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-07-22T14:43:41.361Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://bespin.io/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lintool.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-26T16:43:58.000Z","updated_at":"2024-09-21T16:19:21.000Z","dependencies_parsed_at":"2022-09-06T17:31:20.590Z","dependency_job_id":null,"html_url":"https://github.com/lintool/bespin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lintool/bespin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lintool%2Fbespin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lintool%2Fbespin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lintool%2Fbespin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lintool%2Fbespin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lintool","download_url":"https://codeload.github.com/lintool/bespin/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lintool%2Fbespin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279167232,"owners_count":26118160,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-16T02:00:06.019Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T07:12:37.943Z","updated_at":"2025-10-16T07:51:31.702Z","avatar_url":"https://github.com/lintool.png","language":"Java","funding_links":[],"categories":["大数据"],"sub_categories":["微服务框架"],"readme":"# Bespin\n\n[![Build Status](https://travis-ci.org/lintool/bespin.svg?branch=master)](https://travis-ci.org/lintool/bespin)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/io.bespin/bespin/badge.svg)](https://maven-badges.herokuapp.com/maven-central/io.bespin/bespin)\n[![LICENSE](https://img.shields.io/badge/license-Apache-blue.svg?style=flat-square)](./LICENSE)\n\nBespin is a library that contains reference implementations of \"big data\" algorithms in MapReduce and Spark.\n\n## Getting Started\n\nBuild the package:\n\n```\n$ mvn clean package\n```\n\nGrab the data:\n\n```\n$ mkdir data\n$ curl http://lintool.github.io/bespin-data/Shakespeare.txt \u003e data/Shakespeare.txt\n$ curl http://lintool.github.io/bespin-data/p2p-Gnutella08-adj.txt \u003e data/p2p-Gnutella08-adj.txt\n```\n\nThe datasets are stored in the [Bespin data repo](https://github.com/lintool/bespin-data).\n\n+ The file `Shakespeare.txt` contains the [The Complete Works of William Shakespeare](http://www.gutenberg.org/ebooks/100) from [Project Gutenberg](http://www.gutenberg.org/).\n+ The file `p2p-Gnutella08-adj.txt` contains a [snapshot of the Gnutella peer-to-peer file sharing network from August 2002](http://snap.stanford.edu/data/p2p-Gnutella08.html), where nodes represent hosts in the Gnutella network topology and edges represent connections between the Gnutella hosts. This dataset is available from the [Stanford Network Analysis Project](http://snap.stanford.edu/).\n+ The tarball `taxi-data.tar.gz` contains a [one-day slice NY taxi data](http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml), chopped into one file per minute. See analyses in Todd Schneider's blog post [Analyzing 1.1 Billion NYC Taxi and Uber Trips, with a Vengeance](http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-taxi-and-uber-trips-with-a-vengeance/).\n\n## Word Count in MapReduce and Spark\n\nMake sure you've downloaded the Shakespeare collection (see \"Getting Started\" above). Running word count in Java MapReduce:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.wordcount.WordCount \\\n   -input data/Shakespeare.txt -output wc-jmr-combiner\n```\n\nTo enable the \"in-mapper combining\" optimization, use the `-imc` option.\n\nRunning word count in Scala MapReduce:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.scala.mapreduce.wordcount.WordCount \\\n   --input data/Shakespeare.txt --output wc-smr-combiner\n```\n\nTo enable the \"in-mapper combining\" optimization, use the `--imc` option.\n\nAnd finally, running word count in Spark:\n\n```\n$ spark-submit --class io.bespin.scala.spark.wordcount.WordCount target/bespin-1.0.5-SNAPSHOT-fatjar.jar \\\n    --input data/Shakespeare.txt --output wc-spark-default\n```\n\nTo enable the \"in-mapper combining\" optimization in Spark, use the `--imc` option (although this optimization doesn't do anything: exercise left to the reader as to why).\n\nCompare results to make sure they are the same:\n\n```\n$ hadoop fs -cat wc-jmr-combiner/part-r-0000* | awk '{print $1,$2;}' | sort \u003e counts.jmr.combiner.txt\n$ hadoop fs -cat wc-smr-combiner/part-r-0000* | awk '{print $1,$2;}' | sort \u003e counts.smr.combiner.txt\n$ hadoop fs -cat wc-spark-default/part-0000* | sed -E 's/^\\((.*),([0-9]+)\\)$/\\1 \\2/' | sort \u003e counts.spark.default.txt\n$ diff counts.jmr.combiner.txt counts.smr.combiner.txt\n$ diff counts.jmr.combiner.txt counts.spark.default.txt\n```\n\n**Tip:** We use `awk` in some cases and `sed` in others because `sed` does not accept control characters such as `\\t` (but GNU `sed` does), so you have to insert a literal tab in the command line. This is awkward: on Mac OS X, you need to type `^V^I`; so it's just easier with `awk`.\n\n## Computing Bigram Relative Frequencies in MapReduce\n\nMake sure you've downloaded the Shakespeare collection (see \"Getting Started\" above). Running a simple bigram count:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bigram.BigramCount \\\n   -input data/Shakespeare.txt -output bigram-count\n```\n\nComputing bigram relative frequencies using the \"pairs\" implementation:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bigram.ComputeBigramRelativeFrequencyPairs \\\n   -input data/Shakespeare.txt -output bigram-freq-mr-pairs -textOutput\n```\n\nComputing bigram relative frequencies using the \"stripes\" implementation:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bigram.ComputeBigramRelativeFrequencyStripes \\\n   -input data/Shakespeare.txt -output bigram-freq-mr-stripes -textOutput\n```\n\nTo obtain human-readable output, make sure to use the `-textOutput` option; otherwise, the job defaults to `SequenceFile` output.\n\nLet's spot-check the output to make sure the results are correct. For example, what are the bigrams that begin with \"dream\"?\n\n```\n$ hadoop fs -cat bigram-count/part* | grep '^dream '\ndream again\t2\ndream and\t2\ndream are\t1\ndream as\t1\ndream away\t2\n...\n```\n\nWhat is the sum of all these counts?\n\n```\n$ hadoop fs -cat bigram-count/part* | grep '^dream ' | cut -f 2 | awk '{sum+=$1} END {print sum}'\n79\n```\n\nConfirm that the numbers match the \"pairs\" implementation of the relative frequency computations:\n\n```\n$ hadoop fs -cat bigram-freq-mr-pairs/part* | grep '(dream, '\n```\n\nAnd the \"stripes\" implementation of the relative frequency computations:\n\n```\n$ hadoop fs -cat bigram-freq-mr-stripes/part* | awk '/^dream\\t/'\n```\n\n**Tip:** Note that `grep` in Mac OS X accepts `\\t`, but not on Linux; strictly speaking, `grep` uses regular expressions as defined by POSIX, and for whatever reasons POSIX does not define `\\t` as tab. One workaround is to use `-P`, which specifies Perl regular expressions; however the `-P` option does not exist in Mac OS X.\n\nHere's how you can verify that the pairs and stripes implementation give you the same results:\n\n```\n$ hadoop fs -cat bigram-freq-mr-pairs/part-r-0000* | awk '{print $1$2,$3;}' | grep -v \",\\*)\" | sort \u003e freq.mr.pairs.txt\n$ hadoop fs -cat bigram-freq-mr-stripes/part-r-0000* | perl -ne '%H=();m/([^\\t]+)\\t\\{(.*)\\}/; $k=$1; @k=split \", \",$2; foreach(@k){@p=split \"=\",$_;$H{$p[0]}=$p[1];}; foreach (sort keys %H) {print \"($k,$_) $H{$_}\\n\";}' | sort \u003e freq.mr.stripes.txt\n$ diff freq.mr.stripes.txt freq.mr.pairs.txt\n```\n\n## Computing Term Co-occurrence Matrix in MapReduce\n\nMake sure you've downloaded the Shakespeare collection (see \"Getting Started\" above). Running the \"pairs\" implementation:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.cooccur.ComputeCooccurrenceMatrixPairs \\\n   -input data/Shakespeare.txt -output cooccur-pairs -window 2\n```\n\nRunning the \"stripes\" implementation:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.cooccur.ComputeCooccurrenceMatrixStripes \\\n   -input data/Shakespeare.txt -output cooccur-stripes -window 2\n```\n\nLet's spot check the results. For example, here are all the terms the co-occur with \"dream\" with the \"pairs\" implementation:\n\n```\n$ hadoop fs -cat cooccur-pairs/part* | grep '(dream, '\n```\n\nWe can verify that the \"stripes\" implementation gives the same results.\n\n```\n$ hadoop fs -cat cooccur-stripes/part* | awk '/^dream\\t/'\n```\n\n**Tip:** Note that `grep` in Mac OS X accepts `\\t`, but not on Linux; strictly speaking, `grep` uses regular expressions as defined by POSIX, and for whatever reasons POSIX does not define `\\t` as tab. One workaround is to use `-P`, which specifies Perl regular expressions; however the `-P` option does not exist in Mac OS X.\n\n## Inverted Indexing and Boolean Retrieval in MapReduce\n\nMake sure you've downloaded the Shakespeare collection (see \"Getting Started\" above). Building the inverted index:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.search.BuildInvertedIndex \\\n   -input data/Shakespeare.txt -output index\n```\n\nLooking up an individual postings list:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.search.LookupPostings \\\n   -index index -collection data/Shakespeare.txt -term \"star-cross'd\"\n```\n\nRunning a boolean retrieval:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.search.BooleanRetrieval \\\n   -index index -collection data/Shakespeare.txt -query \"white red OR rose AND pluck AND\"\n```\n\nNote that the query must be in [Reverse Polish notation](https://en.wikipedia.org/wiki/Reverse_Polish_notation), so the above is equivalent to `(white OR red) AND rose AND pluck` in standard infix notation.\n\n## Parallel Breadth-First Search in MapReduce\n\nMake sure you've grabbed the sample graph data (see \"Getting Started\" above). First, convert the plain-text adjacency list representation into Hadoop `Writable` records:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bfs.EncodeBfsGraph \\\n   -input data/p2p-Gnutella08-adj.txt -output graph-BFS/iter0000 -src 367\n```\n\nIn the current implementation, you have to run a MapReduce job for every iteration, like this:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bfs.IterateBfs \\\n   -input graph-BFS/iter0000 -output graph-BFS/iter0001 -partitions 5\n```\n\nHere's a bash script to run a bunch of iterations:\n\n```\n#!/bin/bash\n\nfor i in `seq 0 14`; do\n  cur=`echo $i | awk '{printf \"%04d\\n\", $0;}'`\n  next=`echo $(($i+1)) | awk '{printf \"%04d\\n\", $0;}'`\n  echo \"Iteration $i: reading graph-BFS/iter$cur, writing: graph-BFS/iter$next\"\n  hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bfs.IterateBfs -input \"graph-BFS/iter$cur\" -output \"graph-BFS/iter$next\" -partitions 5\ndone\n```\n\nThe MapReduce job counters tell you how many nodes are reachable at each iteration:\n\n|Iteration |  Reachable | Distance\n|---------:|-----------:|--------:\n|        0 |          1 |       1\n|        1 |          9 |       8\n|        2 |         65 |      56\n|        3 |        257 |     192\n|        4 |        808 |     551\n|        5 |       1934 |    1126\n|        6 |       3479 |    1545\n|        7 |       4790 |    1311\n|        8 |       5444 |     654\n|        9 |       5797 |     353\n|       10 |       5920 |     123\n|       11 |       5990 |      70\n|       12 |       6018 |      28\n|       13 |       6026 |       8\n|       14 |       6028 |       2\n|       15 |       6028 |       0\n\nTo find all the nodes that are reachable at a particular iteration, run the following job:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bfs.FindReachableNodes \\\n   -input graph-BFS/iter0005 -output graph-BFS/reachable-iter0005\n\n$ hadoop fs -cat 'graph-BFS/reachable-iter0005/part*' | wc\n```\n\nThese values should be the same as those in the second column of the table above.\n\nTo find all the nodes that are at a particular distance (e.g., the search frontier), run the following job:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bfs.FindNodeAtDistance \\\n   -input graph-BFS/iter0005 -output graph-BFS/d0005 -distance 5\n\n$ hadoop fs -cat 'graph-BFS/d0005/part*' | wc\n```\n\nThe results are shown in the third column of the table above.\n\nHere's a simple bash script for iterating through the reachability jobs:\n\n```\n#!/bin/bash\n\nfor i in `seq 0 15`; do\n  cur=`echo $i | awk '{printf \"%04d\\n\", $0;}'`\n  echo \"Iteration $i: reading graph-BFS/iter$cur\"\n  hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bfs.FindReachableNodes -input graph-BFS/iter$cur -output graph-BFS/reachable-iter$cur\ndone\n```\n\nHere's a simple bash script for extracting nodes at each distance:\n\n```\n#!/bin/bash\n\nfor i in `seq 0 15`; do\n  cur=`echo $i | awk '{printf \"%04d\\n\", $0;}'`\n  echo \"Iteration $i: reading graph-BFS/iter$cur\"\n  hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.bfs.FindNodeAtDistance -input graph-BFS/iter$cur -output graph-BFS/d$cur -distance $i\ndone\n```\n\n## PageRank in MapReduce\n\nMake sure you've grabbed the sample graph data (see \"Getting Started\" above). First, convert the plain-text adjacency list representation into Hadoop `Writable` records:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.pagerank.BuildPageRankRecords \\\n   -input data/p2p-Gnutella08-adj.txt -output graph-PageRankRecords -numNodes 6301\n```\n\nCreate the directory where the graph is going to go:\n\n```\n$ hadoop fs -mkdir graph-PageRank\n```\n\nPartition the graph:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.pagerank.PartitionGraph \\\n   -input graph-PageRankRecords -output graph-PageRank/iter0000 -numPartitions 5 -numNodes 6301\n```\n\nRun 15 iterations:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.pagerank.RunPageRankBasic \\\n   -base graph-PageRank -numNodes 6301 -start 0 -end 15 -useCombiner\n```\n\nExtract the top 20 nodes by PageRank value and examine the results:\n\n```\n$ hadoop jar target/bespin-1.0.5-SNAPSHOT-fatjar.jar io.bespin.java.mapreduce.pagerank.FindMaxPageRankNodes \\\n   -input graph-PageRank/iter0015 -output graph-PageRank-top20 -top 20\n\n$ hadoop fs -cat graph-PageRank-top20/part-r-00000\n367     -6.03734\n249     -6.12637\n145     -6.18742\n264     -6.21511\n266     -6.23297\n123     -6.28525\n127     -6.28685\n122     -6.29073\n1317    -6.29597\n5       -6.30274\n251     -6.32983\n427     -6.33821\n149     -6.40216\n176     -6.42350\n353     -6.43988\n390     -6.44404\n559     -6.45491\n124     -6.45705\n4       -6.47055\n7       -6.50145\n```\n\nCompare the results with a sequential PageRank implementation:\n\n```\n$ mvn exec:java -Dexec.mainClass=io.bespin.java.mapreduce.pagerank.SequentialPageRank \\\n   -Dexec.args=\"-input data/p2p-Gnutella08-adj.txt -jump 0.15\"\n```\n\nThe results should be the same.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flintool%2Fbespin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flintool%2Fbespin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flintool%2Fbespin/lists"}