https://github.com/swanandp/mapreduce_with_hadoop

Supplementary code I used in the "Build your first MapReduce with Hadoop and Ruby" at the BRUG March meet-up.
https://github.com/swanandp/mapreduce_with_hadoop

Last synced: 3 months ago
JSON representation

Supplementary code I used in the "Build your first MapReduce with Hadoop and Ruby" at the BRUG March meet-up.

• Host: GitHub
• URL: https://github.com/swanandp/mapreduce_with_hadoop
• Owner: swanandp
• Created: 2013-03-18T05:20:25.000Z (almost 12 years ago)
• Default Branch: master
• Last Pushed: 2013-03-18T05:56:07.000Z (almost 12 years ago)
• Last Synced: 2024-04-24T03:00:49.454Z (9 months ago)
• Language: Ruby
• Homepage: https://speakerdeck.com/swanandp/build-your-first-mapreduce-with-hadoop-and-ruby
• Size: 598 KB
• Stars: 9
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
#### Start Hadoop

    start-all.sh

#### Formatting HDFS for the task

    hadoop namenode -format

#### Check if everything went well

    # should list /tmp

    hadoop fs -ls / 

#### Sanity check:

    http://localhost:50070/ # NameNode  

    http://localhost:50030/ # JobTracker  

#### Put Data in HDFS

    # create directories

    hadoop fs -mkdir pg

    hadoop fs -mkdir pg_output

    # copy files

    hadoop fs -put input/pg11.txt pg/

    hadoop fs -put input/pg174.txt pg/

    hadoop fs -put input/pg844.txt pg/

    hadoop fs -put input/pg1661.txt pg/

    # list all

    hadoop fs -ls pg

#### Run tasks

    cd code/alphabeta/

    sh stream.sh