{"id":15693332,"url":"https://github.com/joshdevins/talks-hadoop-getting-started","last_synced_at":"2026-03-07T17:04:22.886Z","repository":{"id":1415270,"uuid":"1500300","full_name":"joshdevins/talks-hadoop-getting-started","owner":"joshdevins","description":"Talk on 'Getting Started with Hadoop'","archived":false,"fork":false,"pushed_at":"2011-06-22T20:30:07.000Z","size":25414,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-10T16:35:39.936Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"cogitatio/vagrant-hostsupdater","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joshdevins.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-03-19T15:52:49.000Z","updated_at":"2023-09-15T13:17:44.000Z","dependencies_parsed_at":"2022-07-07T11:40:50.281Z","dependency_job_id":null,"html_url":"https://github.com/joshdevins/talks-hadoop-getting-started","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/joshdevins/talks-hadoop-getting-started","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Ftalks-hadoop-getting-started","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Ftalks-hadoop-getting-started/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Ftalks-hadoop-getting-started/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Ftalks-hadoop-getting-started/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joshdevins","download_url":"https://codeload.github.com/joshdevins/talks-hadoop-getting-started/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshdevins%2Ftalks-hadoop-getting-started/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30222465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T17:00:40.062Z","status":"ssl_error","status_checked_at":"2026-03-07T17:00:39.026Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T18:42:57.493Z","updated_at":"2026-03-07T17:04:22.851Z","avatar_url":"https://github.com/joshdevins.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"Talk on 'Getting Started with Hadoop'\n===\n\nThis repo tracks all source code, data samples and visualization examples for my talks on 'Getting Started with Hadoop'.\n\nData and Visualizations\n---\n\nData can be found in the `data` directory including raw logs courtesy of [@maxheadroom](http://www.twitter.com/maxheadroom) from [mac-geeks.de](http://www.mac-geeks.de). The final results of the Apache access log throughput script are also loaded up into a [Google Docs spreadsheet](https://spreadsheets.google.com/ccc?key=0AqAx7w3Zbl99dHFwX2l0TU4wQ2tWYWNZTG1DOENOTVE\u0026hl=en) for ad-hoc analysis and visualization. The `src/html` directory contains a simple page with example visualizations using [Google Chart API's](http://code.google.com/apis/chart) [time series chart](http://code.google.com/apis/visualization/documentation/gallery/annotatedtimeline.html).\n\nDependencies\n---\n\nAll dependencies are managed currently in the Maven `pom`. There are a few, however, that need to be added to your repository (local or otherwise) before continuing. They are all documented inline in the `dependencies` of the Maven `pom`. We assume that you have installed Pig and have the `$PIG_HOME` environment variable defined (it's referenced in the `pom` comments on installing local dependencies).\n\nTesting\n---\n\nUnit testing of Pig is currently done using PigUnit, a new xUnit style testing harness released in Pig 0.8.0. Since this is it's debut there are a few quirks to be aware of:\n\n * heap space is at a premium when running the Pig scripts, so max heap space JVM parameters need to be set (to something like`-Xmx1024m`)\n   * running tests from Maven: configured in the Maven `pom` to run the Surefire plugin with a fixed max heap space setting\n   * running tests from Eclipse: use the [JUnit Lanch Fixer](http://code.google.com/p/junitlaunchfixer) plugin and set the max heap space for all JUnit executions automatically (note that if you have previous failed launches, you should delete them before running again)\n\nRunning\n---\n\nCheckout the source and build it with the Maven assembly plugin.\n\n    mvn assembly:assembly\n\nThe demo is done using the Cloudera [training VM v0.3.5](http://cloudera-vm.s3.amazonaws.com/cloudera-demo-0.3.5.tar.bz2?downloads) with CDH3b3. Here are the steps to run the demo. This assumes that you move/copy/mount the Git directory/checkout onto the VM.\n\n    hadoop fs -rmr access-log-throughput-mr; hadoop jar target/hadoop-getting-started-1-SNAPSHOT-jar-with-dependencies.jar net.joshdevins.talks.hadoopstart.mr.AccessLogThroughputDriver\n    hadoop fs -cat access-log-throughput-mr/part-* | more\n\n    hadoop fs -rmr access-log-throughput; java -cp target/hadoop-getting-started-1-SNAPSHOT-jar-with-dependencies.jar:/etc/hadoop/conf org.apache.pig.Main -logfile target/pig.log src/main/pig/access-log-throughput.pig\n    hadoop fs -cat access-log-throughput/part-* | more\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshdevins%2Ftalks-hadoop-getting-started","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoshdevins%2Ftalks-hadoop-getting-started","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshdevins%2Ftalks-hadoop-getting-started/lists"}