{"id":25711653,"url":"https://github.com/bernta/norwegiandemographics","last_synced_at":"2026-05-02T20:41:20.741Z","repository":{"id":97795635,"uuid":"238175923","full_name":"BerntA/NorwegianDemographics","owner":"BerntA","description":"Norwegian Demographics using data from 1900-1910, analyzed with Hadoop \u0026 Spark.","archived":false,"fork":false,"pushed_at":"2020-06-17T12:05:17.000Z","size":117353,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-16T08:18:27.061Z","etag":null,"topics":["ancestry","census","data-science","data-scraping","datascience","demography","hadoop","norway","python","spark"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BerntA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-02-04T10:08:16.000Z","updated_at":"2021-05-28T13:21:28.000Z","dependencies_parsed_at":"2023-10-03T10:28:32.603Z","dependency_job_id":null,"html_url":"https://github.com/BerntA/NorwegianDemographics","commit_stats":{"total_commits":92,"total_committers":2,"mean_commits":46.0,"dds":"0.010869565217391353","last_synced_commit":"95bb54c52d4da4cc22679ac4bf20196075af9da2"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BerntA/NorwegianDemographics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerntA%2FNorwegianDemographics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerntA%2FNorwegianDemographics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerntA%2FNorwegianDemographics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerntA%2FNorwegianDemographics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BerntA","download_url":"https://codeload.github.com/BerntA/NorwegianDemographics/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerntA%2FNorwegianDemographics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32549387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T19:18:06.202Z","status":"ssl_error","status_checked_at":"2026-05-02T19:16:21.335Z","response_time":132,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ancestry","census","data-science","data-scraping","datascience","demography","hadoop","norway","python","spark"],"created_at":"2025-02-25T10:55:02.591Z","updated_at":"2026-05-02T20:41:15.724Z","avatar_url":"https://github.com/BerntA.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Norwegian Census Data Analysis using Hadoop\nDAT500 MSc course\n\n## Configuring the cluster (tested on Ubuntu 16.04 and 18.04)\ncd into setup\n\n### Shared\n* Ensure that setup_env.sh has correct addresses for /etc/hosts/.\n* Run sudo setup.sh, this will setup the environment, install necessary libraries and Hadoop.\n* Copy files \u0026 folders in /scripts/* to /usr/local/hadoop/\n* Ensure that Python 3 is the default Python interpreter! And that it is a version prior to 3.8.0.\n* Ensure that Java 8 is the default Java version. Check ~/.bashrc and ~/.profile for potential Java 11 overrides.\n\n### Master Node Only\n* Run sudo setup_spark.sh, this will install Spark.\n* Copy files \u0026 folders in /master-only/ to /usr/local/hadoop/\n* Copy files \u0026 folders in /spark/ to /usr/local/spark/\n* Update /usr/local/hadoop/etc/hadoop/workers to include your workers. (slave nodes)\n\n### Slave Node Only\n* Copy files \u0026 folders in /slave-only/ to /usr/local/hadoop/\n\n### Finally\nFormat the namenode like this, hdfs namenode -format (type in terminal)\n\n## Running jobs @ master\n\n### Starting/Stopping\n* Start Hadoop and Spark, run start-dfs.sh, start-yarn.sh, and start-history-server.sh.\n* Stop Hadoop and Spark, run stop-history-server.sh, stop-yarn.sh and stop-dfs.sh.\n\n### Testing\nTo test if Hadoop is working, simply run setup/setup_runtest.sh.\n\n### When Hadoop and Spark is up\n* Regular MapReduce, /run/hadoop-streaming.sh \u003cfile_mapper.py\u003e \u003cfile_reducer.py\u003e \u003cinput-from-hdfs\u003e \u003coutput-to-hdfs\u003e\n* With MRJob, python3 some_file.py --hadoop-streaming-jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-3.1.1.jar -r hadoop hdfs:///data/input_data.csv --output-dir hdfs:///output/xyz --no-output\n* With Spark, spark-submit --master yarn some_file.py\n\n### To generate the results for this project (assuming you have access to our dataset)\ncd into src and execute run.sh (~/dat500/src/)\n\n## Visualizing the results locally\n* Sync results retrieved from Hadoop \u0026 Spark\n* cd into src\n* conda install geopandas\n* conda install geoplot -c conda-forge\n* conda install -c conda-forge cartopy\n* pip install -r requirements.txt\n* Open visualize.ipynb to visualize results\n\n(the CSV dataset was generated from the original census dataset by running src/merge_data.ipynb)\n\n## Troubleshooting\n* Check if /usr/local/hadoop/etc/hadoop/hadoop-env.sh has any faulty paths.\n* Python 3.6.9 was built from source, and should be the default Python version. However if a different version is preferred, update /usr/local/spark/conf/spark-env.sh to use python3.X for its Python drivers.\n\n## Reference(s) / Guide(s)\n* Check https://codethief.io/hadoop101/ for similar cluster configuration / setup.\n\n# Example Results\n\n![](src/img/1800_1910_anim_croppd_transparent.gif)\n\n![](src/img/1910_religion_distribution_transp.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbernta%2Fnorwegiandemographics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbernta%2Fnorwegiandemographics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbernta%2Fnorwegiandemographics/lists"}