{"id":18675347,"url":"https://github.com/manuparra/masterdatcom_bdcc_practice","last_synced_at":"2025-04-12T02:11:13.442Z","repository":{"id":79159325,"uuid":"84338944","full_name":"manuparra/MasterDatCom_BDCC_Practice","owner":"manuparra","description":"Practice and Workshop on BigData and Cloud Computing using Docker Containers and OpenNebula. HDFS, hadoop and spark+R","archived":false,"fork":false,"pushed_at":"2017-03-16T17:55:37.000Z","size":45,"stargazers_count":11,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T21:51:12.264Z","etag":null,"topics":["bigdata","cloudcomputing","containers","docker","hadoop","hdfs","linux","opennebula","practices","spark","sparkr"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manuparra.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-08T15:57:54.000Z","updated_at":"2021-07-21T16:48:10.000Z","dependencies_parsed_at":"2023-02-28T01:01:06.530Z","dependency_job_id":null,"html_url":"https://github.com/manuparra/MasterDatCom_BDCC_Practice","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuparra%2FMasterDatCom_BDCC_Practice","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuparra%2FMasterDatCom_BDCC_Practice/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuparra%2FMasterDatCom_BDCC_Practice/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manuparra%2FMasterDatCom_BDCC_Practice/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manuparra","download_url":"https://codeload.github.com/manuparra/MasterDatCom_BDCC_Practice/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248505928,"owners_count":21115354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","cloudcomputing","containers","docker","hadoop","hdfs","linux","opennebula","practices","spark","sparkr"],"created_at":"2024-11-07T09:24:34.721Z","updated_at":"2025-04-12T02:11:13.434Z","avatar_url":"https://github.com/manuparra.png","language":null,"readme":"# Máster en Ciencia de Datos e Ingeniería de Computadores. Prácticas de BigData y Cloud Computing. Curso 2016-2017. \n\n![Header](https://sites.google.com/site/manuparra/home/headerdicits.png)\n\nManuel J. Parra Royón (manuelparra@decsai.ugr.es) \u0026  José. M. Benítez Sánchez (j.m.benitez@decsai.ugr.es)\n\n[UGR](http://www.ugr.es) | [DICITS](http://dicits.ugr.es) | [SCI2S](http://sci2s.ugr.es) | [DECSAI](http://decsai.ugr.es)\n\n\nTable of Contents\n=================\n\n\n   * [1. Environment of the practice](#environment-of-the-practice)\n      * [Connecting to Docker Server UGR](#connecting-to-docker-server-ugr)\n      * [Connecting to Docker Containers](#connecting-to-docker-containers)\n      * [Infrastructure](#infrastructure)\n      * [Docker Cluster port assignment for VIRTUAL MACHINES](#on-docker-cluster-this-is-the-account-port-assignment)\n      * [Hadoop Cluster port assignment for DOCKER CONTAINERS](#on-hadoop-cluster-this-is-the-account-port-assignment-for-docker-containers)\n\n   * [2. Starting with OPENNEBULA](#starting-with-opennebula)\n   * [3. Starting with DOCKER](#starting-with-docker)\n   * [4. Starting with HDFS](#starting-with-hdfs)\n   * [5. Starting with HADOOP](#starting-with-hadoop)\n   * [6. Starting with SPARKR](#starting-with-sparkr)\n\n\n\n# Environment of the practice\n\nThe working environment consists of the following structure for each user:\n\n- Two Clusters:\n   - docker cluster  (15 nodes)\n   - hadoop cluster  (15 nodes)\n\n- docker cluster provides for each user:\n   - Max 1 Virtual machines (and 1 IPs)\n   - Max 1 ports redirected from external to internal Virtual Machine network\n\n- hadoop cluster provides for each user:\n   - Max 2 Docker container working at the same time\n   - Max 2 ports redirected from external to internal Docker Containers\n\n\n![ClusterImage](https://sites.google.com/site/manuparra/home/clusterhadoop.jpg)\n\n## Connecting to Docker Cluster UGR\n\nLog in docker ugr server with your credentials\n\n```\nssh manuparra@docker...\n```\n\nFor this course use the next login credential:\n\n- If your CardID number is spanish and similar to 77464888L,  your login will be: ``ssh mdat77464888@docker....``\n- If your CardID number is not spanish or you are using your passport similar to YM5664883L, your login will be: ``ssh YM5664883@docker......``\n\nThe  general rule is:\n- Remove your last letter of your CardID number\n\n\nAfter login, **PLEASE CHANGE** your password using command:\n\n```\npasswd\n```\n\n**Try to use a strong password mixing: upper, lower, and punctiation**\n\n\n\n## Connecting to Hadoop Cluster UGR\n\nLog in hadoop ugr server with your credentials (the same way):\n\n```\nssh manuparra@hadoop...\n```\n\nFor this course use the next login credential:\n\n- If your CardID number is spanish and similar to 77464888L,  your login will be: ``ssh mdat77464888@hadoop....``\n- If your CardID number is not spanish or you are using your passport similar to YM5664883L, your login will be: ``ssh YM5664883@hadoop......``\n\nThe  general rule is:\n- Remove your last letter of your CardID number\n\n\nAfter login, **PLEASE CHANGE** your password using command:\n\n```\npasswd\n```\n\n\n**Try to use a strong password mixing: upper, lower, and punctiation**\n\n\n## Infrastructure\n\nWhat is the infrastructure of the practice?\n\nThe complete infraestructure for the practice is the next:\n\n![CompleteStruct](https://sites.google.com/site/manuparra/home/ArchitectureBDCC.png)\n\nAs you see, inside docker you will work with VIRTUAL MACHINES and inside hadoopyou will work with CONTAINERS.\n\n*IMPORTANT*: Ports assignment\n\n### On Docker Cluster this is the account port assignment\n\n*Account ending in ....XXX*\n\n### On Hadoop Cluster this is the account port assignment for DOCKER CONTAINERS\n\n### dockerports\n\n*Account ending in ....XXX*\n\n```\n--\u003e DOCKER PORTS redirected. User mdat......638 \nPort: 16000 \nPort: 16001\n--\u003e DOCKER PORTS redirected. User mdat......968 \nPort: 16002 \nPort: 16003\n--\u003e DOCKER PORTS redirected. User mdat......179 \nPort: 16004 \nPort: 16005\n--\u003e DOCKER PORTS redirected. User mdat......515 \nPort: 16006 \nPort: 16007\n--\u003e DOCKER PORTS redirected. User mdat......940 \nPort: 16008 \nPort: 16009\n--\u003e DOCKER PORTS redirected. User mdat......981 \nPort: 16010 \nPort: 16011\n--\u003e DOCKER PORTS redirected. User mdat......685 \nPort: 16012 \nPort: 16013\n--\u003e DOCKER PORTS redirected. User mdat......356 \nPort: 16014 \nPort: 16015\n--\u003e DOCKER PORTS redirected. User mdat......955 \nPort: 16016 \nPort: 16017\n--\u003e DOCKER PORTS redirected. User mdat......478 \nPort: 16018 \nPort: 16019\n--\u003e DOCKER PORTS redirected. User mdat......592 \nPort: 16020 \nPort: 16021\n--\u003e DOCKER PORTS redirected. User mdat......415 \nPort: 16022 \nPort: 16023\n--\u003e DOCKER PORTS redirected. User mdat......285 \nPort: 16024 \nPort: 16025\n--\u003e DOCKER PORTS redirected. User mdat......607 \nPort: 16026 \nPort: 16027\n--\u003e DOCKER PORTS redirected. User mdat......505 \nPort: 16028 \nPort: 16029\n--\u003e DOCKER PORTS redirected. User mdat......474 \nPort: 16030 \nPort: 16031\n--\u003e DOCKER PORTS redirected. User mdat......124 \nPort: 16032 \nPort: 16033\n--\u003e DOCKER PORTS redirected. User mdat......356 \nPort: 16034 \nPort: 16035\n--\u003e DOCKER PORTS redirected. User mdat......738 \nPort: 16036 \nPort: 16037\n--\u003e DOCKER PORTS redirected. User mdat......107 \nPort: 16038 \nPort: 16039\n--\u003e DOCKER PORTS redirected. User mdat......924 \nPort: 16040 \nPort: 16041\n--\u003e DOCKER PORTS redirected. User mdat......914 \nPort: 16042 \nPort: 16043\n--\u003e DOCKER PORTS redirected. User mdat......509 \nPort: 16044 \nPort: 16045\n--\u003e DOCKER PORTS redirected. User mdat......923 \nPort: 16046 \nPort: 16047\n--\u003e DOCKER PORTS redirected. User mdat......129 \nPort: 16048 \nPort: 16049\n--\u003e DOCKER PORTS redirected. User mdat......207 \nPort: 16050 \nPort: 16051\n--\u003e DOCKER PORTS redirected. User mdat......844 \nPort: 16052 \nPort: 16053\n--\u003e DOCKER PORTS redirected. User mdat......612 \nPort: 16054 \nPort: 16055\n--\u003e DOCKER PORTS redirected. User mdat......479 \nPort: 16056 \nPort: 16057\n--\u003e DOCKER PORTS redirected. User mdat......609 \nPort: 16058 \nPort: 16059\n--\u003e DOCKER PORTS redirected. User mdat......403 \nPort: 16060 \nPort: 16061\n--\u003e DOCKER PORTS redirected. User mdat......929 \nPort: 16062 \nPort: 16063\n--\u003e DOCKER PORTS redirected. User mdat......546 \nPort: 16064 \nPort: 16065\n--\u003e DOCKER PORTS redirected. User mdat......027 \nPort: 16066 \nPort: 16067\n--\u003e DOCKER PORTS redirected. User mdat......473 \nPort: 16068 \nPort: 16069\n--\u003e DOCKER PORTS redirected. User mdat......161 \nPort: 16070 \nPort: 16071\n--\u003e DOCKER PORTS redirected. User mdat......522 \nPort: 16072 \nPort: 16073\n--\u003e DOCKER PORTS redirected. User mdat......629 \nPort: 16074 \nPort: 16075\n--\u003e DOCKER PORTS redirected. User mdat......209 \nPort: 16076 \nPort: 16077\n--\u003e DOCKER PORTS redirected. User mdat......107 \nPort: 16078 \nPort: 16079\n--\u003e DOCKER PORTS redirected. User mdat......824 \nPort: 16080 \nPort: 16081\n--\u003e DOCKER PORTS redirected. User mdat......974 \nPort: 16082 \nPort: 16083\n--\u003e DOCKER PORTS redirected. User mdat......625 \nPort: 16084 \nPort: 16085\n--\u003e DOCKER PORTS redirected. User mdat......982 \nPort: 16086 \nPort: 16087\n--\u003e DOCKER PORTS redirected. User mdat......510 \nPort: 16088 \nPort: 16089\n--\u003e DOCKER PORTS redirected. User mdat......836 \nPort: 16090 \nPort: 16091\n--\u003e DOCKER PORTS redirected. User mdat......564 \nPort: 16092 \nPort: 16093\n--\u003e DOCKER PORTS redirected. User mdat......369 \nPort: 16094 \nPort: 16095\n--\u003e DOCKER PORTS redirected. User mdat......443 \nPort: 16096 \nPort: 16097\n--\u003e DOCKER PORTS redirected. User mdat......057 \nPort: 16098 \nPort: 16099\n--\u003e DOCKER PORTS redirected. User mdat......166 \nPort: 16100 \nPort: 16101\n--\u003e DOCKER PORTS redirected. User mdat......202 \nPort: 16102 \nPort: 16103\n--\u003e DOCKER PORTS redirected. User mdat......044 \nPort: 16104 \nPort: 16105\n--\u003e DOCKER PORTS redirected. User mdat......450 \nPort: 16106 \nPort: 16107\n--\u003e DOCKER PORTS redirected. User mdat......569 \nPort: 16108 \nPort: 16109\n```\n\n# Starting with DOCKER\n\nGo to: [First steps with Docker Containers](./starting_docker.md)\n\n# Starting with OPENNEBULA\n\nGo to: [First steps with OpenNebula](./starting_opennebula.md)\n\n# Starting with HDFS\n\nGo to: [First steps with HDFS](./starting_hdfs.md)\n\n# Starting with Hadoop\n\nGo to: [First steps with Hadoop](./starting_hadoop.md)\n\n# Starting with SparkR (spanish)\n\nGo to: [First steps with SparkR](./starting_sparkR.md)\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanuparra%2Fmasterdatcom_bdcc_practice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanuparra%2Fmasterdatcom_bdcc_practice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanuparra%2Fmasterdatcom_bdcc_practice/lists"}