{"id":13588863,"url":"https://github.com/fabiogjardim/bigdata_docker","last_synced_at":"2025-04-04T23:08:54.750Z","repository":{"id":37620365,"uuid":"248647323","full_name":"fabiogjardim/bigdata_docker","owner":"fabiogjardim","description":"Big Data Ecosystem Docker","archived":false,"fork":false,"pushed_at":"2023-04-29T17:50:45.000Z","size":132242,"stargazers_count":407,"open_issues_count":2,"forks_count":319,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-03-28T22:12:28.662Z","etag":null,"topics":["hadoop","hbase","hdfs","hive","hue","jupyter-notebook","metabase","mongo","mysql","nifi","presto","spark","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"VBA","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fabiogjardim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-20T02:07:24.000Z","updated_at":"2025-03-23T16:26:54.000Z","dependencies_parsed_at":"2024-09-02T15:51:47.911Z","dependency_job_id":null,"html_url":"https://github.com/fabiogjardim/bigdata_docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiogjardim%2Fbigdata_docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiogjardim%2Fbigdata_docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiogjardim%2Fbigdata_docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiogjardim%2Fbigdata_docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fabiogjardim","download_url":"https://codeload.github.com/fabiogjardim/bigdata_docker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247261604,"owners_count":20910108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","hbase","hdfs","hive","hue","jupyter-notebook","metabase","mongo","mysql","nifi","presto","spark","zookeeper"],"created_at":"2024-08-01T15:06:59.700Z","updated_at":"2025-04-04T23:08:54.730Z","avatar_url":"https://github.com/fabiogjardim.png","language":"VBA","funding_links":[],"categories":["VBA"],"sub_categories":[],"readme":"\n# BIG DATA ECOSYSTEM COM DOCKER\n\nAmbiente para estudo dos principais frameworks big data em docker.\n\u003cbr\u003e Esse setup vai criar dockers com os frameworks HDFS, HBase, Hive, Presto, Spark, Jupyter, Hue, Mongodb, Metabase, Nifi, kafka, Mysql e Zookeeper com a seguinte arquitetura:\n\u003cbr\u003e  \n\n![Ecossistema](ecosystem.jpeg)\n\n## SOFTWARES NECESSÁRIOS\n#### Para a criação e uso do ambiente vamos utilizar o git e o Docker \n   * Instalação do Docker Desktop no Windows [Docker Desktop](https://hub.docker.com/editions/community/docker-ce-desktop-windows) ou o docker no [Linux](https://docs.docker.com/install/linux/docker-ce/ubuntu/)\n   *  [Instalação do git](https://git-scm.com/book/pt-br/v2/Come%C3%A7ando-Instalando-o-Git)\n   \n## SETUP\n*OBS: Esse passo deve ser realizado apena uma vez. Após o ambiente criado, utilizar o docker-compose para iniciar os containers como mostrado no tópico INICIANDO O AMBIENTE*\n\n#### Criação do diretório docker:\n*OBS: Criar um diretório chamado docker*\n\n   *  Sugestão no Windows:\n      *  Criar na raiz do seu drive o diretório docker\n         ex: C:\\docker\n          \n   * Sugestão no Linux:\n      * Criar o diretório na home do usuário\n        ex: /home/user/docker\n\n#### Em um terminal/DOS, dentro diretório docker, realizar o clone do projeto no github\n          git clone https://github.com/fabiogjardim/bigdata_docker.git\n\n#### No diretório bigdata_docker vai existir os seguintes objetos\n![ls](ls.JPG)\n\n   \n## INICIANDO O AMBIENTE\n   \n  *No Windows abrir PowerShell, do Linux um terminal*\n\n### No terminal, no diretorio bigdata_docker, executar o docker-compose\n          docker-compose up -d        \n\n### Verificar imagens e containers\n \n         docker image ls\n\n![docker image ls](docker_image_ls.JPG)\n\n         docker container ls\n\n![docker container](docker_container_ls.JPG)\n\n## SOLUCIONANDO PROBLEMAS \n   \n  *No Windows abrir o Docker Quickstart Terminal*\n\n### Parar um containers\n         docker stop [nome do container]      \n\n### Parar todos containers\n         docker stop $(docker ps -a -q)\n  \n### Remover um container\n         docker rm [nome do container]\n\n### Remover todos containers\n         docker rm $(docker ps -a -q)         \n\n### Dados do containers\n         docker container inspect [nome do container]\n\n### Iniciar um container\n         docker-compose up -d [nome do container]\n\n### Iniciar todos os containers\n         docker-compose up -d \n\n### Acessar log do container\n         docker container logs [nome do container] \n\n## Acesso WebUI dos Frameworks\n \n* HDFS *http://localhost:50070*\n* Presto *http://localhost:8080*\n* Hbase *http://localhost:16010/master-status*\n* Mongo Express *http://localhost:8081*\n* Kafka Manager *http://localhost:9000*\n* Metabase *http://localhost:3000*\n* Nifi *http://localhost:9090*\n* Jupyter Spark *http://localhost:8889*\n* Hue *http://localhost:8888*\n* Spark *http://localhost:4040*\n\n## Acesso por shell\n\n   ##### HDFS\n\n          docker exec -it datanode bash\n\n   ##### HBase\n\n          docker exec -it hbase-master bash\n\n   ##### Sqoop\n\n          docker exec -it datanode bash\n        \n   ##### Kafka\n\n          docker exec -it kafka bash\n\n## Acesso JDBC\n\n   ##### MySQL\n          jdbc:mysql://database/employees\n\n   ##### Hive\n\n          jdbc:hive2://hive-server:10000/default\n\n   ##### Presto\n\n          jdbc:presto://presto:8080/hive/default\n\n## Usuários e senhas\n\n   ##### Hue\n    Usuário: admin\n    Senha: admin\n\n   ##### Metabase\n    Usuário: bigdata@class.com\n    Senha: bigdata123 \n\n   ##### MySQL\n    Usuário: root\n    Senha: secret\n   \n   ##### MongoDB\n    Usuário: root\n    Senha: root\n    Authentication Database: admin\n\n## Imagens   \n\n[Docker Hub](https://hub.docker.com/u/fjardim)\n\n## Documentação Oficial\n\n* https://zookeeper.apache.org/\n* https://kafka.apache.org/\n* https://nifi.apache.org/\n* https://prestodb.io/\n* https://spark.apache.org/\n* https://www.mongodb.com/\n* https://www.metabase.com/\n* https://jupyter.org/\n* https://hbase.apache.org/\n* https://sqoop.apache.org/\n* https://hadoop.apache.org/\n* https://hive.apache.org/\n* https://gethue.com/\n* https://github.com/yahoo/CMAK\n* https://www.docker.com/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabiogjardim%2Fbigdata_docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabiogjardim%2Fbigdata_docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabiogjardim%2Fbigdata_docker/lists"}