{"id":15208247,"url":"https://github.com/aless19/pyspark-dev","last_synced_at":"2026-01-28T01:06:29.071Z","repository":{"id":248708196,"uuid":"828313765","full_name":"Aless19/pyspark-dev","owner":"Aless19","description":"ezpz pyspark dev environment with docker","archived":false,"fork":false,"pushed_at":"2024-07-20T09:15:52.000Z","size":37,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-14T09:37:41.021Z","etag":null,"topics":["docker","docker-compose","jupyter-lab","pyspark","spark"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aless19.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-13T18:47:40.000Z","updated_at":"2024-07-20T09:15:55.000Z","dependencies_parsed_at":"2024-09-28T07:01:15.570Z","dependency_job_id":null,"html_url":"https://github.com/Aless19/pyspark-dev","commit_stats":{"total_commits":3,"total_committers":2,"mean_commits":1.5,"dds":"0.33333333333333337","last_synced_commit":"add3328bb30025a8307ecd0dca832fa877294553"},"previous_names":["aless19/pyspark-dev"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aless19%2Fpyspark-dev","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aless19%2Fpyspark-dev/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aless19%2Fpyspark-dev/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aless19%2Fpyspark-dev/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aless19","download_url":"https://codeload.github.com/Aless19/pyspark-dev/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243021773,"owners_count":20223068,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","jupyter-lab","pyspark","spark"],"created_at":"2024-09-28T07:01:10.234Z","updated_at":"2026-01-28T01:06:29.029Z","avatar_url":"https://github.com/Aless19.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PySpark Dev environment\n\n## Español\n\nEntorno para desarrollo con PySpark usando docker (imagen de [bitcami](https://github.com/bitnami/containers?tab=License-1-ov-file)).\n\n### Requisitos\n- Docker\n- Docker compose\n\n### Uso\n\nPara poner a funcionar el entorno simplemente es necesario iniciarlo con:\n```bash\ndocker-compose up -d\n```\n\nUna vez hemos iniciado los contenedores es necesario esperar unos segundos ya que tarda unos instantes en instalar e iniciar el servidor de jupyter.\n\nPor defecto el servidor estará funcionando en el `localhost` pero en caso de querer usar la ip podemos obtenerla con el siguiente comando:\n```bash\ndocker inspect  -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-spark-master-1\n```\n- Web\n![Captura de pantalla del navegador con Jupyter-Lab](./img/captura-web.png)\n\n\n- VSCode   \n![Captura de pantalla del vscode con Jupyter-Lab](./img/captura-vscode.png)\n\nPara apagarlo:\n```bash\ndocker-compose down\n```\n\n### Explicación\n\nEl entorno por defecto consta de 2 `workers` y 1 `master`.\n\nDesde el archivo [docker-compose](./docker/docker-compose.yml) el `master` ejecuta el script [docker-start](./docker/docker-start.sh), este lo que hace es:\n1. Inicia spark\n2. Instala las bibliotecas de `pyspark` y `notebook`.\n3. Inicia el servidor de Jupyter.\n\n**Importante**   \nEl servidor de Jupyter se inicia de manera que no requiere de contraseña y admite conexiones desde cualquier lado, no es recomendable mantener esta configuración en la mayoría de los casos.\n\n```bash\njupyter-lab --ip=0.0.0.0 --port 8888 --no-browser --notebook-dir=/proyecto --NotebookApp.allow_origin=\"*\" --allow-root --ServerApp.password='' --ServerApp.token='' --ServerApp.password_required=False\n```\n\n### Errores\n\nEn caso de usar linux es posible que el contenedor `master` devuelva un error, en ese caso es posible que sea necesario hacer lo siguiente antes de iniciar el entorno:\n```shell\nchmod +x docker-start.sh\n```\n\n\n## English\n\nEnvironment for PySpark development using docker ([bitcami](https://github.com/bitnami/containers?tab=License-1-ov-file) image).\n\n### Requisites\n- Docker\n- Docker compose\n\n### Use\n\nTo get the environment working, you just need to start it with:\n```bash\ndocker-compose up -d\n```\n\nOnce we have started the containers, we need to wait a few seconds for the jupyter server to install and start.\n\nBy default the server will run on `localhost` but if you want to use the ip it's possible to get it with:\n```bash\ndocker inspect  -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-spark-master-1\n```\n- Web\n![Navigator capture with Jupyter-Lab](./img/captura-web.png)\n\n\n- VSCode   \n![Vscode capture with Jupyter-Lab](./img/captura-vscode.png)\n\nTo shutdown:\n```bash\ndocker-compose down\n```\n\n### Explanation\n\nBy default, the environment has 2 `workers' and 1 `master'.\n\nFrom the [docker-compose](./docker/docker-compose.yml) file the `master` runs the [docker-start](./docker/docker-start.sh) script, which that script does the following:\n\n1. Start Spark\n2. Install the `pyspark` and `notebook` libraries.\n3. Start the Jupyter server.\n\n**Important**   \nThe Jupyter server is started in a way that doesn't require a password and allows connections from anywhere, which is not recommended for most uses.\n\n```bash\njupyter-lab --ip=0.0.0.0 --port 8888 --no-browser --notebook-dir=/proyecto --NotebookApp.allow_origin=\"*\" --allow-root --ServerApp.password='' --ServerApp.token='' --ServerApp.password_required=False\n```\n\n### Errors\n\nIn case of using linux it is possible that the `master` container returns an error, in that case it may be necessary to do the following before starting the environment:\n```shell\nchmod +x docker-start.sh\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faless19%2Fpyspark-dev","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faless19%2Fpyspark-dev","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faless19%2Fpyspark-dev/lists"}