https://github.com/aless19/pyspark-dev
ezpz pyspark dev environment with docker
https://github.com/aless19/pyspark-dev
docker docker-compose jupyter-lab pyspark spark
Last synced: 5 months ago
JSON representation
ezpz pyspark dev environment with docker
- Host: GitHub
- URL: https://github.com/aless19/pyspark-dev
- Owner: Aless19
- Created: 2024-07-13T18:47:40.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-07-20T09:15:52.000Z (almost 2 years ago)
- Last Synced: 2025-02-14T09:37:41.021Z (over 1 year ago)
- Topics: docker, docker-compose, jupyter-lab, pyspark, spark
- Language: Shell
- Homepage:
- Size: 36.1 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# PySpark Dev environment
## Español
Entorno para desarrollo con PySpark usando docker (imagen de [bitcami](https://github.com/bitnami/containers?tab=License-1-ov-file)).
### Requisitos
- Docker
- Docker compose
### Uso
Para poner a funcionar el entorno simplemente es necesario iniciarlo con:
```bash
docker-compose up -d
```
Una vez hemos iniciado los contenedores es necesario esperar unos segundos ya que tarda unos instantes en instalar e iniciar el servidor de jupyter.
Por defecto el servidor estará funcionando en el `localhost` pero en caso de querer usar la ip podemos obtenerla con el siguiente comando:
```bash
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-spark-master-1
```
- Web
- VSCode
Para apagarlo:
```bash
docker-compose down
```
### Explicación
El entorno por defecto consta de 2 `workers` y 1 `master`.
Desde el archivo [docker-compose](./docker/docker-compose.yml) el `master` ejecuta el script [docker-start](./docker/docker-start.sh), este lo que hace es:
1. Inicia spark
2. Instala las bibliotecas de `pyspark` y `notebook`.
3. Inicia el servidor de Jupyter.
**Importante**
El servidor de Jupyter se inicia de manera que no requiere de contraseña y admite conexiones desde cualquier lado, no es recomendable mantener esta configuración en la mayoría de los casos.
```bash
jupyter-lab --ip=0.0.0.0 --port 8888 --no-browser --notebook-dir=/proyecto --NotebookApp.allow_origin="*" --allow-root --ServerApp.password='' --ServerApp.token='' --ServerApp.password_required=False
```
### Errores
En caso de usar linux es posible que el contenedor `master` devuelva un error, en ese caso es posible que sea necesario hacer lo siguiente antes de iniciar el entorno:
```shell
chmod +x docker-start.sh
```
## English
Environment for PySpark development using docker ([bitcami](https://github.com/bitnami/containers?tab=License-1-ov-file) image).
### Requisites
- Docker
- Docker compose
### Use
To get the environment working, you just need to start it with:
```bash
docker-compose up -d
```
Once we have started the containers, we need to wait a few seconds for the jupyter server to install and start.
By default the server will run on `localhost` but if you want to use the ip it's possible to get it with:
```bash
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-spark-master-1
```
- Web
- VSCode
To shutdown:
```bash
docker-compose down
```
### Explanation
By default, the environment has 2 `workers' and 1 `master'.
From the [docker-compose](./docker/docker-compose.yml) file the `master` runs the [docker-start](./docker/docker-start.sh) script, which that script does the following:
1. Start Spark
2. Install the `pyspark` and `notebook` libraries.
3. Start the Jupyter server.
**Important**
The Jupyter server is started in a way that doesn't require a password and allows connections from anywhere, which is not recommended for most uses.
```bash
jupyter-lab --ip=0.0.0.0 --port 8888 --no-browser --notebook-dir=/proyecto --NotebookApp.allow_origin="*" --allow-root --ServerApp.password='' --ServerApp.token='' --ServerApp.password_required=False
```
### Errors
In case of using linux it is possible that the `master` container returns an error, in that case it may be necessary to do the following before starting the environment:
```shell
chmod +x docker-start.sh
```