https://github.com/piotr-kalanski/big-data-dev-environment-docker

Big Data Development environment based on Docker
https://github.com/piotr-kalanski/big-data-dev-environment-docker

big-data docker elasticsearch hadoop kafka kibana spark

Last synced: 4 months ago
JSON representation

Big Data Development environment based on Docker

• Host: GitHub
• URL: https://github.com/piotr-kalanski/big-data-dev-environment-docker
• Owner: piotr-kalanski
• License: apache-2.0
• Created: 2018-02-03T12:19:16.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2018-02-07T16:29:11.000Z (over 8 years ago)
• Last Synced: 2025-02-28T13:39:23.530Z (over 1 year ago)
• Topics: big-data, docker, elasticsearch, hadoop, kafka, kibana, spark
• Language: Python
• Size: 372 KB
• Stars: 5
• Watchers: 3
• Forks: 2
• Open Issues: 14
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# big-data-dev-environment-docker

Big Data development environment based on Docker.

![](images/environment_diagram.png)

# Run

Run environment with command:

    docker-compose up

    

You can run selected services by specifying names in the end, for example to run PostgreSQL and Elasticsearch run:

    docker-compose up postgres elasticsearch 

    

# Bind mounts

## Airflow dags

Airflow dags can be copied on host to directory [dags](/dags), so that they will be visible inside Docker container with airflow. 

## Spark applications

Spark applications can be copied to directory [spark-jobs](/spark-jobs), so that they will be visible inside Docker container with Spark at `/opt/transformations` directory.

## Kafka Connect JDBC drivers

If you want to add new JDBC drivers to Kafka Connect you can copy them locally to directory [kafka-connect-jdbc](/kafka-connect-jdbc).

# Examples

- [How to connect to local S3 with S3 browser](/examples/example-s3-browser)

- [How to integrate Java AWS SDK API with fake S3](/examples/example-java-s3-local)

- [How to integrate Spark with local Hadoop](/examples/example-spark-hdfs)

- [How to connect from AWS CLI to local Dynamo DB](/examples/example-dynamodb-with-awscli)

- [How to integrate Java AWS SDK API with local Dynamo DB](/examples/example-java-dynamodb-local)