https://github.com/mvillafuertem/spark
📊 📈 📉
https://github.com/mvillafuertem/spark
ansible cdktf molecule packer python scala spark terraform
Last synced: 3 months ago
JSON representation
📊 📈 📉
- Host: GitHub
- URL: https://github.com/mvillafuertem/spark
- Owner: mvillafuertem
- Created: 2021-10-23T17:56:00.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2022-10-26T21:44:49.000Z (over 3 years ago)
- Last Synced: 2024-12-29T03:19:46.529Z (over 1 year ago)
- Topics: ansible, cdktf, molecule, packer, python, scala, spark, terraform
- Language: Scala
- Homepage:
- Size: 6.62 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
:warning: PLEASE, DO NOT EDIT THIS FILE.
IT IS AUTOGENERATE YOU SHOULD EDIT `modules/docs/src/main/mdoc/README.md`
WITH THE COMMAND `sbt docs/mdoc`
---
Spark
****
Spark is a project with many proof of concept modules...
****
## Simple Data Lake
## Simple Data Lake using AWS Lake Formation
## Loading Real-time Data Into Amazon Redshift Cluster
## Useful commands
### eks
```shell
# Install aws cli and configure
brew install awscli
aws configure
# configure kube config
aws eks --profile awspro --region eu-west-2 update-kubeconfig --name eks-cluster-name
```
### pyenv
Installing pyenv on Mac OS X is very simple with Homebrew.
```shell
brew install pyenv
brew install pyenv-virtualenv
```
After installation, add the following at the bottom of your profile (~/.bash_profile or ~/.zshenv)
```shell
if which pyenv > /dev/null; then eval "$(pyenv init -)"; fi
if which pyenv-virtualenv-init > /dev/null; then eval "$(pyenv virtualenv-init -)"; fi
```
```shell
export PATH="$(pyenv root)/shims:$PATH"
```
Installing new Python versions is very straightforward. All Python versions are installed in the versions directory under the pyenv root.
```shell
pyenv install 3.9.5
```
The pyenv versions command will also show the virtual environments along with the installed versions of Python.
```shell
pyenv versions
```
Create a virtual environment for a specific Python version.
```shell
pyenv virtualenv 3.9.5 python-3.9.5
```
List existing virtual environments created by pyenv
```shell
pyenv virtualenvs
```
Activate a virtual environment.
```shell
pyenv activate python-3.9.5
pyenv local python-3.9.5
```
### pip
```shell
pip --version
```
### molecule
Install molecule
```shell
pip install 'molecule[docker]'
```
Create a role
```shell
molecule init role --driver-name docker bastion
```
Run test
```shell
molecule test
```
Run test
```shell
molecule destroy
```
#### Links
https://www.akbaribrahim.com/managing-multiple-python-versions-with-pyenv/
https://www.akbaribrahim.com/managing-python-virtual-environments-with-pyenv-virtualenv/
https://zacks.one/aws-cloud-architecting/