{"id":24173074,"url":"https://github.com/young-ook/data-lab-on-wsl","last_synced_at":"2025-03-02T13:17:31.043Z","repository":{"id":270453376,"uuid":"909235021","full_name":"Young-ook/data-lab-on-wsl","owner":"Young-ook","description":"Data on WSL (Windows Subsystem for Linux)","archived":false,"fork":false,"pushed_at":"2025-02-17T22:55:14.000Z","size":2663,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-02T13:17:24.812Z","etag":null,"topics":["analytics","data-science","jupyter","spark"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Young-ook.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-28T05:13:27.000Z","updated_at":"2025-02-17T22:55:16.000Z","dependencies_parsed_at":"2024-12-31T09:31:47.304Z","dependency_job_id":"ff9ad109-55dd-4182-a057-e50346dfc8ce","html_url":"https://github.com/Young-ook/data-lab-on-wsl","commit_stats":null,"previous_names":["young-ook/data-lab-on-wsl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Young-ook%2Fdata-lab-on-wsl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Young-ook%2Fdata-lab-on-wsl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Young-ook%2Fdata-lab-on-wsl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Young-ook%2Fdata-lab-on-wsl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Young-ook","download_url":"https://codeload.github.com/Young-ook/data-lab-on-wsl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241509655,"owners_count":19974071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","data-science","jupyter","spark"],"created_at":"2025-01-13T01:14:21.161Z","updated_at":"2025-03-02T13:17:31.035Z","avatar_url":"https://github.com/Young-ook.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data on WSL(Windows Subsystem for Linux)\n\u003e [!TIP]\n\u003e This is a project shows how to build an AI/ML system in a WSL environment. Of course, the same can be applied in a general Linux environment.\n\n## Install Debian on WSL\nOpen Settings \u003e Apps \u003e Programs and Features \u003e Turn Windows features on or off dialog and select the *Windows Subsystem for Linux* to enable WSL on your system. You may reboot your system.\n\n![enable-wsl](images/enable-wsl.png)\n\nAfter you have enabled WSL, you can install linux distribution via Microsoft Store. We will use the latest version of Debian linux for the hands-on lab. Open Microsoft Store app and search *Debian* (Debian 12, Bookworm), and install.\n\nTo verify your install, open windows terminal or command terminal and run `wsl -l -v` command to list WSL distributions.\n\n## Setup Workspace\n### JupyterLab\n#### Install Python\nIn this example, we will use jupyter notebook as primary interactive interface for AI, ML, Analytics examples. The runtime requiremet is Python 3.11.2 on Debian 12 linux. The first step is installing python and python virtual environment module:\n```\nsudo apt update\nsudo apt install python3 python3-venv python3-pip-whl python-is-python3\n```\n\nVerify your python installation status using `python -V` or `python --version`. \n**Note** Configure alias to the python version 3 binary file if you are not able to install *python-is-python3* package.\n```\nedit ~/.bashrc\nalias python=\"/usr/bin/python3\"\n```\n\n#### Activate your virtual environment and install Jupyter\nThe next step is activating your python virtual environment for jupyter workspace. Under the cloned *data-lab-on-wsl* repositiry on your local environment, run source command to activate a new virtual environment:\n```\npython -m venv .venv\nsource ./.venv/bin/activate\n```\n\nAfter you are in your virtual environment, install jupyter package and dependencies using PIP(Package Installer for Python). The packages described in the *requirements.txt* file are tested on python 3.11.2, therefore you may see error if you are running on different python version.\n```\n(.venv) pip install -r requirements.txt\n```\n\n#### Launch Jupyter Lab\nLaunch a jupyter lab and open a web browser to access:\n```\n(.venv) jupyter-lab --no-brower (--port 8080)\n```\n\n### Apache Spark\n#### Install Java\nFor Apache Spark, we need to install apache spark application. The runtime environment is OpenJDK 11 (or Amazon Corretto 11) on Debian 12 linux. The first step is installing java. The default JDK for Debian 12 is OpenJDK 11, and follow the instructions to install OpenJDK 11.\n```\nsudo apt install default-jdk\n```\nOr, you can use Amazon Corretto\n```\ncurl -LO https://corretto.aws/downloads/latest/amazon-corretto-11-x64-linux-jdk.deb\nsudo apt install java-common \u0026\u0026 sudo dpkg -i amazon-corretto-11-x64-linux-jdk.deb\n```\n\nIf you have installed the multiple java runtimes, you can switch to the java runtime you want using `update-alternatives` command.\n```\nsudo update-alternatives java\n\n*+  1   /usr/lib/jvm/java-7-openjdk-amd64/bin/java\n    2   /usr/lib/jvm/java-8-openjdk-amd64/bin/java\n\nEnter to keep the current selection[+], or type selection number:\n```\n\n#### Install Spark\nAfter install java, you need to install spark. The spark version we will use for this example is 3.5.4. Download the binary package file:\n```\nwget https://dlcdn.apache.org/spark/spark-3.5.4/spark-3.5.4-bin-hadoop3.tgz\n```\nOr you can download using curl if you don't have wget.\n```\ncurl -O https://dlcdn.apache.org/spark/spark-3.5.4/spark-3.5.4-bin-hadoop3.tgz\n```\n\nAnd move the extracted directory where you want to install. In this example, we will install spark under the *$HOME/.local/lib/*. If you finished to move the file, you have to set system environment variable to the current spark runtime:\n```\nexport SPARK_HOME=\"$HOME/.local/lib/spark-3.5.4-bin-hadoop3\"\n```\n\nTo set the environment variables permanently, add the script below to your profile *$HOME/.profile* (if zsh, *$HOME/.zprofile*).\n```\n# It is required to set the SPARK_HOME environment variable.\n# Please make sure the variable indicates to the right path to your spark.\nif [ -z $SPARK_HOME ] ; then\n  export SPARK_HOME=\"$HOME/.local/lib/spark-3.5.4-bin-hadoop3\"\nfi\n```\n\nFor compatibility, you can also define environment variable settings in your *$HOME/.profile* and set zsh to emulate sh-mode in effect.\n```\n[[ -e ~/.profile ]] \u0026\u0026 emulate sh -c 'source ~/.profile'\n```\n\n#### Launch a Standalone Cluster\nRun your local standalone spark cluster:\n```\nsh spark-ctl.sh -r\n```\n\n### Apache Airflow\n#### Install Airflow\nYou might have installed Airflow when you tried to install the Jupyter using `requirements.txt`, but if not, follow the *(Optional) Set Airlfow home directory* and *Install Airflow using the constraints file, which is determined based on the URL we pass* steps to install Airflow from PyPI(Python Package Index).\n\n\u003e [!WARNING]\n\u003e **Don't Forget** to make sure that you've activated the python virtual environment with `source .venv/bin/activate` in the *data-lab-on-wsl* local directory.\n\n##### (Optional) Set Airlfow home directory\nThe first time you run Airflow, it will create a file called `airflow.cfg` in your `AIRFLOW_HOME` directory (`$HOME/airflow` by default). The `AIRFLOW_HOME` environment variable is used to inform Airflow of the desired location. This step of setting the environment variable should be done before installing Airflow so that the installation process knows where to store the necessary files.\n\nYou can set the home directory to a similar path commonly used by other tools: `export AIRFLOW_HOME=$HOME/.airflow`\n. Or, if you want one easy way to manage things, you can put everything in one place. Go to the *data-lab-on-wsl/examples/jupyter/ml-ops* and configure the environment variable: `export AIRFLOW_HOME=$PWD/airflow`.\n\nIn this example, we will use the default home directory, skip for now.\n\n##### Install Airflow using the constraints file, which is determined based on the URL we pass\nRun the below where the same virtual environment Jupyter is running Jupyter. In this example we will install version 2.10.4, but if you always want to install the latest version, just remove the version from the pip install command (`pip install apache-airflow`).\n```\nAIRFLOW_VERSION=2.10.4\n\n# Extract the version of Python you have installed. If you're currently using a Python version that is not supported by Airflow, you may want to set this manually.\n# See above for supported versions.\nPYTHON_VERSION=\"$(python -c 'import sys; print(f\"{sys.version_info.major}.{sys.version_info.minor}\")')\"\n\n#CONSTRAINT_URL=\"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt\"\n# For example this would install 2.10.4 with python 3.8: https://raw.githubusercontent.com/apache/airflow/constraints-2.10.4/constraints-3.8.txt\n\npip install \"apache-airflow==${AIRFLOW_VERSION}\" #--constraint \"${CONSTRAINT_URL}\"\n```\n\n#### Run Airflow Standalone\nRun `airflow standalone` command to initialize the database, creates a user, and starts all components at once. The PID file for the webserver will be stored in `$AIRFLOW_HOME/airflow-webserver.pid` or in `/run/airflow/webserver.pid` if started by systemd.\n\n\u003e [!NOTE]\n\u003e This local system is simple and easy to use for testing or practice, but we recommend enable security, governance, monitoring, reverse proxing, persistent backend and more for use in production.\n\nIf you want to run the individual parts of Airflow manually rather than using the all-in-one standalone command, you can instead run:\n```\nairflow db migrate\n\nairflow users create --username admin --firstname FIRST_NAME --lastname LAST_NAME --role Admin --email admin@example.org\nPassword:\n\nairflow webserver --port 8080\nairflow scheduler\n```\n\n# Examples\n- [Airflow](examples/airflow/README.md)\n- [Jupyter](examples/jupyter/README.md)\n- [Spark](examples/spark/README.md)\n\n# Clean up\nYou can stop and terminate the running Juypter and Airflow by simply pressing `ctrl+c` and following the instructions that appear. Then, type `deactivate` to exit the virtual environment:\n```\n(.venv) deactivate\n```\n\nThen, you can stop your local standalone Spark cluster after your Spark job was finished:\n```\nsh spark-ctl.sh -t\n```\n\n# Additional Resources\n- [Terraform: Amazon SageMaker](https://github.com/Young-ook/terraform-aws-sagemaker/tree/main/examples/blueprint)\n- [Terraform: Data on Amazon EKS](https://github.com/Young-ook/terraform-aws-eks/tree/main/examples/data-ai)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyoung-ook%2Fdata-lab-on-wsl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyoung-ook%2Fdata-lab-on-wsl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyoung-ook%2Fdata-lab-on-wsl/lists"}