{"id":26991622,"url":"https://github.com/0-mostafa-rezaee-0/docker_for_data_science_projects","last_synced_at":"2026-05-10T03:52:01.893Z","repository":{"id":285814697,"uuid":"959440090","full_name":"0-mostafa-rezaee-0/Docker_for_Data_Science_Projects","owner":"0-mostafa-rezaee-0","description":"QuickStart template for data science projects using Docker instead of Conda or venv.","archived":false,"fork":false,"pushed_at":"2025-04-03T00:16:39.000Z","size":115,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-03T00:27:31.226Z","etag":null,"topics":["containerization-with-docker","data-science","development-environment","docker","jupyter-notebook","python","reproducible-research"],"latest_commit_sha":null,"homepage":"https://www.linkedin.com/in/mostafa-rezaee/","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0-mostafa-rezaee-0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-04-02T19:35:56.000Z","updated_at":"2025-04-03T00:20:48.000Z","dependencies_parsed_at":"2025-04-03T00:27:34.364Z","dependency_job_id":"7456edd7-e704-4ad3-a703-402993327367","html_url":"https://github.com/0-mostafa-rezaee-0/Docker_for_Data_Science_Projects","commit_stats":null,"previous_names":["0-mostafa-rezaee-0/docker_data-science_quickstart","0-mostafa-rezaee-0/docker_for_data_science_projects"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0-mostafa-rezaee-0%2FDocker_for_Data_Science_Projects","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0-mostafa-rezaee-0%2FDocker_for_Data_Science_Projects/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0-mostafa-rezaee-0%2FDocker_for_Data_Science_Projects/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0-mostafa-rezaee-0%2FDocker_for_Data_Science_Projects/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0-mostafa-rezaee-0","download_url":"https://codeload.github.com/0-mostafa-rezaee-0/Docker_for_Data_Science_Projects/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248067237,"owners_count":21042258,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containerization-with-docker","data-science","development-environment","docker","jupyter-notebook","python","reproducible-research"],"created_at":"2025-04-03T22:16:21.516Z","updated_at":"2026-05-10T03:52:01.879Z","avatar_url":"https://github.com/0-mostafa-rezaee-0.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 🧰 How to Use This Template    \n\nClick the green **\"Use this template\"** button at the top of the page, then choose **\"Create a new repository\"**.   \n\nThis will create your own copy of this project, which you can modify freely — no need to fork!   \n\n---    \n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"figures/logo.png\" alt=\"logo\" width=\"50%\"\u003e\n\u003c/div\u003e\n\n# Docker for Data Science Projects\n\nQuickStart template for data science projects using Docker instead of Conda or venv.\n\n## Table of Contents\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003ca href=\"#1-about-this-repository\"\u003e\u003ci\u003e\u003cb\u003e1. About this Repository\u003c/b\u003e\u003c/i\u003e\u003c/a\u003e\u003c/summary\u003e\n  \u003cdiv\u003e\n              \u003ca href=\"#11-who-is-this-project-for\"\u003e1.1. Who Is This Project For?\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#12-what-will-you-learn\"\u003e1.2. What Will You Learn?\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#13-prerequisites\"\u003e1.3. Prerequisites\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#14-contents-of-this-repository\"\u003e1.4. Contents of this Repository\u003c/a\u003e\u003cbr\u003e\n  \u003c/div\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003ca href=\"#2-docker-concepts\"\u003e\u003ci\u003e\u003cb\u003e2. Docker Concepts\u003c/b\u003e\u003c/i\u003e\u003c/a\u003e\u003c/summary\u003e\n  \u003cdiv\u003e\n              \u003ca href=\"#21-dockerfile\"\u003e2.1. Dockerfile\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#22-build-command\"\u003e2.2. Build Command\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#23-docker-image\"\u003e2.3. Docker Image\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#24-run-command\"\u003e2.4. Run Command\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#25-docker-container\"\u003e2.5. Docker Container\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#26-docker-ignore-dockerignore\"\u003e2.6. Docker Ignore (.dockerignore)\u003c/a\u003e\u003cbr\u003e\n  \u003c/div\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003ca href=\"#3-installing-docker\"\u003e\u003ci\u003e\u003cb\u003e3. Installing Docker\u003c/b\u003e\u003c/i\u003e\u003c/a\u003e\u003c/summary\u003e\n  \u003cdiv\u003e\n              \u003ca href=\"#31-installing-docker-on-ubuntu\"\u003e3.1. Installing Docker on Ubuntu\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#32-installing-docker-on-windows\"\u003e3.2. Installing Docker on Windows\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#33-after-installing-docker\"\u003e3.3. After Installing Docker\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#34-automating-docker-startup-in-wsl\"\u003e3.4. Automating Docker Startup in WSL\u003c/a\u003e\u003cbr\u003e\n  \u003c/div\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003ca href=\"#4-setting-up-docker-for-a-data-science-project\"\u003e\u003ci\u003e\u003cb\u003e4. Setting Up Docker for a Data Science Project\u003c/b\u003e\u003c/i\u003e\u003c/a\u003e\u003c/summary\u003e\n  \u003cdiv\u003e\n              \u003ca href=\"#41-step-1-install-prerequisites\"\u003e4.1. Step 1: Install Prerequisites\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#42-step-2-set-up-your-project-repository\"\u003e4.2. Step 2: Set Up Your Project Repository\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#43-step-3-write-the-dockerfile\"\u003e4.3. Step 3: Write the Dockerfile\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#44-step-4-write-the-dockerignore-file\"\u003e4.4. Step 4: Write the .dockerignore file\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#45-step-5-write-the-docker-compose-file\"\u003e4.5. Step 5: Write the Docker Compose File\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#46-step-6-requirementstxt\"\u003e4.6. Step 6: requirements.txt\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#47-step-7-build-and-run-your-container\"\u003e4.7. Step 7: Build and Run Your Container\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#48-step-8-verify-the-container\"\u003e4.8. Step 8: Verify the Container\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#49-step-9-attach-vs-code-to-the-container\"\u003e4.9. Step 9: Attach VS Code to the Container\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#410-step-10-run-the-python-script\"\u003e4.10. Step 10: Run the Python Script\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#411-step-11-work-with-jupyter-notebooks-in-vs-code\"\u003e4.11. Step 11: Work with Jupyter Notebooks in VS Code\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#412-step-12-stop-and-remove-the-container\"\u003e4.12. Step 12: Stop and remove the container\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#413-note-1-jupyter-on-browser\"\u003e4.13. Note 1: Jupyter on browser\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#414-note2-keeping-your-environment-up-to-date\"\u003e4.14. Note2: Keeping Your Environment Up-to-Date\u003c/a\u003e\u003cbr\u003e\n  \u003c/div\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003ca href=\"#5-essential-docker-commands\"\u003e\u003ci\u003e\u003cb\u003e5. Essential Docker Commands\u003c/b\u003e\u003c/i\u003e\u003c/a\u003e\u003c/summary\u003e\n  \u003cdiv\u003e\n              \u003ca href=\"#51-managing-images\"\u003e5.1. Managing Images\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#52-managing-containers\"\u003e5.2. Managing Containers\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#53-port-mapping-commands\"\u003e5.3. Port Mapping Commands\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#54-working-with-containers\"\u003e5.4. Working with Containers\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#55-custom-container-names\"\u003e5.5. Custom Container Names\u003c/a\u003e\u003cbr\u003e\n  \u003c/div\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003ca href=\"#6-advanced-topics-and-faq\"\u003e\u003ci\u003e\u003cb\u003e6. Advanced Topics and FAQ\u003c/b\u003e\u003c/i\u003e\u003c/a\u003e\u003c/summary\u003e\n  \u003cdiv\u003e\n              \u003ca href=\"#61-understanding-network-ports\"\u003e6.1. Understanding Network Ports\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#62-docker-port-mapping-in-detail\"\u003e6.2. Docker Port Mapping in Detail\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#63-common-issues-and-solutions\"\u003e6.3. Common Issues and Solutions\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#64-data-science-specific-considerations\"\u003e6.4. Data Science Specific Considerations\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#65-docker-shortcuts-alias\"\u003e6.5. Docker Shortcuts (alias)\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#66-understanding-and-cleaning-dangling-images\"\u003e6.6. Understanding and Cleaning Dangling Images\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#67-tagging-docker-images\"\u003e6.7. Tagging Docker Images\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#68-working-with-docker-volumes\"\u003e6.8. Working with Docker Volumes\u003c/a\u003e\u003cbr\u003e\n              \u003ca href=\"#69-frequently-asked-questions-faq\"\u003e6.9. Frequently Asked Questions (FAQ)\u003c/a\u003e\u003cbr\u003e\n  \u003c/div\u003e\n\u003c/details\u003e\n\n## 1. About this Repository\n\nThis project demonstrates an end-to-end Docker workflow for data science applications. You can train machine learning models, develop Python scripts, experiment with Jupyter notebooks, and manage your data – all within Docker containers. The project is designed to be reproducible and maintainable.\n\n### 1.1. Who Is This Project For?\n\nThis project is designed for anyone interested in data science, Python development, or containerization with Docker. Whether you're a student, developer, or data scientist, this resource will guide you through building and deploying a data science environment using Docker.\n\n### 1.2. What Will You Learn?\n\nBy the end of this project, you will:\n\n- Develop a foundational understanding of Docker and containerization\n- Learn how to set up a complete data science environment in containers\n- Understand how to manage dependencies using Docker\n- Explore how to develop and run Python scripts and Jupyter notebooks in containers\n- Work with practical examples to build reproducible data science workflows\n- Gain insights into Docker best practices for data scientists\n\n### 1.3. Prerequisites\n\nThis project is suitable for three types of learners:\n\n- **For those familiar with Docker:**\n  - You can dive straight into the data science applications. The examples and configurations provided will help you enhance your skills and explore best practices.\n- **For those who know Python/data science but are new to Docker:**\n  - This project will introduce you to containerization, guiding you through building and deploying reproducible environments.\n- **For beginners**:\n- This project is designed with you in mind. You'll start with the basics, learning how to set up Docker and then move on to building data science applications in containers.\n\n### 1.4. Contents of this Repository\n\n```\nFolder PATH listing\n.\n+---data                          \u003c-- Contains sample datasets\n|       README.md                 \u003c-- Documentation for the data folder\n|       sample.csv                \u003c-- Example dataset for experimentation\n|\n+---figures                       \u003c-- Contains images for documentation\n|       README.md                 \u003c-- Documentation for the figures folder\n|       docker.jpg                \u003c-- Docker concepts illustration\n|       port.jpg                  \u003c-- Network port illustration\n|       volume.jpg                \u003c-- Docker volumes illustration\n|\n+---notebooks                     \u003c-- Jupyter notebooks\n|       README.md                 \u003c-- Documentation for the notebooks folder\n|       exploratory_analysis.ipynb \u003c-- Sample notebook for data exploration\n|\n+---scripts                       \u003c-- Python scripts\n|       README.md                 \u003c-- Documentation for the scripts folder\n|       data_prep.py              \u003c-- Sample data preparation script\n|\n|   .dockerignore                 \u003c-- Files to exclude from Docker build\n|   .gitignore                    \u003c-- Files to exclude from git\n|   docker-compose.yml            \u003c-- Docker Compose configuration\n|   Dockerfile                    \u003c-- Docker image definition\n|   LICENSE                       \u003c-- License information\n|   README.md                     \u003c-- This documentation file\n|   requirements.txt              \u003c-- Python dependencies\n```\n\n## 2. Docker Concepts\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"figures/docker.jpg\" alt=\"docker\" width=\"50%\"\u003e\n\u003c/div\u003e\n\nIn simple terms:\n\n- Docker: The most advanced environment manager\n- Dockerfile: A recipe for a dish\n- Docker Image: A cooked dish\n- Docker Compose: Instructions for serving the dish\n- Docker Container: A served dish\n\nIn technical terms:\n\n### 2.1. Dockerfile\n\n- A file named \"Dockerfile\" (with capital D) that specifies how the image should be built. For example, it mentions the Python version and states that the list of Python packages is in the requirements.txt file.\n- This file is usually placed in the root of our project.\n\n### 2.2. Build Command\n\n- With this command, an image is created based on the instructions written in the Dockerfile.\n\n### 2.3. Docker Image\n\n- The created image is actually a file containing a lightweight Ubuntu Linux with installed packages. For example, a lightweight Python and some Python libraries.\n- The created image is like a compressed (zipped) file.\n- Therefore, it's easily portable and shareable.\n- But it can't be used until it's unpacked.\n\n### 2.4. Run Command\n\nThis command creates a container from an image.\n\n- It unpacks the image (which is like a compressed file) to make it usable.\n- This command is usually long and complex, and differs for each image. Therefore, it's not easy to memorize.\n- docker-compose.yml file: To solve the run command problem, this command is written in a yml file and placed in the root of the image. From now on, the work becomes simple because with a simple command (constant for all images), a container can be started and stopped.\n  ```\n  docker-compose up --build -d\n  docker-compose down\n  ```\n- Writing the docker-compose.yml file is practically the hardest part of Docker and has its own specific points for each project. We have prepared this file for everyday data science tasks, which follows. For other projects, such as a website, you need to learn specifically for that task. ChatGPT can also be very helpful.\n\n### 2.5. Docker Container\n\n- A container is a lightweight Ubuntu Linux with installed packages.\n- Obviously, a container does not have the ability to be moved and shared, and whenever we make changes to it and want to share it with others, we need to create an image from it again. Then share the resulting image.\n\n### 2.6. Docker Ignore (.dockerignore)\n\nThe following questions are covered in order:\n\n- Is a .dockerignore file still needed when there's a .gitignore in the project? Yes\n- What's the difference between .dockerignore and .gitignore?\n- An appropriate .dockerignore for a data science project.\n- Explanation of .dockerignore content.\n\nLet me help you with creating a .dockerignore file. Yes, you should have a .dockerignore file even if you already have a .gitignore. While both files serve similar purposes of excluding files from operations, they have different contexts:\n\n1. .gitignore prevents files from being tracked in Git version control\n2. .dockerignore prevents files from being copied into Docker images during the build process\n\nHaving a .dockerignore file is important because it:\n\n- Reduces the build context size, making builds faster\n- Prevents sensitive information from being copied into your Docker images\n- Improves build cache efficiency\n- Prevents unnecessary files from bloating your Docker images\n\nA .dockerignore file specifically tailored for Docker builds includes patterns to exclude:\n\n1. Python-specific: Compiled Python files, cache, and build artifacts that shouldn't be in the Docker image\n2. Virtual environments: Local virtual environments that shouldn't be copied into the image\n3. Development and IDE files: Editor configs and Git-related files that aren't needed in production\n4. Docker-specific: Dockerfile, docker-compose files, and .dockerignore itself\n5. Build and distribution: Local build artifacts\n6. System files: OS-specific files like .DS_Store and Windows Zone identifiers\n\n## 3. Installing Docker\n\n### 3.1. Installing Docker on Ubuntu\n\nYou can easily install Docker using the official documentation or with ChatGPT assistance. After installation, verify it's working properly by running Docker commands like `docker images`, `docker ps`, and the hello-world container.\n\n```bash\n# Add Docker's official GPG key:\nsudo apt-get update\nsudo apt-get install ca-certificates curl\nsudo install -m 0755 -d /etc/apt/keyrings\nsudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc\nsudo chmod a+r /etc/apt/keyrings/docker.asc\n\n# Add the repository to Apt sources:\necho \\\n  \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \\\n  $(. /etc/os-release \u0026\u0026 echo \"$VERSION_CODENAME\") stable\" | \\\n  sudo tee /etc/apt/sources.list.d/docker.list \u003e /dev/null\nsudo apt-get update\n\n# Install Docker and Docker Compose\nsudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin\n\n# Install the latest Docker Compose\nsudo curl -L \"https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)\" -o /usr/local/bin/docker-compose\n\nsudo chmod +x /usr/local/bin/docker-compose\n```\n\n### 3.2. Installing Docker on Windows\n\nThere's nothing special about this process. Simply download and install the 64-bit version of Docker Desktop for Windows.\n\n**Note**: To connect VS Code to Docker, Docker must be installed on Windows itself; installing it in WSL is not sufficient.\n\n### 3.3. After Installing Docker\n\nAfter installation, verify Docker is working correctly:\n\n```bash\ndocker --version\nsudo systemctl enable docker\nsudo service docker start\n```\n\n**Note for WSL users**:\nWSL does not use systemd, so `systemctl` commands don't work inside WSL. In WSL, you'll need to run `sudo service docker start` each time you boot your laptop. You can automate this with a script or alias.\n\n```bash\n# Check Docker installation\nsudo docker images\nsudo docker ps\n```\n\nTo use Docker without `sudo`:\n\n```bash\nsudo usermod -aG docker $USER\n```\n\nTest it:\n\n```bash\ndocker images\ndocker ps\n```\n\n### 3.4. Automating Docker Startup in WSL\n\nIn WSL, the `sudo systemctl enable docker` command doesn't work because WSL doesn't use systemd. Here are options to start Docker automatically:\n\n#### Option 1: Manual\n\nIf you're okay with typing a command daily, just stick with:\n\n```bash\nsudo service docker start\n```\n\n#### Option 2: Using an alias\n\nCreate an alias to shorten the command:\n\n```bash\necho 'alias start-docker=\"sudo service docker start\"' \u003e\u003e ~/.bashrc\nsource ~/.bashrc\n```\n\nNow, you can just type:\n\n```bash\nstart-docker\n```\n\n#### Option 3: Automatic\n\nTo start Docker automatically when you open WSL:\n\n1. Open WSL and edit the WSL configuration file:\n\n   ```bash\n   sudo nano /etc/wsl.conf\n   ```\n2. Add the following lines:\n\n   ```\n   [boot]\n   command=\"service docker start\"\n   ```\n3. Save the file (Ctrl + X, then Y, then Enter).\n4. Restart WSL:\n\n   ```bash\n   wsl --shutdown\n   ```\n\n## 4. Setting Up Docker for a Data Science Project\n\nDocker Project Setup for Python and Jupyter Notebooks\nThis guide creates a portable and reproducible Docker project template that lets you develop Python scripts and Jupyter notebooks using VS Code in a containerized environment.\n\n### 4.1. Step 1: Install Prerequisites\n\n- Install Docker Desktop with WSL integration on Windows 11.\n- Install Visual Studio Code.\n- In VS Code, install these extensions: Docker, Remote - Containers, Python, and Jupyter.\n\n### 4.2. Step 2: Set Up Your Project Repository\n\n- Create a new Git repository (or clone an existing one).\n- In the repository folder, create these files:\n  - Dockerfile\n  - .dockerignore\n  - docker-compose.yml\n  - requirements.txt\n  - data_prep.py\n  - exploratory_analysis.ipynb\n\n### 4.3. Step 3: Write the Dockerfile\n\nPlace the following content in your Dockerfile:\n\n```\n# Base image with Python 3.9\nFROM python:3.9\n\n# Set the working directory\nWORKDIR /app\n\n# Copy requirements and install dependencies\nCOPY requirements.txt .\nRUN pip install --no-cache-dir -r requirements.txt\n\n# Install Jupyter Notebook and JupyterLab\nRUN pip install notebook jupyterlab\n\n# Expose port 8888 for Jupyter\nEXPOSE 8888\n\n# Start Jupyter Notebook with no token for development\nENTRYPOINT [\"sh\", \"-c\", \"exec jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token=''\"]\n```\n\n### 4.4. Step 4: Write the .dockerignore file\n\nCreate a .dockerignore file in your project root to prevent unnecessary files from being included in your Docker image.\n\n### 4.5. Step 5: Write the Docker Compose File\n\nIn docker-compose.yml, add:\n\n```\nservices:\n  your-project:\n    build: .\n    image: your-project_image\n    container_name: your-project_container\n    volumes:\n      - .:/app\n    stdin_open: true\n    tty: true\n    ports:\n      - \"8888:8888\"\n```\n\nThis mounts your entire project folder into the container at /app.\nNote:\nIn lines 2, 3, and 4, replace your-project with your project's name, for example: dockerproject1.\n\n### 4.6. Step 6: requirements.txt\n\nIt's important to keep the `requirements.txt` file clean and up-to-date. This ensures that all necessary dependencies are installed correctly and helps maintain compatibility and performance. The `ipykernel` package is crucial for Jupyter notebook functionality, so make sure it is included.\n\n```\nipykernel # This package is essential for running Jupyter notebooks.\nnumpy==1.26.0\npandas==2.1.3\nmatplotlib==3.8.0\n```\n\nBest Practice:\nUse pip in Docker unless Conda is essential.\nStick to requirements.txt for best compatibility and performance.\n\n### 4.7. Step 7: Build and Run Your Container\n\nOn your host machine (in the project folder), you have two options:\n\n* **First (recommended):**\n  This method extracts the project name to use as the image and container names.\n\n  To make start.sh executable if it is not:\n\n  ```bash\n  chmod +x start.sh \n  ```\n\n  To extracts the project name and then build the image and run the container:\n\n  ```bash\n  ./start.sh \n  ```\n* **Second:**    \n  In this method, the image and container names default to `data-science-project`.\n\n  ```bash\n  docker-compose up --build -d\n  ```\n\n**Note:**\n\n* `--build`: \"We could omit \"--build\", but then changes to Dockerfile or dependencies would not be applied.\n* `-d`: The \"-d\" flag runs the container in detached mode, allowing you to continue using the terminal for other tasks.\n\n### 4.8. Step 8: Verify the Container\n\nRun:\n\n```\ndocker-compose ps\n```\n\nMake sure the container status is \"Up\" and port 8888 is mapped.\n\n### 4.9. Step 9: Attach VS Code to the Container\n\nFollow these steps carefully:\n\n1. Press `Ctrl+Shift+P` to open the command palette.\n2. Type and select `Dev Containers: Attach to Running Container…`.\n3. Choose the container named `your-project_name`. A second VS Code window will open.\n4. In the second VS Code window, click `Open Folder`. In the top box, you will see `/root`. Delete `root` to reveal `app`. Select `app` and click `OK`. You will then see all your project's folders and files.\n5. In the second VS Code window, install the following extensions: `Docker`, `Dev Containers`, `Python`, and `Jupyter`. If you see a `Reload the window` button after installing each extension, make sure to click it every time.\n6. You are all set and can continue.\n\n**Note**: In `Step 11`, if you cannot select the kernel, close the second VS Code window and repeat steps 1, 2, 3, and 4. The correct kernel will then be automatically attached to the notebooks.\n\n### 4.10. Step 10: Run the Python Script\n\nIn the VS Code terminal, open the terminal. You will see a bash which means you are inside the container. Run:\n\n```\npython scripts/data_prep.py\n```\n\nYou should see the expected output (for example, \"hi\").\n\n### 4.11. Step 11: Work with Jupyter Notebooks in VS Code\n\n- Open exploratory_analysis.ipynb in VS Code.\n- In the top-right corner of the notebook, you should see a kernel with the same name as your project. If not, click the `Select Kerne`l button and choose the `Jupyter kernel` option. This will display a kernel with your project's name and the Python kernel specified in the Dockerfile. The libraries from the `requirements.txt` file, installed in the Docker container, will be automatically available for use.\n- You can now run and edit cells within the container.\n\n### 4.12. Step 12: Stop and remove the container\n\n```\ndocker-compose down\n```\n\n### 4.13. Note 1: Jupyter on browser\n\nSee localhost:8888/tree?\n\n### 4.14. Note2: Keeping Your Environment Up-to-Date\n\n- To rebuild your container with any changes, run on your host:\n\n  ```\n  docker-compose up --build\n  ```\n- After installing a new package, update requirements.txt inside the container by running:\n\n  ```\n  pip freeze \u003e requirements.txt\n  ```\n- For pulling the latest base image, run:\n\n  ```\n  docker-compose build --pull\n  ```\n\n## 5. Essential Docker Commands\n\n### 5.1. Managing Images\n\n```\n# Pull images from Docker Hub\ndocker pull nginx\ndocker pull hello-world\n\n# List all images\ndocker images\n\n# Remove images\ndocker rmi \u003cimage1\u003e \u003cimage2\u003e ...\n```\n\n### 5.2. Managing Containers\n\n```\n# List running containers\ndocker ps\n\n# List all containers (including stopped ones)\ndocker ps -a\n\n# List only container IDs\ndocker ps -aq\n\n# Remove containers\ndocker rm \u003cCONTAINER1\u003e \u003cCONTAINER2\u003e ...\n\n# Remove all containers\ndocker rm $(docker ps -aq)\n\n# Run a container in detached mode\ndocker run -d \u003cIMAGE name or ID\u003e\n\n# Start/stop containers\ndocker start \u003cCONTAINER name or ID\u003e\ndocker stop \u003cCONTAINER name or ID\u003e\n\n# Start/stop all containers at once\ndocker start $(docker ps -aq)\ndocker stop $(docker ps -aq)\n```\n\n**Note**: You can use just the first two letters of a container ID for identification. For example: `docker stop 2f`\n\n### 5.3. Port Mapping Commands\n\n```\n# Run nginx and map port 80 of the host to port 80 of the container\ndocker run -d -p 80:80 nginx\n\n# Run another nginx instance on a different host port\ndocker run -d -p 8080:80 nginx\n\n# Map multiple ports\ndocker run -d -p 80:80 -p 443:443 nginx\n\n# Map all exposed ports to random ports\ndocker run -d -P nginx\n```\n\nThe `-p host_port:container_port` option maps ports between your host system and the container.\n\n### 5.4. Working with Containers\n\n```\n# Enter a container's bash shell\ndocker exec -it \u003cCONTAINER name or ID\u003e bash\n\n# Save an image to a tar file\ndocker save -o /home/mostafa/docker-projects/nginx.tar nginx\n\n# Load an image from a tar file\ndocker load -i /home/mostafa/docker-projects/nginx.tar\n```\n\n### 5.5. Custom Container Names\n\nDocker assigns random names to containers by default. To specify a custom name:\n\n```\ndocker run -d --name \u003carbitrary-name\u003e -p 80:80 \u003cimage-name\u003e\n```\n\nExample:\n\n```\ndocker run -d --name webserver -p 80:80 nginx\n```\n\n## 6. Advanced Topics and FAQ\n\n### 6.1. Understanding Network Ports\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"figures/port.jpg\" alt=\"port\" width=\"50%\"\u003e\n\u003c/div\u003e\n\nIn networking:\n\n- IP address identifies which device you're communicating with (\"who\")\n- Port number specifies which service or application on that device (\"what\")\n\nFor example, when you access: `google.com =\u003e 215.114.85.17:80`\n\n- `215.114.85.17` is Google's IP address (who you're talking to)\n- `80` is the port number for HTTP (what service you're requesting)\n\nPorts can range from `0` to `65,535` (2^16 - 1), with standard services typically using well-known ports:\n\n- Web servers:\n\n  - HTTP: port `80`\n  - HTTPS: port `443`\n- Development servers:\n\n  - FastAPI: port `8000`\n  - Jupyter: port `8888`\n  - SSH: port `22`\n- Database Management Systems (DBMS):\n\n  - MySQL: port `3306`\n  - PostgreSQL: port `5432`\n  - MongoDB: port `27017`\n\n**Important Notes on Database Ports:**\n\n- Databases themselves don't have ports; the Database Management Systems (DBMS) do.\n- All databases within a single DBMS instance typically use the same port.\n- If you want to run two versions of the same DBMS on one server, you must use different ports.\n- Exception: Some DBMS like MongoDB allow each database to run on a different port, but by default, all databases share a common port.\n\n### 6.2. Docker Port Mapping in Detail\n\nThe port mapping in Docker (`-p 80:80`) allows you to:\n\n1. Access containerized services from your host machine\n2. Run multiple instances of the same service on different host ports\n3. Avoid port conflicts when multiple containers need the same internal port\n\nWith these commands:\n\n- First container: access via `localhost:80` in browser\n- Second container: access via `localhost:8080` in browser\n- Both containers are running nginx on their internal port `80`\n\nThis approach is especially useful for data science projects when you need to:\n\n- Run multiple Jupyter servers\n- Access databases from both containerized applications and host tools\n- Expose machine learning model APIs\n\n### 6.3. Common Issues and Solutions\n\n#### Container Won't Start\n\nIf your container won't start, check:\n\n- Port conflicts: Is another service using the same port?\n- Resource limitations: Do you have enough memory/CPU?\n- Permission issues: Are volume mounts correctly configured?\n\n#### File Permissions Issues\n\nWhen using volume mounts, file permission issues can occur. Solutions:\n\n- Use the `--user` flag when running the container\n- Set appropriate permissions in the Dockerfile\n- Use Docker Compose's `user` option\n\n#### Performance Considerations\n\n- Use `.dockerignore` to reduce build context size\n- Minimize the number of layers in your Dockerfile\n- Consider multi-stage builds for smaller images\n\n### 6.4. Data Science Specific Considerations\n\n#### Jupyter Notebook Security\n\nFor production:\n\n- Don't use `--NotebookApp.token=''`\n- Set up proper authentication\n- Use HTTPS for connections\n\n#### GPU Support\n\nFor deep learning:\n\n- Install NVIDIA Container Toolkit\n- Use the `--gpus all` flag with docker run\n- Use appropriate base images (e.g., tensorflow/tensorflow:latest-gpu)\n\n#### Large Data Files\n\nWhen working with large datasets:\n\n- Don't include data in the Docker image\n- Use volume mounts for data directories\n- Consider using data volumes or bind mounts\n\n### 6.5. Docker Shortcuts (alias)\n\nAdd these aliases to your `.bashrc` or `.zshrc` file to make Docker commands more convenient:\n\n```bash\n#-----------------------------------------------------------------------------------------\n# Docker aliases\n\n# --- Image Management ---\nalias di=\"    docker images    --format 'table {{.ID}}\\t{{.Repository}}\\t{{.Tag}}\\t{{.Size}}\\t{{.CreatedSince}}'\"\nalias dia=\"   docker images -a --format 'table {{.ID}}\\t{{.Repository}}\\t{{.Tag}}\\t{{.Size}}\\t{{.CreatedSince}}'\"\nalias drmi=\"  docker rmi\"\n\ndrmia() {     docker rmi $(docker images -aq)       }  # Remove All Images\ndrmif() {                                              # Remove All dangling images\n local images=$(docker images -q -f dangling=true)\n if [ -n \"$images\" ]; then\n   echo \"Removing dangling images: $images\"\n   docker rmi $images\n else\n   echo \"No dangling images to remove.\"\n fi\n}\n\n# --- Container Management ---\nalias dps=\"   docker ps     --format 'table {{.ID}}\\t{{.Image}}\\t{{.Names}}\\t{{.Status}}\\t{{.Ports}}'\"\nalias dpsa=\"  docker ps -a  --format 'table {{.ID}}\\t{{.Image}}\\t{{.Names}}\\t{{.Status}}\\t{{.Ports}}'\"\nalias dpsaq=\" docker ps -aq --format 'table {{.ID}}\\t{{.Image}}\\t{{.Names}}\\t{{.Status}}\\t{{.Ports}}'\"\n\nalias dst=\"   docker start\"\nalias dsp=\"   docker stop\"\nalias drm=\"   docker rm\"\n\ndsta() {      docker start $(docker ps -aq)   }  # Start  All Containers\ndspa() {      docker stop  $(docker ps -aq)   }  # Stop   All Containers\ndrma() {      docker rm    $(docker ps -aq)   }  # Remove All Containers\n\n# --- Docker Compose Commands ---\nalias dcu=\"   docker compose up   -d --build\"\nalias dcd=\"   docker compose down\"\n\n# --- Docker Exec Bash ---\ndeb() {       docker exec -it \"$1\" bash   }\n```\n\nThese shortcuts provide:\n\n#### Better Formatted Output\n\n- `di`/`dia`: Lists regular/all (including dangling) images with formatted output showing ID, repository, tag, size, and age.\n- `dps`/`dpsa`: Shows running/all containers with formatted output\n\n#### Bulk Operations\n\n- `drmia`: Removes all images\n- `drmif`: Removes only \"dangling\" images (untagged images)\n- `dsta`/`dspa`: Starts/stops all containers\n- `drma`: Removes all containers\n\n#### Shorter Commands\n\n- `dst`/`dsp`: Quick container start/stop\n- `dcu`/`dcd`: Docker compose up/down with build and detached mode\n\nTo use these aliases:\n\n1. Add the code block to your shell profile file (~/.bashrc or ~/.zshrc)\n2. Run `source ~/.bashrc` or `source ~/.zshrc` to apply changes\n3. Start using the shortened commands\n\n### 6.6. Understanding and Cleaning Dangling Images\n\nWhen you run `docker images`, you might see several entries with `\u003cnone\u003e` as their repository and tag:\n\n```\nREPOSITORY                                 TAG     IMAGE ID       CREATED        SIZE\np1-ml-engineering-api-fastapi-docker-jupyter  latest  5afe18f4594a  13 hours ago   745MB\n\u003cnone\u003e                                     \u003cnone\u003e  808f843b9362  13 hours ago   748MB\n\u003cnone\u003e                                     \u003cnone\u003e  5706fd96eca0  14 hours ago   742MB\n\u003cnone\u003e                                     \u003cnone\u003e  1e904ba38c6d  14 hours ago   742MB\n```\n\n#### What are these `\u003cnone\u003e` images?\n\nThese are called \"dangling images\" and they typically appear in these scenarios:\n\n- When you rebuild an image with the same tag - the old image becomes \"dangling\" and shows up as `\u003cnone\u003e:\u003cnone\u003e`\n- When a build fails or is interrupted in the middle\n- When you pull a new version of an image, and the old one loses its tag\n\n#### Why should you care?\n\nDangling images:\n\n- Take up disk space unnecessarily\n- Make your image list harder to read\n- Serve no practical purpose\n\n#### How to remove dangling images:\n\nYou can safely remove all dangling images using:\n\n```bash\ndocker image prune -f\n```\n\nOr use the alias we defined earlier:\n\n```bash\ndrmif\n```\n\nAfter running this command, you'll see output listing all the deleted images:\n\n```\nDeleted Images:\ndeleted: sha256:1e904ba38c6dabb0c8c9dd896954c07b5f1b1cf196364ff1de5da46d18aa9fb\ndeleted: sha256:c73b8c1cc3550886ac1cc5965f89c6c2553b08fb0c472e1a1f9106b26ee4b14\n...\n```\n\nThis helps keep your Docker environment clean and efficient.\n\n### 6.7. Tagging Docker Images\n\nProperly tagging Docker images is essential for organizing, versioning, and deploying your containerized applications, especially in data science projects where model versions are important.\n\n#### Best Practices for Tagging Images\n\n- Use semantic versioning (e.g., `v1.0.1`, `v2.1`)\n- Avoid relying on `latest` in production environments\n- Use environment-specific tags (`dev`, `staging`, `prod`)\n- Tag images before pushing to a registry\n\n#### Basic Tagging Command\n\nTo tag a Docker image, use the following syntax:\n\n```bash\ndocker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]\n```\n\n#### Examples\n\n**Simple version tagging:**\n\n```bash\n# Tag the current 'latest' image with a version number\ndocker tag my-datascience-app:latest my-datascience-app:v1.0\n```\n\n**Preparing for Docker Hub:**\n\n```bash\n# Tag for pushing to Docker Hub\ndocker tag my-datascience-app:latest username/my-datascience-app:v1.0\n\n# Then push to Docker Hub\ndocker push username/my-datascience-app:v1.0\n```\n\n**Multiple tags for different environments:**\n\n```bash\n# Create production-ready tag\ndocker tag my-ml-model:v1.2.3 my-ml-model:prod\n\n# Create development tag\ndocker tag my-ml-model:latest my-ml-model:dev\n```\n\n#### For Data Science Projects\n\nFor data science projects, consider including model information in your tags:\n\n```bash\n# Include model architecture and training data version\ndocker tag my-model:latest my-model:lstm-v2-dataset20230512\n\n# Include accuracy metrics\ndocker tag my-model:latest my-model:v1.2-acc95.4\n```\n\nProper tagging helps you maintain reproducibility and track which model version is deployed where.\n\n### 6.8. Working with Docker Volumes\n\nBy default, when a container is stopped or removed, all data inside it is lost. Docker volumes solve this problem by providing persistent storage that exists outside of containers.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"figures/volume.jpg\" alt=\"volume\" width=\"50%\"\u003e\n\u003c/div\u003e\n\n#### Why Use Volumes?\n\n- **Data Persistence**: Keep data even when containers are removed\n- **Data Sharing**: Share data between multiple containers\n- **Performance**: Better I/O performance than bind mounts, especially on Windows/Mac\n- **Isolation**: Manage container data separately from host filesystem\n\n#### Basic Volume Usage\n\n**Syntax for mounting volumes:**\n\n```bash\ndocker run -v /host/path:/container/path[:options] image_name\n```\n\n#### Examples\n\n**Example 1: Exploring a Container's Default Storage**\n\nFirst, let's see what's inside a container without volumes:\n\n```bash\n# Start an nginx container\ndocker run -d --name nginx-test -p 80:80 nginx\n\n# Enter the container\ndocker exec -it nginx-test bash\n\n# Check the content of nginx's web directory\ncd /usr/share/nginx/html\nls -la\n```\n\n**Example 2: Using a Volume for Persistence**\n\nNow let's mount a local directory to nginx's web directory:\n\n```bash\ndocker run -d -p 3000:80 -v /home/username/projects/my-website:/usr/share/nginx/html nginx\n```\n\nThis mounts your local directory `/home/username/projects/my-website` to the container's `/usr/share/nginx/html` directory. Any changes in either location will be reflected in the other.\n\n#### Security Considerations\n\nThe previous example gives full read/write access to the container. For better security, add the `:ro` (read-only) option:\n\n```bash\ndocker run -d -p 3000:80 -v /home/username/projects/my-website:/usr/share/nginx/html:ro nginx\n```\n\nThis prevents the container from modifying files in your local directory.\n\n#### Volumes in Data Science Projects\n\nFor data science projects, volumes are particularly useful for:\n\n**Persisting Jupyter notebooks and data:**\n\n```bash\ndocker run -d -p 8888:8888 -v /home/username/ds-project:/app jupyter/datascience-notebook\n```\n\n**Sharing datasets between containers:**\n\n```bash\n# Create a named volume\ndocker volume create dataset-vol\n\n# Mount the volume to multiple containers\ndocker run -d --name training -v dataset-vol:/data training-image\ndocker run -d --name inference -v dataset-vol:/data inference-image\n```\n\n**Storing model artifacts:**\n\n```bash\ndocker run -d -p 8501:8501 -v /home/username/models:/models -e MODEL_PATH=/models/my_model ml-serving-image\n```\n\n#### Volume Types in Docker\n\n1. **Named Volumes** (managed by Docker):\n\n   ```bash\n   docker volume create my-volume\n   docker run -v my-volume:/container/path image_name\n   ```\n2. **Bind Mounts** (direct mapping to host):\n\n   ```bash\n   docker run -v /absolute/host/path:/container/path image_name\n   ```\n3. **Tmpfs Mounts** (stored in host memory):\n\n   ```bash\n   docker run --tmpfs /container/path image_name\n   ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0-mostafa-rezaee-0%2Fdocker_for_data_science_projects","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0-mostafa-rezaee-0%2Fdocker_for_data_science_projects","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0-mostafa-rezaee-0%2Fdocker_for_data_science_projects/lists"}