{"id":20609788,"url":"https://github.com/brunocampos01/understanding-the-python-ecosystem","last_synced_at":"2025-04-15T04:32:17.103Z","repository":{"id":45234784,"uuid":"142902866","full_name":"brunocampos01/understanding-the-python-ecosystem","owner":"brunocampos01","description":"This project focuses on understanding the language ecosystem","archived":false,"fork":false,"pushed_at":"2023-10-07T17:29:03.000Z","size":153934,"stargazers_count":104,"open_issues_count":0,"forks_count":17,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-12T02:51:15.250Z","etag":null,"topics":["advanced-python","becoming-an-expert","code-quality","cpython","determi-build","learning-python","linter","package-manager","python","python-best-practices","python-compiler","python-configuration","python-fundamentals","python-interpreter","python-threading","requirements","virtualenv"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brunocampos01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-07-30T16:46:58.000Z","updated_at":"2024-12-24T11:09:26.000Z","dependencies_parsed_at":"2022-08-12T11:50:47.566Z","dependency_job_id":null,"html_url":"https://github.com/brunocampos01/understanding-the-python-ecosystem","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brunocampos01%2Funderstanding-the-python-ecosystem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brunocampos01%2Funderstanding-the-python-ecosystem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brunocampos01%2Funderstanding-the-python-ecosystem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brunocampos01%2Funderstanding-the-python-ecosystem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brunocampos01","download_url":"https://codeload.github.com/brunocampos01/understanding-the-python-ecosystem/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249006449,"owners_count":21197279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["advanced-python","becoming-an-expert","code-quality","cpython","determi-build","learning-python","linter","package-manager","python","python-best-practices","python-compiler","python-configuration","python-fundamentals","python-interpreter","python-threading","requirements","virtualenv"],"created_at":"2024-11-16T10:14:29.519Z","updated_at":"2025-04-15T04:32:17.078Z","avatar_url":"https://github.com/brunocampos01.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Understanding the Python Ecosystem \u003cimg src=\"images/python_logo.png\" width=\"10%\" height=\"50%\" align=\"right\" valign=\"center\"/\u003e \n\n![License](https://img.shields.io/badge/Code%20License-MIT-blue.svg)\n\u003c!-- [![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fbrunocampos01%2Funderstanding-the-python-ecosystem\u0026count_bg=%233D81C8\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=Views\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n --\u003e\nThis project focuses on understanding the language ecosystem, not getting into programming details.\n\n# Summary\n\n:sunrise_over_mountains: **_Python's Habitat_**\n\nThis topic describes how to set up the environment for Python development.\n- [Preparing the Environment for the Python](#preparing-the-environment-for-the-python)\n- [Check Python Configuration](#check-python-configuration)\n- [Advanced settings of Python](#advanced-settings-of-python)\n- [What is a virtual environment and how it works](#What-is-a-virtual-environment-and-how-it-works)\n\n\u003cbr/\u003e\n\n:snake: **_Python's Taxonomy_**\n\nThis topic describes features of the pattern of Python projects.\n- [Package manager](#package-manager)\n- [Requirements file](#requirements-file)\n- [Deterministic build](#deterministic-build)\n\u003c!-- - Principals Files --\u003e\n\n\u003cbr/\u003e\n\n:anger: **_Python's Behavior_**\n\nThis topic describes how the language is designed and how it works.\n- [Compiler and interpreter](#compiler-and-interpreter)\n- [How Python runs a program](#how-python-runs-a-program)\n- [How Python search path module](#how-python-search-path-module)\n- [How Python manages process and threads](#how-python-manages-process-and-threads)\n- [How Python manages memory](#how-python-manages-memory)\n- [How to deeply understand Python code execution (debug)](#how-to-deeply-understand-python-code-execution)\n\n\u003c!-- \n- threads\n- process\n- Concurrency async\n- Debug\n--\u003e\n\n\u003cbr/\u003e\n\n:bug: **_Python's Feeding_**\n\nThis topic describes static code analysis, formatting patterns and style guides.\n- [Static code analysis](#static-code-analysis)\n- [Principal style guides](#principal-style-guides)\n- [Knobs (Google YAPF)](https://github.com/google/yapf#id11)\n- [My Knobs](#my-knobs)\n- [Docstrings](#docstrings)\n\n\u003c!-- TODO - https://realpython.com/documenting-python-code/\nTODO - https://docs.python-guide.org/ --\u003e\n\n\n\u003cbr/\u003e\n\n:mag: **_Python's Other Features_**\n\nExtra topics.\n- [Awesome Python](https://github.com/vinta/awesome-python) \n- [Why Is CPython Written in C and Not Python?](https://realpython.com/cpython-source-code-guide/#why-is-cpython-written-in-c-and-not-python)\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **Preparing the Environment for the Python**\n\n\u003cdetails\u003e\n    \u003csummary\u003e\u003cb\u003e\u003ca href=\"#preparing-the-environment-for-the-python\"\u003e\u003cimg src=\"images/icon_ubuntu.png\"/\u003e\u003c/a\u003e Linux\u003c/b\u003e\u003c/summary\u003e\n  \n  Python needs a set of tools that are system requirements. If necessary, install these requirements with this command:\n  ```bash\n  sudo apt update\n\n  sudo apt install\\\n    software-properties-common\\\n    build-essential\\\n    libffi-dev\\\n    python3-pip\\\n    python3-dev\\\n    python3-venv\\\n    python3-setuptools\\\n    python3-pkg-resources\n  ```\n  \n  Now, the environment is done to install Python\n  ```bash\n  sudo apt install python\n  ```\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e  \u003ca href=\"#preparing-the-environment-for-the-python\"\u003e\u003cimg src=\"images/icon_windows.png\"/\u003e\u003c/a\u003e Windows\u003c/b\u003e\u003c/summary\u003e\n\n  On Windows, I recommend using the package manager [chocolatey](https://chocolatey.org/) and setting your Powershell to work as admin. See [this](devpos/infra-as-code) tutorial.\n\n  Now, install Python\n  ```powershell\n  choco install python \n  ```\n  \n  \u003cimg src='images/windows_python_4.png' height=auto width=\"100%\"\u003e\n  \u003cimg src='images/windows_python_5.png' height=auto width=\"100%\"\u003e\n  \n  \u003cbr/\u003e\n  \n  Test\n  ```powershell\n  python --version \n  ```\n  \n  \u003cimg src='images/windows_python_6.png' height=100% width=\"100%\"\u003e\n\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **Check Python Configuration**\n### Check **current version**\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e Watch\u003c/summary\u003e\n  \u003cimg src='images/version_python.gif' height=auto width=\"100%\"\u003e\n\u003c/details\u003e\n\n```bash\npython --version\n```\n\n### Check **where** installed Python\n\u003cdetails\u003e\t\n  \u003csummary\u003e Watch\u003c/summary\u003e\n  \u003cimg src='images/which_python.gif' height=auto width=\"100%\"\u003e\n\u003c/details\u003e\n\n```bash\nwhich python\n```\n\n### Check **which Python versions** are installed\n\u003cdetails\u003e\t\n  \u003csummary\u003e Watch\u003c/summary\u003e\n  \u003cimg src='images/list_versions.gif' height=auto width=\"100%\"\u003e\n\u003c/details\u003e\n  \n```bash\nsudo update-alternatives --list python\n```\n\n---\n\n\u003cbr/\u003e\n\n## **Advanced settings of Python**\n\n\u003cdetails\u003e\t\n   \u003csummary\u003e\u003cb\u003e Install multiples Python versions\u003c/b\u003e\u003c/summary\u003e\n  \u003c!-- ### **Install multiples Python versions** --\u003e\n  Sometimes you might work on different projects simultaneously with different versions of Python. Normally using Anaconda is the easiest solution, however, there are restrictions.\n  \n  1. Add repository\n     \u003cdetails\u003e\t\n       \u003csummary\u003e Watch\u003c/summary\u003e\n       \u003cimg src='images/install_python.gif' height=auto width=\"100%\"\u003e\n     \u003c/details\u003e\n\n     This PPA contains more recent Python versions packaged for Ubuntu.\n     ```bash\n     sudo add-apt-repository ppa:deadsnakes/ppa -y\n     ```\n\n  2. Update packages\n     ```bash\n     sudo apt update -y\n     ```\n     \n  3. Check which Python version is installed\n     ```bash\n     python --version\n     ```\n     \n  4. Install Python\n     ```bash\n     sudo apt install python3.\u003cVERSION\u003e\n     ```\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n   \u003csummary\u003e\u003cb\u003e Install multiples Python versions Using Pyenv\u003c/b\u003e\u003c/summary\u003e\n  \u003c!-- ### **Install multiples Python versions** --\u003e\n  \n  1. Add dependencies\n     ```bash\n     sudo apt install curl -y\n     ```\n  \n  2. Update packages\n     ```bash\n     sudo apt update -y\n     ```\n\n  3. Install pyenv\n     ```bash\n     curl https://pyenv.run | bash\n     ```\n\n  4. Add these three lines from .bashrc or .zhshrc\n     ```bash\n     export PATH=\"$HOME/.pyenv/bin:$PATH\"\n     eval \"$(pyenv init --path)\"\n     eval \"$(pyenv virtualenv-init -)\"\n     ```\n\n  5. Open a new terminal and execute\n     ```bash\n     exec $SHELL\n     pyenv --version\n     ```\n     \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n   \u003csummary\u003e\u003cb\u003e Change system's Python\u003c/b\u003e\u003c/summary\u003e\n  \n  Before installing other versions of Python it's necessary to set which system's Python will be used.\n  \n  1. Use `update-alternatives`\n  \n     It's possible use the `update-alternatives` command to set priority to different versions of the same software installed in Ubuntu systems. Now, define priority of versions:\n     \n     ```bash\n     sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1\n     \n     sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.10 2\n      \n     sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.8 3\n  \n     sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 4\n     ```\n  \n     In directory `/usr/bin` will be create simbolic link: `/usr/bin/python -\u003e /etc/alternatives/python*`\n  \n  2. Choose version\n  \n     \u003cdetails\u003e\t\n       \u003csummary\u003e Watch\u003c/summary\u003e\n       \u003cimg src='images/change_python.gif' height=auto width=\"100%\"\u003e\n     \u003c/details\u003e\n  \n     ```bash\n     sudo update-alternatives --config python\n     ```\n  \n  3. Test\n     ```bash\n     python --version\n     ```\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Change Python2 to Python3\u003c/b\u003e\u003c/summary\u003e\n  \n  If return Python **2**, try set a alias in `/home/$USER/.bashrc`, see this [example](https://github.com/brunocampos01/home-sweet-home/blob/master/config/.bashrc).\n  \n  ```bash\n  alias python=python3\n  ```\n  \n  **NOTE:**\n  The important thing to realize is that Python 3 is not backwards compatible with Python 2. This means that if you try to run Python 2 code as Python 3, it will probably break.\n\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n  \n  \u003cdetails\u003e\n    \u003csummary\u003e\u003cb\u003e Set Python's Environment Variables\u003c/b\u003e\u003c/summary\u003e\n  \n  \u003c!-- ### **Set Python's Environment Variables** --\u003e\n  - `PYTHONPATH` is an environment variable which you can set to add additional directories where python will look for modules and packages. Example: [Apache Airflow](https://airflow.apache.org/) read `dag/` folder and add automatically any file that is in this directory. \n  - To interpreter `PYTHONHOME` indicate standard packages.\n  \n  \u003cbr/\u003e\n  \n   \u003cdetails\u003e\n     \u003csummary\u003e\u003cb\u003e    Set PYTHONPATH\u003c/b\u003e\u003c/summary\u003e\n    \n   1. Open profile\n      ```bash\n      sudo vim ~/.bashrc\n      ```\n    \n   2. Insert Python PATH\n      ```bash\n      export PYTHONHOME=/usr/bin/python\u003cNUMER_VERSION\u003e\n      ```\n    \n   3. Update profile/bashrc\n      ```bash\n      source ~/.bashrc\n      ```\n    \n   4. Test\n      ```bash\n      \u003e\u003e\u003e import sys\n      \u003e\u003e\u003e from pprint import pprint\n      \u003e\u003e\u003e pprint(sys.path)\n      ['',\n       '/usr/lib/python311.zip',\n       '/usr/lib/python3.11',\n       '/usr/lib/python3.11/lib-dynload',\n       '/usr/local/lib/python3.11/dist-packages',\n       '/usr/lib/python3/dist-packages']\n      ```\n      \n      Example with Apache Airflow\n      ```bash\n      \u003e\u003e\u003e import sys\n      \u003e\u003e\u003e from pprint import pprint\n      \u003e\u003e\u003e pprint(sys.path)\n      ['',\n       '/home/project_name/dags',\n       '/home/project_name/config',\n       '/home/project_name/utilities',\n       ...\n       ]\n      ```\n   \u003c/details\u003e\n   \u003cbr/\u003e\n\u003c/details\u003e\n\n\n---\n\n\u003cbr/\u003e\n\n## **What is a virtual environment and how it works**\nPython can run in a virtual environment with **isolation** from the system. \n\n\u003cimg src=\"images/virtualenv.png\"  align=\"center\" height=auto width=60%/\u003e\n\n###### Image source: https://vincenttechblog.com/fix-change-python-virtualenv-settings/\n\n\u003cbr/\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Arquitecture of Execution\u003c/b\u003e\u003c/summary\u003e\n\n  \u003cimg src=\"images/org_python.jpg\"  align=\"center\" height=auto width=70%/\u003e\n  \n  \u003cbr/\u003e\n  \n  Virtualenv enables us to create multiple Python environments which are isolated from the global Python environment as well as from each other.\n  \n  \u003cimg src=\"images/org_python_virtualenv.jpg\" align=\"center\" height=auto width=70%/\u003e\n  \n  \u003cbr/\u003e\n  \n  When Python is initiating, it analyzes the path of its binary. In a virtual environment, it's actually just a copy or Symbolic link to your system's Python binary. Next, set the `sys.prefix` location which is used to locate the `site-packages` (third party packages/libraries)\n  \n  \n  \u003cimg src=\"images/virtualenv.jpg\" align=\"middle\" height=auto width=50%/\u003e\n  \n  \u003cbr/\u003e\n  \n  #### **Symbolic link**\n  - `sys.prefix` points to the virtual environment directory.\n  - `sys.base.prefix` points to the **non-virtual** environment.\n  \n  #### **Folder of virtual environment**\n  ```bash\n  ll\n  \n  # random.py -\u003e /usr/lib/python3.6/random.py\n  # reprlib.py -\u003e /usr/lib/python3.6/reprlib.py\n  # re.py -\u003e /usr/lib/python3.6/re.py\n  # ...\n  ```\n  \n  ```bash\n  tree\n  \n  ├── bin\n  │   ├── activate\n  │   ├── activate.csh\n  │   ├── activate.fish\n  │   ├── easy_install\n  │   ├── easy_install-3.8\n  │   ├── pip\n  │   ├── pip3\n  │   ├── pip3.8\n  │   ├── python -\u003e python3.8\n  │   ├── python3 -\u003e python3.8\n  │   └── python3.8 -\u003e /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8\n  ├── include\n  ├── lib\n  │   └── python3.8\n  │       └── site-packages\n  └── pyvenv.cfg\n  ```\n  \n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Create Virtual Environment\u003c/b\u003e\u003c/summary\u003e  \n  \u003cdetails\u003e\t\n      \u003csummary\u003e Watch\u003c/summary\u003e\n      \u003cimg src='images/create_virtualenv.gif' height=auto width=\"100%\"\u003e\n  \u003c/details\u003e\n    \n  Create virtual environment\n  ```bash\n  virtualenv -p python3  \u003cNAME_ENVIRONMENT\u003e\n  ```\n    \n  Activate \n  ```bash\n  source \u003cNAME_ENVIRONMENT\u003e/bin/activate\n  ```\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **Package manager**\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Pipenv\u003c/b\u003e\u003c/summary\u003e \n  \n  Create and manage automatically a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages. It also generates the ever-important `Pipfile.lock`, which is used to produce deterministic builds.\n  \n  #### **Features**\n  - Deterministic builds\n  - Separates development and production environment packages into a single file `Pipefile`\n  - Automatically adds/removes packages from your `Pipfile`\n  - Automatically create and manage a virtualenv\n  - Check PEP 508 requirements\n  - Check installed package safety\n  \n  #### **Pipfile X requirements**\n  ```bash\n  # Pipfile\n  \n  [[source]]\n  name = \"pypi\"\n  url = \"https://pypi.org/simple\"\n  verify_ssl = true\n  \n  [dev-packages]\n  \n  [packages]\n  requests = \"*\"\n  numpy = \"==1.18.1\"\n  pandas = \"==1.0.1\"\n  wget = \"==3.2\"\n  \n  [requires]\n  python_version = \"3.8\"\n  platform_system = 'Linux'\n  ```\n  \n  ```bash\n  # requirements.txt\n  \n  requests\n  matplotlib==3.1.3\n  numpy==1.18.1\n  pandas==1.0.1\n  wget==3.2\n  ```\n  \n  \u003cbr/\u003e\n  \n  ### **Install**\n  ```bash\n  pip3 install --user pipenv\n  ```\n  \n  \u003cbr/\u003e\n\n  ### Create Pipfile and virtual environment\n  1. Create environment\n     \u003cdetails\u003e\t\n       \u003csummary\u003e Watch\u003c/summary\u003e\n       \u003cimg src='images/pipenv.gif' height=auto width=\"100%\"\u003e\n     \u003c/details\u003e\n  \n     ```bash\n     pipenv --python 3\n     ```\n  \n  2. See **where** virtual environment is installed\n     ```bash\n     pipenv --venv\n     ```\n  \n  3. Activate environment\n     ```bash\n     pipenv run\n     ```\n  \n  4. Install packages with Pipefile\n     ```bash\n     pipenv install flask\n     # or\n     pipenv install --dev flask\n     ```\n  \n  5. Create lock file\n     \u003cdetails\u003e\t\n       \u003csummary\u003e Watch\u003c/summary\u003e\n       \u003cimg src='images/pipenv_lock.gif' height=auto width=\"100%\"\u003e\n     \u003c/details\u003e\n   \n     ```bash\n     pipenv lock\n     ```\n     \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Python Package Index\u003c/b\u003e\u003c/summary\u003e \n   \n   [Doc Python Package Index](https://pypi.org/)\n   \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Poetry\u003c/b\u003e\u003c/summary\u003e \n    \n   [Doc Poetry](https://python-poetry.org/)\n   \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Conda\u003c/b\u003e\u003c/summary\u003e \n  \n  [Doc Conda](https://docs.conda.io/en/latest/)\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **Requirements File**\n`Requirements.txt` is file containing a list of items to be installed using pip install.\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Principal Comands\u003c/b\u003e\u003c/summary\u003e\n\n  1. Visualize instaled packages\n  ```bash\n  pip3 freeze\n  ```\n  \n  2. Generate file `requirements.txt`\n  ```bash\n  pip3 freeze \u003e requirements.txt\n  ```\n  \n  3. Test \n  ```bash\n  cat requirements.txt\n  ```\n  \n  4. Install packages in requirements\n  ```bash\n  pip3 install -r requirements.txt\n  ```\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **Deterministic Build**\n\u003c!-- ### **The issue with Pip** --\u003e\nUsing pip and `requirements.txt` file, have a **real issue here is that the build isn’t [deterministic](https://pt.wikipedia.org/wiki/Algoritmo_determin%C3%ADstico)**. What I mean by that is, given the same input (the `requirements.txt` file), pip does not always produce the same environment.\n\n\n### **pip-tools**\nA set of command line tools to help you keep your pip-based packages fresh and ensure the deterministic build.\n\n#### **Features**\n- Distinguish direct dependencies and versions\n- Freeze a set of exact packages and versions that we know work\n- Make it reasonably easy to update packages\n- Take advantage of pip's hash checking to give a little more confidence that packages haven't been modified (DNS attack)\n- Stable\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Principal Comands\u003c/b\u003e\u003c/summary\u003e\n\n  1. Install\n  ```\n  pip install pip-tools\n  ```\n  \n  2. Get packages's version\n  ```bash\n  pip3 freeze \u003e requirements.in\n  ```\n  \n  3. Generate hashes and list dependeces\n  ```bash\n  pip-compile --generate-hashes requirements.in\n  ```\n  output: [requirements.txt](requirements.txt)\n  \n  4. Install packages and hash checking\n  ```bash\n  pip-compile --generate-hashes requirements.in\n  ```\n  \u003cbr/\u003e\n\n\u003c/details\u003e\t\n\n---\n\n\u003cbr/\u003e\n\n## **Compiler and interpreter**\nCPython can be defined as both an interpreter and a compiler.\n- The **compiler** converts the `.py` source file into a `.pyc` bytecode for the Python virtual machine.\n- The **interpreter** executes this bytecode on the virtual machine.\n\n\u003cimg src=\"images/interpreter.png\"  align=\"center\" height=auto width=80%/\u003e\n\n\u003cbr/\u003e\n\n\n\u003c!-- \u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e CPython's Design\u003c/b\u003e\u003c/summary\u003e --\u003e\n\n### **CPython's Design**\nThe principal feature of [CPython](https://en.wikipedia.org/wiki/CPython#:~:text=8%20External%20links-,Design,bytecode%20at%20any%20one%20time.), is that it makes use of a global interpreter lock (GIL). This is a mechanism used in computer-language interpreters to synchronize the execution of threads so that only one native thread can execute at a time. \n\u003cbr/\u003e\nTherefore, for a CPU-bound task in Python, single-process multi-thread Python program would not improve the performance. However, this does not mean multi-thread is useless in Python. For a I/O-bound task in Python, multi-thread could be used to improve the program performance.\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Multithreading in Python\u003c/b\u003e\u003c/summary\u003e\n  The Python has multithreads despite the GIL. Using Python threading, we are able to make better use of the CPU sitting idle when waiting for the I/O bound, how memory I/O, hard drive I/O, network I/O.\n  \n  \u003cimg src=\"images/multithread.png\"  align=\"center\" height=auto width=80%/\u003e\n  \n  \u003cbr/\u003e\n  \n  This can happen when multiple threads are servicing separate clients. One thread may be waiting for a client to reply, and another may be waiting for a database query to execute, while the third thread is actually processing Python code or other example is read multiples images from disk.\n  \n  **NOTE:** we would have to be careful and use locks when necessary. Lock and unlock make sure that only one thread could write to memory at one time, but this will also introduce some overhead. \n  \n  \u003c!--\n  ver mais sobre thread and process\n  https://leimao.github.io/blog/Python-Concurrency-High-Level/ --\u003e\n  \n   \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Community Consensus\u003c/b\u003e\u003c/summary\u003e\n\n  Removing the GIL would have made **Python 3 slower in comparison to Python 2** in single-threaded performance. Other problem if remove the GIL it's would **broke the existing C extensions** which depend heavily on the solution that the GIL provides.\n  \u003cbr/\u003e\n  Although many proposals have been made to eliminate the GIL, the general consensus has been that in most cases, the advantages of the GIL outweigh the disadvantages; in the few cases where the GIL is a bottleneck, the application should be built around the multiprocessing structure.\n  \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\n---\n\n\u003cbr/\u003e\n\n## **How Python runs a program**\n1. Tokenize the source code: `Parser/tokenizer.c`\n2. Parse the stream of tokens into an Abstract Syntax Tree (AST): `Parser/parser.c`\n3. Transform AST into a Control Flow Graph: `Python/compile.c`\n4. Emit bytecode based on the Control Flow Graph: `Python/compile.c`\n\n\n---\n\n\u003cbr/\u003e\n\n## **How Python search path module**\nWhen Python executes this statement:\n```python\nimport my_lib\n```\nThe interpreter searches `my_lib.py` a list of directories\nassembled from the following sources: \u003c!-- montada a partir das seguintes fontes: --\u003e\n- Current directory\n- The list of directories contained in the **PYTHONPATH** environment variable\n- In directory which Python was is installed. E.g. \n  \u003cimg src=\"images/show_path_lib.png\"  align=\"center\" height=auto width=80%/\u003e\n\n\nThe resulting search can be accessed using the **sys** module:\n```python\nimport sys\n\nsys.paths\n# ['', '/usr/lib/python38.zip', \n# '/usr/lib/python3.8',\n# '/usr/lib/python3.8/lib-dynload',\n# '/home/campos/.local/lib/#python3.8/site-packages',\n# '/usr/local/lib/python3.8/dist-packages',\n# '/usr/lib/python3/dist-packages']\n```\n\n\u003cbr/\u003e\n\nNow, to see where a packeage was imported from you can use the attribute `__file__`:\n```python\nimport zipp\n\nzipp.__file__\n# '/usr/lib/python3/dist-packages/zipp.py'\n```\n\n\u003e NOTE: you can see that the `__file__` directory is in the list of directories searched by the interpreter.\n\n\u003c!-- TODO\n- how check what was imported ?\n  - `dir()` function\n- package initiazion\n  - __init__\n  - __all__\nhttps://realpython.com/python-modules-packages/#package-initialization --\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **How Python manages process and threads**\n\nTODO\n\n\u003c!-- \nhttps://realpython.com/python-gil/\nhttps://realpython.com/courses/speed-python-concurrency/\nhttps://realpython.com/intro-to-python-threading/\nhttps://data-flair.training/blogs/python-multithreading/\nhttps://sobolevn.me/2020/06/how-async-should-have-been\n--\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **How Python manages memory**\n\nTODO\n\n\u003c!-- https://realpython.com/courses/how-python-manages-memory/ --\u003e\n\n\n---\n\n\u003cbr/\u003e\n\n## **How to deeply understand Python code execution**\n\n\nTODO\n\n\u003c!-- trace via Pycharm\ntrace via VS CODE\ntrace memory allocations: https://docs.python.org/3/library/tracemalloc.html#module-tracemalloc\ncode coverage: https://pypi.org/project/coverage/\nPySnooper: https://github.com/cool-RR/PySnooper --\u003e\n\n\n---\n\n\u003cbr/\u003e\n\n## **Static code analysis**\nThe static code analysis serves to **evaluate the coding**. This analysis must be done before submitting for a code review. The static code analysis can check:\n- Code styling analysis\n- Comment styling analysis\n- Error detection\n- Duplicate code detection\n- Unused code detection\n- Complexity analysis\n- Security linting\n\nThe characteristics of a static analysis are:\n- Provides insight into code without executing it\n- Can automate code quality maintenance\n- Can automate the search for bugs at the early stages\n- Can automate the finding of security problems\n\nA lint, is a [static code analysis tool](https://en.wikipedia.org/wiki/Lint_(software)).\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Pylint\u003c/b\u003e\u003c/summary\u003e\n\n  [Pylint](https://pylint.org/) is a lint that checks for errors in Python code, tries to enforce a coding standard and looks for code smells. The principal features is:\n  - Pylint follow the [PEP8](https://www.python.org/dev/peps/pep-0008/) style guide.\n  - It's possible automate with Jenkins.\n  - It is fully customizable through a `.pylintrc` file where you can choose which errors or agreements are relevant to you.\n  - Usage\n    ```bash\n    # Get Errors \u0026 Warnings\n    pylint -rn \u003cfile/dir\u003e --rcfile=\u003c.pylintrc\u003e\n\n    # Get Full Report\n    pylint \u003cfile/dir\u003e --rcfile=\u003c.pylintrc\u003e\n    ```  \n  \u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Example execution\u003c/b\u003e\u003c/summary\u003e\n    \u003cimg src=\"images/pylint_example.gif\"  align=\"center\" height=auto width=100%/\u003e\n  \u003cbr/\u003e\n  \u003c/details\u003e\n  \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Pyflakes\u003c/b\u003e\u003c/summary\u003e\n\n  [Documentation Pyflakes](https://github.com/PyCQA/pyflakes)\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Mypy\u003c/b\u003e\u003c/summary\u003e\n\n  [Documentation Mypy](http://mypy-lang.org/)\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Prospector\u003c/b\u003e\u003c/summary\u003e\n\n  [Documentation Propector](https://prospector.landscape.io/en/master/) \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n---\n\n\u003cbr/\u003e\n\n## Other Tools to make an effective Python style guide\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Isort\u003c/b\u003e\u003c/summary\u003e\n\n  [isort](https://pypi.org/project/isort/) is a Python tool/library for sorting imports alphabetically, automatically divided into sections. It is very useful in projects where we deal with a lot of imports [6].\n  ```bash\n  # sort the whole project\n  isort --recursive ./src/\n  \n  # just check for errors\n  isort script.py --check-only\n  ``` \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Unify\u003c/b\u003e\u003c/summary\u003e\n\n  Someone likes to write them in single quotes, someone in double ones. To unify the whole project, there is a tool that allows you to automatically align with your style guide — [unify](https://pypi.python.org/pypi/unify) [6].\n  ```bash\n  unify --in-place -r ./src/\n  ```\n  Work recursively for files in the folder.\n  \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e docformatter\u003c/b\u003e\u003c/summary\u003e\n\n  [Docformater](https://pypi.org/project/docformatter/) is utility helps to bring your docstring under the [PEP 257](https://www.python.org/dev/peps/pep-0257/) [6]. The agreement specifies how documentation should be written.\n  ```bash\n  docformatter --in-place example.py\n  ```\n  \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Autoformatters\u003c/b\u003e\u003c/summary\u003e\n\n  There are also automatic code formatters now, here are the popular one [6]:\n  - [yapf](https://github.com/google/yapf) (here you can make a align with your own style guide)\n  - [black](https://github.com/psf/black) (you don't need a style guide because you don't have a choice)\n  - [autopep8](https://github.com/hhatto/autopep8) (makes your python script to conform PEP8 style guide)\n  \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e Settings files to text editor and IDE\u003c/b\u003e\u003c/summary\u003e\n\n  - EditorConfig\n  - Gitattributes\n \n \u003cbr/\u003e\n\u003c/details\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **Principal style guides**\nTo make the code consistent and make sure it's readable the _style guides_ can help.\n- [Google style guide: THE BETTER](https://google.github.io/styleguide/pyguide.html) \n- [Real Python: Naming Conventions](https://realpython.com/python-pep8/#naming-conventions)\n- [PEP 08: Style Guide](https://www.python.org/dev/peps/pep-0008/)\n- [PEP 257: Docstrings](https://www.python.org/dev/peps/pep-0257/)\n- [PEP 484: Type Hints](https://www.python.org/dev/peps/pep-0484/)\n\n---\n\n\u003cbr/\u003e\n\n## **My Knobs**\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Identation and Length\u003c/b\u003e\u003c/summary\u003e \n  \n  - 4 spaces\n  - Limit all lines to a maximum **72 characteres to docstring or comments**\n  - Limit all lines to a maximum **79 characteres to code**  \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Naming Convention\u003c/b\u003e\u003c/summary\u003e \n\n  - Class Name (camelCase): `CapWords()`\n  - Variables (snack_case): `cat_words`\n  - Constants: `MAX_OVERFLOW`\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Exception\u003c/b\u003e\u003c/summary\u003e \n  \n  Limit the clausule `try:` minimal code necessary.\n  \n  Yes:\n  ```Python\n  try:\n      value = collection[key]\n  except KeyError:\n      return key_not_found(key)\n  else:\n      return handle_value(value)\n  ```\n  \n  No:\n  ```Python\n  try:\n      # Too broad!\n      return handle_value(collection[key])\n  except KeyError:\n      # Will also catch KeyError raised by handle_value()\n      return key_not_found(key)\n  ```\n  \n  - The goal to answer the question **\"What went wrong?\"** programmatically rather than just claiming that _\"There was a problem\"_\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Return\u003c/b\u003e\u003c/summary\u003e \n  \n  \"_Should explicitly state this as return None_\"\n  \n  - Be consistent in return statements.\n  - Todas as instruções de retorno em uma função devem retornar uma expressão ou nenhuma delas deve.\n  \n  Yes:\n  ```Python\n  def foo(x):\n      if x \u003e= 0:\n          return math.sqrt(x)\n      else:\n          return None\n  ```\n  \n  No:\n  ```Python\n  def foo(x):\n      if x \u003e= 0:\n          return math.sqrt(x)\n  ```\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n---\n\n\u003cbr/\u003e\n\n## **Docstrings**\n\nDocstrings must have:\n- Args\n- Returns\n- Raises\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Example Google Style Guide\u003c/b\u003e\u003c/summary\u003e \n\n  ```Python\n  def fetch_bigtable_rows(big_table, keys, other_silly_variable=None):\n      \"\"\"Fetches rows from a Bigtable.\n  \n      Retrieves rows pertaining to the given keys from the Table instance\n      represented by big_table.  Silly things may happen if\n      other_silly_variable is not None.\n  \n      Args:\n          big_table: An open Bigtable Table instance.\n          keys: A sequence of strings representing the key of each table row\n              to fetch.\n          other_silly_variable: Another optional variable, that has a much\n              longer name than the other args, and which does nothing.\n  \n      Returns:\n          A dict mapping keys to the corresponding table row data\n          fetched. Each row is represented as a tuple of strings. For\n          example:\n  \n          {'Serak': ('Rigel VII', 'Preparer'),\n           'Zim': ('Irk', 'Invader'),\n           'Lrrr': ('Omicron Persei 8', 'Emperor')}\n  \n          If a key from the keys argument is missing from the dictionary,\n          then that row was not found in the table.\n  \n      Raises:\n          IOError: An error occurred accessing the bigtable.Table object.\n      \"\"\"\n      return None\n  ``` \n  \u003cbr/\u003e\n\u003c/details\u003e\n\n\n\u003c!-- TODO\n- Side effects: https://realpython.com/defining-your-own-python-function/#side-effects\n- return: https://realpython.com/defining-your-own-python-function/#exiting-a-function --\u003e\n\n---\n\n\u003cbr/\u003e\n\n\u003cdetails\u003e\t\n  \u003csummary\u003e\u003cb\u003e References\u003c/b\u003e\u003c/summary\u003e\n  \n  - 1: [Python 3 Installation \u0026 Setup Guide](https://realpython.com/installing-python/)\n  - 2: [An Effective Python Environment: Making Yourself at Home](https://realpython.com/effective-python-environment/)\n  - 3: [Import Scripts as Modules](https://realpython.com/python-import/#import-scripts-as-modules)\n  - 4: [Python Modules and Packages – An Introduction](https://realpython.com/python-modules-packages/)\n  - 5: [What Is the Python Global Interpreter Lock (GIL)?](https://realpython.com/python-gil/)\n  - 6: [7 tips to make an effective python style guide](https://luminousmen.com/post/7-tips-to-make-an-effective-python-style-guide)\n  - 7: [Python Static Analysis Tools](https://luminousmen.com/post/python-static-analysis-tools)\n  - 8: [My unpopular opinion about black code formatter](https://luminousmen.com/post/my-unpopular-opinion-about-black-code-formatter)\n  \u003cbr/\u003e\n\u003c/details\u003e\n\n---\n\n\u003cp  align=\"left\"\u003e\n\t\u003cbr/\u003e\n\t\u003ca href=\"mailto:brunocampos01@gmail.com\" target=\"_blank\"\u003e\u003cimg src=\"https://github.com/brunocampos01/brunocampos01/blob/main/images/email.png\" width=\"30\"\u003e\n\t\u003c/a\u003e\n\t\u003ca href=\"https://stackoverflow.com/users/8329698/bruno-campos\" target=\"_blank\"\u003e\u003cimg src=\"https://github.com/brunocampos01/brunocampos01/blob/main/images/stackoverflow.png\" width=\"30\"\u003e\n\t\u003c/a\u003e\n\t\u003ca href=\"https://www.linkedin.com/in/brunocampos01\" target=\"_blank\"\u003e\u003cimg src=\"https://github.com/brunocampos01/brunocampos01/blob/main/images/linkedin.png\" width=\"30\"\u003e\n\t\u003c/a\u003e\n\t\u003ca href=\"https://github.com/brunocampos01\" target=\"_blank\"\u003e\u003cimg src=\"https://github.com/brunocampos01/brunocampos01/blob/main/images/github.png\" width=\"30\"\u003e\u003c/a\u003e\n\t\u003ca href=\"https://medium.com/@brunocampos01\" target=\"_blank\"\u003e\u003cimg src=\"https://github.com/brunocampos01/brunocampos01/blob/main/images/medium.png\" width=\"30\"\u003e\n\t\u003c/a\u003e\n    \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-sa/4.0/\"\u003e\u003cimg alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-sa/4.0/88x31.png\",  align=\"right\" /\u003e\n    \u003c/a\u003e\n    \u003cbr/\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrunocampos01%2Funderstanding-the-python-ecosystem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrunocampos01%2Funderstanding-the-python-ecosystem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrunocampos01%2Funderstanding-the-python-ecosystem/lists"}