awesome-python-backend

Index for online reading materials in order to learn Python and backend development/engineering concepts from scratch and develop a mastery sufficient for Senior/Principal Backend Engineers and Data Engineers
https://github.com/Eldar1205/awesome-python-backend

Last synced: 7 days ago
JSON representation

Topics Index
- QoL Libraries
  - https://github.com/vinta/awesome-python
  - type annotations - for cases where static type checks aren't sufficient, e.g. you want to assert type annotations not statically checked, or you are developing a Python library (its users might not perform static type checks when calling your library's classes/functions/etc.) - [_beartype_](https://github.com/beartype/beartype)
  - _asyncstdlib_ - thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#cleanup-in-generators-and-async-generators) for details.
  - _aiomisc_
  - Decorator support
  - **PeriodicCallback** - callback)
  - dog-piling - [_py-memoize_](https://memoize.readthedocs.io/en/latest/)
  - _bidict_
  - _toolz_ - itertools_](https://more-itertools.readthedocs.io/en/stable/api.html)
  - _classes_
  - _immutables_
  - _sortedcontainers_
  - _uvloop_
  - _pampy_
  - _pipe_ - clean-python-code-using-pipes-1239a0f3abf5)
  - _object-mapper_
  - _python-varname_
  - _py-linq_
  - _result_
- Distributed programming frameworks
  - _Kubernetes_
  - _Airflow_
  - _Great Expectations_
  - there are workarounds
  - Useful production ready deployment reference
  - _Faust_ - a microservices framework which provides abstractions for stream processing with stateful agents that process infinite streams of messages built on top of [_Kafka_](https://kafka.apache.org/) for pub-sub and queue based messaging (extensible), [_RocksDB_](https://rocksdb.org/) as local persistent tables store and (optionally) [_Redis_](https://redis.io/) as distributed cache. Co-founded by [_Celery_](https://docs.celeryproject.org/en/latest/index.html) founder.
  - leader election
  - _Airflow_ - A workflow framework for orchestrating distributed scheduled/triggered workflows (a.k.a DAGs) described using Python scripts such that workers can execute tasks (a.k.a operators) in workflows. Built on top of SQL databases for workflow state persistence and can execute on a cluster directly using [_Celery_](https://docs.celeryproject.org/en/latest/getting-started/)or [_Dask_](https://dask.org/), on a [_Kubernetes_](https://kubernetes.io/) cluster, or managed in [AWS](https://aws.amazon.com/managed-workflows-for-apache-airflow/)/[GCP](https://cloud.google.com/composer). Comes with several core operators and there's a great deal of [independent ones](https://airflow.apache.org/docs/apache-airflow-providers/operators-and-hooks-ref/index.html) and a [registry of built-in & community provided ones](https://registry.astronomer.io/).
  - Python code - branch-operator), [Run DAG](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/operators/trigger_dagrun/index.html#module-airflow.operators.trigger_dagrun), [HTTP request](https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html#howto-operator-simplehttpoperator), [SQL checks](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/operators/sql/index.html#module-airflow.operators.sql), etc.
  - Python code - airflow/stable/_api/airflow/sensors/sql/index.html#module-airflow.sensors.sql), [Time Wait](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/sensors/time_delta/index.html#module-airflow.sensors.time_delta), etc.
  - _Jinja_
  - the Linux subsystem can be used
  - _Ray_ - A workers & actors framework for implementing distributed, fault tolerant and scalable applications specialized for data science but usable for any distributed computing. Workers are stateless, actors are stateful, both are Python processes and therefore heavyweight and coarse grained. [Ray has an academic whitepaper](https://www.usenix.org/system/files/osdi18-moritz.pdf).
  - _Thespian_
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - _gevent_ - blocking I/O
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - _useful reference_
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - the Linux subsystem can be used
  - _useful reference_
  - Useful production ready deployment reference
- Commonly Useful Knowledge
  - _SQLAlchemy_
  - NumPy - ** a Python library used for working with arrays
  - Scipy - a scientific computation library that uses NumPy underneath
  - _Django_ - a very extensive framework with many many features, essentially an ecosystem, well documented, [limited explicit async IO support](https://docs.djangoproject.com/en/3.2/topics/async/)
  - _FastAPI_ - modern, specialized for type hints, supports explicit async IO and auto-generates Swagger UI (API spec)
  - Integrates with
  - Real Python reference
  - Docker official docs reference
  - _attrs_
  - _websockets_
  - _MessagePack_ - [_msgpack_](https://github.com/msgpack/msgpack-python)
  - _graphene_
  - _Flask_ - exists since 2010, no explicit async IO support
  - _dynaconf_ - environment, many formats, external config stores (e.g. Redis), unit tests and more
  - _typer_
  - _aiohttp_
  - _arq_
  - _tenacity_ - aioretry) which requires implementing the retry policy yourself but supports type hints.
  - _httpx_
  - **Generic**
  - Real Python reference
  - \_\_init\_\_.py
  - benefit from Python Wheels
  - Real Python reference
  - Real Python reference
  - Real Python reference - method-names)
  - **Protocol**
  - Official docs built-in exceptions reference
  - Official docs reference
  - Real Python Concurrency reference - io-python/) & [Official docs asyncio reference](https://docs.python.org/3/library/asyncio.html)
  - _orjson_
  - _MyPy_ - docs.helpmanual.io/usage/mypy/) to cover static type check scenarios for creating models using the _pydantic_ syntax
  - Integrates with - docs.helpmanual.io/hypothesis_plugin/)[library](https://pydantic-docs.helpmanual.io/hypothesis_plugin/) for theory testing of data models
  - Pandas - fast, powerful, flexible and simple data analysis & manipulation library
  - _pydantic_ - environment](https://rednafi.github.io/digressions/python/2020/06/03/python-configs.html) and custom loaders can make it leverage _dynaconf_
  - _aiofiles_
  - Real Python reference
  - via another package
  - Integrates with - docs.helpmanual.io/usage/models/#orm-mode-aka-arbitrary-class-instances) to map _SQLAlchemy_ models to/from _pydantic_ models
  - Integrates with - databases/) to expose CRUD API on top of SQL databases
  - Real Python reference
  - _daiquiri_
  - _structlog_
  - _lagom_
  - web frameworks reference
  - Real Python reference
  - Integrates with _Flask_
  - Integrates with _Django_
  - _eventsourcing_
  - Docker with Python in VS Code reference
  - Real Python Docker tutorials reference
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - _Pyright_ - in with VS Code Python extension and supports execution in CI technologies, however at time of writing not supported by Pycharm.
  - Integrates with _Flask_
  - _apscheduler_
  - _rodi_ - in DI container, less features and less Github activity (commits/contributors/etc.) but simpler to use
  - _RxPY_
  - _orjson_
  - Integrates with _pydantic_
  - _pybreaker_
  - OAuth2 extension
  - Integrates with _FastAPI_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with - sqlalchemy.palletsprojects.com/en/2.x/) to expose CRUD API on top of SQL databases
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
  - Integrates with _Flask_
- Tips and Tricks
  - Limiting concurrency for outgoing HTTP requests sent with _aiohttp_
  - Centralize imports in a reusable imports file
  - Official docs reference
  - Efficient string concat - converse memory that would be used in loop of string concat ops
  - here
  - Parameter Specific Variables
  - Python's logging source code
  - Translate (some) LINQ to Python - linq_ library above
  - Correctly executing sync/async generators
  - Multi-core async IO
  - Fork-join pattern with pool executors - join concurrency pattern [here](https://en.wikipedia.org/wiki/Fork%E2%80%93join_model).
  - Limiting concurrency for large number of async IO tasks
  - Translate (some) LINQ to Python - linq_ library above
  - Centralize imports in a reusable imports file
  - Multi-core async IO
- Documentation
  - _Markdown All in One_
  - supported as is in VS Code
  - Real Python reference
  - NumPy Docstrings guidelines - recommended standard docstrings style, not only one
  - _pydocstyle_
  - _Flake8_ plugin for _pydocstyle_
  - Professional documentation with _reStructuredText_
- Versioning and Environments Management
  - _Poetry_ - Virtual envs package management solution similar to _npm_
  - pyenv - Python versioning manager
- Unit Testing
  - Real Python reference
  - _faker_
  - _Python Test Explorer_
  - Useful article - mock-library/) & [Library docs reference](https://github.com/pytest-dev/pytest-mock)
  - _pydantic-factories_
  - _pytest-cov_
  - _pytest-bdd_
  - _hypothesis_
  - _pytest-lazy-fixture_
  - _pytest-asyncio_
  - _aioresponses_
  - _freezegun_
  - _assertpy_
  - _nextmock_
  - _respx_
- Code Quality and Linting
  - _Flake8_
  - _MyPy_
  - VS Code Python Linting - so VS Code will run _MyPy_, _Flake8_ (and others)
  - VS Code _Black_ and _isort_
  - Github action for Python Code Quality and Linting
  - _Black_
  - _isort_
  - _pep8-naming_ - checks that names follow Python standards defined in PEP8
  - _flake8-builtins_ - checks for accidental use of builtin functions as names
  - _flake8-logging-format_ - ensures logs use extra arguments and exception()
  - _flake8-mutable_ - checks for mutable default parameter values Python issue
  - _flake8-pytest-style_ - check that pytest unit tests are written according to style
  - _flake8-comprehensions_ - checks for misuse or lack of use of comprehensions
  - _flake8-simplify_ - _ checks for general Python best practices for simpler code
Environment setup and installations
- PyCharm IDE - PyCharm is a very popular IDE for Python from JetBrains, the same company that delivers IntelliJ IDE.
- JupyterLab and Jupyter Notebook
- Google Colab - online Python notebook style editor by Google. Provides access to free GPU.
- VS Code reference - development-visual-studio-code/).
- Windows - for Windows make sure to have both Python and its scripts in your PATH environment variables, they'll be in your Python installation directory, example:
- Real Python reference
- ![image
- deadsnakes PPA
- Windows
- Dataspell - a relatively new IDE from JetBrains which supports running notebooks cells (.ipynb files). Has features of traditional IDE as opposed to Jupyter Notebooks and JupyterLabs.
- Hyper-V - V works as well, for personal development it's a matter of preference, you can read more [here](https://www.makeuseof.com/tag/virtualbox-vs-vmware-vs-hyper-v/).
- Hyper-V - V works as well, for personal development it's a matter of preference, you can read more [here](https://www.makeuseof.com/tag/virtualbox-vs-vmware-vs-hyper-v/).
General tutorials and guides
- Automate the boring stuff - beginner-intermediate. Task oriented online book.
- W3Schools - beginners
- Real Python - intermediate, in depth, articles referencing useful open source packages
- Python official docs tutorial - exhaustive, most in depth, tutorial for must and should know built-in Python capabilities
- GeekForGeeks - beginner-intermediate, website with short tutorials in many subjects. Contains many examples and a "try it out" widget.
- GeekForGeeks - beginner-intermediate, website with short tutorials in many subjects. Contains many examples and a "try it out" widget.
Important Standard Library Modules
- https://docs.python.org/3/py-modindex.html
- a dictionary with typed values

Programming Languages

Python 32 Cython 1 Roff 1 C 1

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

awesome-python-backend

Topics Index

QoL Libraries

Distributed programming frameworks

Commonly Useful Knowledge

Tips and Tricks

Documentation

Versioning and Environments Management

Unit Testing

Code Quality and Linting

Environment setup and installations

General tutorials and guides

Important Standard Library Modules