Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
awesome-python-backend
Index for online reading materials in order to learn Python and backend development/engineering concepts from scratch and develop a mastery sufficient for Senior/Principal Backend Engineers and Data Engineers
https://github.com/Eldar1205/awesome-python-backend
Last synced: 2 days ago
JSON representation
-
Topics Index
-
Distributed programming frameworks
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- _Kubernetes_
- there are workarounds
- Useful production ready deployment reference
- _Faust_ - a microservices framework which provides abstractions for stream processing with stateful agents that process infinite streams of messages built on top of [_Kafka_](https://kafka.apache.org/) for pub-sub and queue based messaging (extensible), [_RocksDB_](https://rocksdb.org/) as local persistent tables store and (optionally) [_Redis_](https://redis.io/) as distributed cache. Co-founded by [_Celery_](https://docs.celeryproject.org/en/latest/index.html) founder.
- leader election
- _Airflow_ - A workflow framework for orchestrating distributed scheduled/triggered workflows (a.k.a DAGs) described using Python scripts such that workers can execute tasks (a.k.a operators) in workflows. Built on top of SQL databases for workflow state persistence and can execute on a cluster directly using [_Celery_](https://docs.celeryproject.org/en/latest/getting-started/)or [_Dask_](https://dask.org/), on a [_Kubernetes_](https://kubernetes.io/) cluster, or managed in [AWS](https://aws.amazon.com/managed-workflows-for-apache-airflow/)/[GCP](https://cloud.google.com/composer). Comes with several core operators and there's a great deal of [independent ones](https://airflow.apache.org/docs/apache-airflow-providers/operators-and-hooks-ref/index.html) and a [registry of built-in & community provided ones](https://registry.astronomer.io/).
- Python code - branch-operator), [Run DAG](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/operators/trigger_dagrun/index.html#module-airflow.operators.trigger_dagrun), [HTTP request](https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html#howto-operator-simplehttpoperator), [SQL checks](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/operators/sql/index.html#module-airflow.operators.sql), etc.
- Python code - airflow/stable/_api/airflow/sensors/sql/index.html#module-airflow.sensors.sql), [Time Wait](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/sensors/time_delta/index.html#module-airflow.sensors.time_delta), etc.
- _Jinja_
- _Great Expectations_
- the Linux subsystem can be used
- _Ray_ - A workers & actors framework for implementing distributed, fault tolerant and scalable applications specialized for data science but usable for any distributed computing. Workers are stateless, actors are stateful, both are Python processes and therefore heavyweight and coarse grained. [Ray has an academic whitepaper](https://www.usenix.org/system/files/osdi18-moritz.pdf).
- _Airflow_
- _Thespian_
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- _useful reference_
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- the Linux subsystem can be used
- _gevent_ - blocking I/O
- the Linux subsystem can be used
-
Commonly Useful Knowledge
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Real Python reference
- \_\_init\_\_.py
- benefit from Python Wheels
- Real Python reference
- Real Python reference
- Real Python reference - method-names)
- Real Python reference
- **Generic**
- **Protocol**
- Official docs built-in exceptions reference
- Official docs reference
- Real Python Concurrency reference - io-python/) & [Official docs asyncio reference](https://docs.python.org/3/library/asyncio.html)
- _orjson_
- _MyPy_ - docs.helpmanual.io/usage/mypy/) to cover static type check scenarios for creating models using the _pydantic_ syntax
- Integrates with - docs.helpmanual.io/hypothesis_plugin/)[library](https://pydantic-docs.helpmanual.io/hypothesis_plugin/) for theory testing of data models
- _attrs_
- NumPy - ** a Python library used for working with arrays
- Pandas - fast, powerful, flexible and simple data analysis & manipulation library
- Scipy - a scientific computation library that uses NumPy underneath
- _dynaconf_ - environment, many formats, external config stores (e.g. Redis), unit tests and more
- _pydantic_ - environment](https://rednafi.github.io/digressions/python/2020/06/03/python-configs.html) and custom loaders can make it leverage _dynaconf_
- _aiofiles_
- Real Python reference
- _httpx_
- _aiohttp_
- via another package
- _websockets_
- _SQLAlchemy_
- Integrates with
- Integrates with - docs.helpmanual.io/usage/models/#orm-mode-aka-arbitrary-class-instances) to map _SQLAlchemy_ models to/from _pydantic_ models
- Integrates with - databases/) to expose CRUD API on top of SQL databases
- Integrates with - sqlalchemy.palletsprojects.com/en/2.x/) to expose CRUD API on top of SQL databases
- _tenacity_ - aioretry) which requires implementing the retry policy yourself but supports type hints.
- _MessagePack_ - [_msgpack_](https://github.com/msgpack/msgpack-python)
- Real Python reference
- _daiquiri_
- _structlog_
- _lagom_
- _typer_
- _FastAPI_ - modern, specialized for type hints, supports explicit async IO and auto-generates Swagger UI (API spec)
- _Flask_ - exists since 2010, no explicit async IO support
- _Django_ - a very extensive framework with many many features, essentially an ecosystem, well documented, [limited explicit async IO support](https://docs.djangoproject.com/en/3.2/topics/async/)
- web frameworks reference
- _arq_
- Real Python reference
- _graphene_
- Integrates with _Flask_
- Integrates with _Django_
- _eventsourcing_
- Docker official docs reference
- Docker with Python in VS Code reference
- Real Python Docker tutorials reference
- Integrates with _Flask_
- _orjson_
- OAuth2 extension
- _pybreaker_
- _rodi_ - in DI container, less features and less Github activity (commits/contributors/etc.) but simpler to use
- _apscheduler_
- Integrates with _pydantic_
- Integrates with _FastAPI_
- Integrates with _Flask_
- _RxPY_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- _Pyright_ - in with VS Code Python extension and supports execution in CI technologies, however at time of writing not supported by Pycharm.
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
- Integrates with _Flask_
-
Versioning and Environments Management
-
QoL Libraries
- https://github.com/vinta/awesome-python
- _yarl_
- _asyncstdlib_ - thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#cleanup-in-generators-and-async-generators) for details.
- _aiomisc_
- Decorator support
- _bidict_
- dog-piling - [_py-memoize_](https://memoize.readthedocs.io/en/latest/)
- _multidict_
- _toolz_ - itertools_](https://more-itertools.readthedocs.io/en/stable/api.html)
- _classes_
- type annotations - for cases where static type checks aren't sufficient, e.g. you want to assert type annotations not statically checked, or you are developing a Python library (its users might not perform static type checks when calling your library's classes/functions/etc.) - [_beartype_](https://github.com/beartype/beartype)
- _python-varname_
- _uvloop_
- _immutables_
- _sortedcontainers_
- _py-linq_
- _pipe_ - clean-python-code-using-pipes-1239a0f3abf5)
- _pampy_
- _object-mapper_
- **PeriodicCallback** - callback)
- _result_
-
Unit Testing
- _Python Test Explorer_
- Real Python reference
- Useful article - mock-library/) & [Library docs reference](https://github.com/pytest-dev/pytest-mock)
- _faker_
- _pydantic-factories_
- _pytest-cov_
- _pytest-bdd_
- _hypothesis_
- _pytest-lazy-fixture_
- _pytest-asyncio_
- _assertpy_
- _nextmock_
- _freezegun_
- _aioresponses_
- _respx_
-
Code Quality and Linting
- _Flake8_
- VS Code Python Linting - so VS Code will run _MyPy_, _Flake8_ (and others)
- VS Code _Black_ and _isort_
- Github action for Python Code Quality and Linting
- _flake8-builtins_ - checks for accidental use of builtin functions as names
- _flake8-comprehensions_ - checks for misuse or lack of use of comprehensions
- _flake8-logging-format_ - ensures logs use extra arguments and exception()
- _flake8-mutable_ - checks for mutable default parameter values Python issue
- _pep8-naming_ - checks that names follow Python standards defined in PEP8
- _flake8-pytest-style_ - check that pytest unit tests are written according to style
- _flake8-simplify_ - _ checks for general Python best practices for simpler code
- _Black_
- _isort_
- _MyPy_
-
Tips and Tricks
- Centralize imports in a reusable imports file
- Official docs reference
- Efficient string concat - converse memory that would be used in loop of string concat ops
- here
- Parameter Specific Variables
- Python's logging source code
- Translate (some) LINQ to Python - linq_ library above
- Correctly executing sync/async generators
- Multi-core async IO
- Fork-join pattern with pool executors - join concurrency pattern [here](https://en.wikipedia.org/wiki/Fork%E2%80%93join_model).
- Limiting concurrency for large number of async IO tasks
- Limiting concurrency for outgoing HTTP requests sent with _aiohttp_
- Multi-core async IO
-
Documentation
- Real Python reference
- NumPy Docstrings guidelines - recommended standard docstrings style, not only one
- _pydocstyle_
- Professional documentation with _reStructuredText_
- supported as is in VS Code
- _Markdown All in One_
- _Flake8_ plugin for _pydocstyle_
-
-
Environment setup and installations
- Hyper-V - V works as well, for personal development it's a matter of preference, you can read more [here](https://www.makeuseof.com/tag/virtualbox-vs-vmware-vs-hyper-v/).
- Windows - for Windows make sure to have both Python and its scripts in your PATH environment variables, they'll be in your Python installation directory, example:
- 
Categories
Sub Categories
Keywords
python
18
python3
6
flake8
4
flake8-plugin
3
flake8-extensions
3
asyncio
3
linter
2
pytest
2
httpx
2
formatter
2
async
2
mock
2
testing
2
graphql
1
graphene
1
fastapi
1
starlette
1
reactive
1
reactive-extensions
1
reactivex
1
rxpy
1
shell
1
debugging-tool
1
nameof
1
python-variables
1
python-varname
1
variable-name-recovery
1
variables
1
async-await
1
async-python
1
event-loop
1
high-performance
1
awesome
1
collections
1
python-framework
1
python-library
1
python-resources
1
dataclasses
1
datetime
1
deserialization
1
json
1
numpy
1
pyo3
1
rust
1
serialization
1
active-directory
1
api-key
1
auth
1
aws
1
azure
1