awesome-python-backend
Index for online reading materials in order to learn Python and backend development/engineering concepts from scratch and develop a mastery sufficient for Senior/Principal Backend Engineers and Data Engineers
https://github.com/Eldar1205/awesome-python-backend
Last synced: about 7 hours ago
JSON representation
Environment setup and installations
- Python installation - for Windows make sure to have both Python and its scripts in your PATH environment variables, they'll be in your Python installation directory, example:
-
deadsnakes PPA
- to install Python 3.10:
$ sudo apt-get install software-properties-common$ sudo add-apt-repository ppa:deadsnakes/ppa$ sudo apt-get update$ sudo apt-get install python3.10You can specify the exact python version if needed.To verify which version of Python 3 you have installed, open a command prompt and run:$ python3 --version - Real Python reference - & Official docs reference
- JupyterLab and Jupyter Notebook - are notebook style editors for Python where you can write code, save data, write free text explanations, etc.
- Python & Jupyter Notebook Windows installation guide
- PyCharm IDE - PyCharm is a very popular IDE for Python from JetBrains, the same company that delivers IntelliJ IDE.
- VS Code reference - & Real Python reference.
- Google Colab - online Python notebook style editor by Google. Provides access to free GPU.
- Dataspell - a relatively new IDE from JetBrains which supports running notebooks cells (.ipynb files). Has features of traditional IDE as opposed to Jupyter Notebooks and JupyterLabs.
General tutorials and guides
- W3Schools - beginners
- Real Python - intermediate, in depth, articles referencing useful open source packages
- Python official docs tutorial - exhaustive, most in depth, tutorial for must and should know built-in Python capabilities
- Automate the boring stuff - beginner-intermediate. Task oriented online book.
- GeekForGeeks - beginner-intermediate, website with short tutorials in many subjects. Contains many examples and a "try it out" widget.
Important Standard Library Modules
Topics Index
Code Quality and Linting
- Flake8 - . Useful extensions:
- VS Code Python Linting - so VS Code will run MyPy, Flake8 (and others)
- VS Code Black and isort
- Github action for Python Code Quality and Linting
- flake8-builtins - checks for accidental use of builtin functions as names
- flake8-comprehensions - checks for misuse or lack of use of comprehensions
- flake8-logging-format - ensures logs use extra arguments and exception()
- flake8-mutable - checks for mutable default parameter values Python issue
- pep8-naming - checks that names follow Python standards defined in PEP8
- flake8-pytest-style - check that pytest unit tests are written according to style
- flake8-simplify - - checks for general Python best practices for simpler code
- Black
- isort
Commonly Useful Knowledge
- Real Python reference - & Official docs reference
- __init__.py
- benefit from Python Wheels
- Real Python reference - & Official docs reference
- Real Python reference
- Real Python reference - & Official docs reference
- Real Python reference - & Official docs reference
- Generic - (generic constraints, covariance, contravariance, etc.)
- Protocol - (essentially define interfaces for static type check purposes, can support runtime isinstance and issubclass checks using a decorator)
- Official docs built-in exceptions reference
- Official docs reference
- Real Python Concurrency reference - & Real Python asyncio reference & Official docs asyncio reference
- attrs - , comparisons can be found online
- NumPy - - a Python library used for working with arrays
- Pandas - fast, powerful, flexible and simple data analysis & manipulation library
- Scipy - a scientific computation library that uses NumPy underneath
- dynaconf - supporting multi-environment, many formats, external config stores (e.g. Redis), unit tests and more
- Real Python reference
- httpx
- aiohttp
- via another package - )
- websockets
- SQLAlchemy
- Integrates with - MyPy for type checking SQLAlchemy models
- Integrates with - FastAPI to expose CRUD API on top of SQL databases
- tenacity - , type hints support incomplete. There's also aioretry which requires implementing the retry policy yourself but supports type hints.
- MessagePack - is a very efficient general purpose format - msgpack
- Real Python reference
- daiquiri
- structlog
- lagom
- typer
- FastAPI - modern, specialized for type hints, supports explicit async IO and auto-generates Swagger UI (API spec)
- Flask - exists since 2010, no explicit async IO support
- Django - a very extensive framework with many many features, essentially an ecosystem, well documented, limited explicit async IO support* Comparisons exist online, FastAPI is mostly preferred due to explicit async IO* Useful web frameworks reference on how to evaluate them
- arq - is a lightweight distributed task queue built on top of Redis
- Real Python reference - & Official Google docs reference
- graphene
- Integrates with Django - to expose GraphQL API over Django models
- eventsourcing
- Docker official docs reference
- Docker with Python in VS Code reference
- Real Python Docker tutorials reference
- Pyright - is another popular tool for static type checks, more far ahead than Mypy regarding support for newest Python typing features, comes built-in with VS Code Python extension and supports execution in CI technologies, however at time of writing not supported by Pycharm.
- orjson
- OAuth2 extension
- pybreaker
- rodi - which is inspired by .Net built-in DI container, less features and less Github activity (commits/contributors/etc.) but simpler to use
- apscheduler - is a scheduler for a single worker with flexible jobs store choices
- Integrates with pydantic - to query over pydantic models
- Integrates with FastAPI - to expose GraphQL API, usually over pydantic models
- RxPY
- Integrates with - Flask to expose CRUD API on top of SQL databases
- Integrates with - pydantic to map SQLAlchemy models to/from pydantic models
- MyPy - is a very popular tool for static type checks, and the docs are very useful to learn how to use type hints, integrable into Pycharm IDE as well as VS Code, and supports execution in CI technologies.
- orjson
- MyPy - plugin to cover static type check scenarios for creating models using the pydantic syntax
- Integrates with - hypothesislibrary for theory testing of data models
- pydantic - library above, can be extended to support multi-environment and custom loaders can make it leverage dynaconf
- aiofiles
- Integrates with Flask - to expose GraphQL API over whatever data you choose
Distributed programming frameworks
- Kubernetes - deployment
- there are workarounds - * Useful production ready deployment reference
- Faust - a microservices framework which provides abstractions for stream processing with stateful agents that process infinite streams of messages built on top of Kafka for pub-sub and queue based messaging (extensible), RocksDB as local persistent tables store and (optionally) Redis as distributed cache. Co-founded by Celery founder.Features:
- leader election
- Airflow - A workflow framework for orchestrating distributed scheduled/triggered workflows (a.k.a DAGs) described using Python scripts such that workers can execute tasks (a.k.a operators) in workflows. Built on top of SQL databases for workflow state persistence and can execute on a cluster directly using Celeryor Dask, on a Kubernetes cluster, or managed in AWS/GCP. Comes with several core operators and there's a great deal of independent ones and a registry of built-in & community provided ones.Features:
- Python code - , Branching, Run DAG, HTTP request, SQL checks, etc.
- Jinja - and injecting values from env variables/JSON files/etc.
- Great Expectations
- Ray - A workers & actors framework for implementing distributed, fault tolerant and scalable applications specialized for data science but usable for any distributed computing. Workers are stateless, actors are stateful, both are Python processes and therefore heavyweight and coarse grained. Ray has an academic whitepaper.Features:
- Airflow - and other useful technologies
- Thespian - , much less popular than above.
- gevent - for implicit non-blocking I/O
- Celery - a distributed task queue & scheduling framework which provides workers that execute RPC jobs consumed from a flexible choice of message brokers (usually RabbitMQ or Redis) with job results persisted to a flexible choice of stores.Features:
- Python code - , SQL query, Time Wait, etc.
- useful reference - ):
- the Linux subsystem can be used
Documentation
- Real Python reference
- NumPy Docstrings guidelines - recommended standard docstrings style, not only one
- Professional documentation with reStructuredText - (to publish PDF/HTML/etc.)
- supported as is in VS Code - , e.g. Git README.md files
- Markdown All in One
- pydocstyle
- Flake8 plugin for pydocstyle
QoL Libraries
- asyncstdlib - , especially important aclosing to properly cleanup after async generators, see this discussion for details.
- aiomisc
- dog-piling - , flexible yet simple - py-memoize
- bidict
- toolz - & more-itertools
- classes
- type annotations - with custom validations - for cases where static type checks aren't sufficient, e.g. you want to assert type annotations not statically checked, or you are developing a Python library (its users might not perform static type checks when calling your library's classes/functions/etc.) - beartype
- python-varname
- uvloop
- immutables
- sortedcontainers
- py-linq
- pipe - , useful reference
- pampy - is simple and very popular
- object-mapper - or odin
- result
- yarl - * pydantic also ships a URL type, however it's not useful for constructing URLs as yarl
- Decorator support - for asyncio.waitfor function
- select
- multidict
Tips and Tricks
- Official docs reference
- Efficient string concat - converse memory that would be used in loop of string concat ops
- here - .* There's an issue with type-hints to express the decorator receiving a sync callable and returning an async callable, solved in Python 3.10: Parameter Specific Variables.
- Python's logging source code
- Correctly executing sync/async generators - so a finally clause runs explicitly
- Limiting concurrency for outgoing HTTP requests sent with aiohttp
- Translate (some) LINQ to Python - instead of using py-linq library above
- Centralize imports in a reusable imports file
- Multi-core async IO - (multiple event loops)
- Fork-join pattern with pool executors - , can read about Fork-join concurrency pattern here.
- Limiting concurrency for large number of async IO tasks
Unit Testing
- Python Test Explorer
- Real Python reference - & Library docs reference
- Useful article - & Real Python reference & Library docs reference* Issue: no type hints for the mocked class/function.
- faker
- pytest-cov
- pytest-bdd
- hypothesis - library
- pytest-asyncio
- assertpy
- nextmock
- freezegun
- aioresponses
- respx
- pydantic-factories - to build fake models
- pytest-lazy-fixture