{"id":13501363,"url":"https://github.com/tolomea/django-auto-prefetch","last_synced_at":"2025-03-29T08:32:25.410Z","repository":{"id":37941154,"uuid":"249713097","full_name":"tolomea/django-auto-prefetch","owner":"tolomea","description":"Automatically prefetch foreign key values as needed","archived":false,"fork":false,"pushed_at":"2024-10-29T10:55:32.000Z","size":754,"stargazers_count":354,"open_issues_count":8,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-29T12:46:46.430Z","etag":null,"topics":["django"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tolomea.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-24T13:18:23.000Z","updated_at":"2024-10-29T10:55:35.000Z","dependencies_parsed_at":"2023-10-14T14:04:03.107Z","dependency_job_id":"80413100-e258-4f45-93ef-2055c9cfee5f","html_url":"https://github.com/tolomea/django-auto-prefetch","commit_stats":{"total_commits":267,"total_committers":7,"mean_commits":"38.142857142857146","dds":0.348314606741573,"last_synced_commit":"14468b3a1f81a01e661f980c10c1f681ca7033d1"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolomea%2Fdjango-auto-prefetch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolomea%2Fdjango-auto-prefetch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolomea%2Fdjango-auto-prefetch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolomea%2Fdjango-auto-prefetch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tolomea","download_url":"https://codeload.github.com/tolomea/django-auto-prefetch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246162092,"owners_count":20733351,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django"],"created_at":"2024-07-31T22:01:34.664Z","updated_at":"2025-03-29T08:32:25.403Z","avatar_url":"https://github.com/tolomea.png","language":"Python","funding_links":[],"categories":["Python","Database","Recently Updated","Third-Party Packages"],"sub_categories":["Tools","[Oct 03, 2024](/content/2024/10/03/README.md)","Models"],"readme":"django-auto-prefetch\n====================\n\n.. image:: https://img.shields.io/github/actions/workflow/status/tolomea/django-auto-prefetch/main.yml.svg?branch=main\u0026style=for-the-badge\n   :target: https://github.com/tolomea/django-auto-prefetch/actions?workflow=CI\n\n.. image:: https://img.shields.io/badge/Coverage-100%25-success?style=for-the-badge\n   :target: https://github.com/tolomea/django-auto-prefetch/actions?workflow=CI\n\n.. image:: https://img.shields.io/pypi/v/django-auto-prefetch.svg?style=for-the-badge\n   :target: https://pypi.org/project/django-auto-prefetch/\n\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg?style=for-the-badge\n   :target: https://github.com/python/black\n\n.. image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white\u0026style=for-the-badge\n   :target: https://github.com/pre-commit/pre-commit\n   :alt: pre-commit\n\nAutomatically prefetch foreign key values as needed.\n\nPurpose\n-------\n\nWhen accessing a ``ForeignKey`` or ``OneToOneField`` (including in reverse) on\na model instance, if the field’s value has not yet been loaded then\nauto-prefetch will prefetch the field for all model instances loaded by\nthe same ``QuerySet`` as the current model instance. This is enabled at the\nmodel level and totally automatic and transparent for users of the\nmodel.\n\nRequirements\n------------\n\nPython 3.9 to 3.13 supported.\n\nDjango 4.2 to 5.2 supported.\n\nUsage\n-----\n\n1. Install with ``python -m pip install django-auto-prefetch``.\n\n2. Change all these imports from ``django.db.models`` to ``auto_prefetch``:\n\n   *  ``ForeignKey``\n   *  ``Manager``\n   *  ``Model`` - including inheriting ``Meta`` from ``auto_prefetch.Model.Meta``\n   *  ``OneToOneField``\n   *  ``QuerySet``\n\n   If you use custom subclasses of any of these classes, you should be able to swap for the ``auto_prefetch`` versions in your subclasses’ bases.\n\n   For example, if you had:\n\n   .. code:: python\n\n      from django.db import models\n\n\n      class Book(models.Model):\n          author = models.ForeignKey(\"Author\", on_delete=models.CASCADE)\n\n          class Meta:\n              verbose_name = \"Book\"\n\n   …swap to:\n\n   .. code:: python\n\n      import auto_prefetch\n      from django.db import models\n\n\n      class Book(auto_prefetch.Model):\n          author = auto_prefetch.ForeignKey(\"Author\", on_delete=models.CASCADE)\n\n          class Meta(auto_prefetch.Model.Meta):\n              verbose_name = \"Book\"\n\n3. Run ``python manage.py makemigrations`` to generate migrations for all the models you modified.\n   These migrations will set the |Meta.base_manager_name option|__ to ``prefetch_manager`` for every model that you’ve converted.\n   This change ensures that auto-prefetching happens on related managers.\n   Such migrations do not change anything in the database.\n\n   .. |Meta.base_manager_name option| replace:: ``Meta.base_manager_name`` option\n   __ https://docs.djangoproject.com/en/stable/ref/models/options/#base-manager-name\n\n   (If you instead set ``Meta.base_manager_name`` on your models, make sure it inherits from ``auto_prefetch.Manager``.)\n\nBackground and Rationale\n------------------------\n\nCurrently when accessing an uncached foreign key field, Django will\nautomatically fetch the missing value from the database. When this\noccurs in a loop it creates 1+N query problems. Consider the following\nsnippet:\n\n.. code:: python\n\n   for choice in Choice.objects.all():\n       print(choice.question.question_text, \":\", choice.choice_text)\n\nThis will do one query for the choices and then one query per choice to\nget that choice’s question.\n\nThis behavior can be avoided with correct application of\n``prefetch_related()`` like this:\n\n.. code:: python\n\n   for choice in Choice.objects.prefetch_related(\"question\"):\n       print(choice.question.question_text, \":\", choice.choice_text)\n\nThis has several usability issues, notably:\n\n* Less experienced users are generally not aware that it’s necessary.\n* Cosmetic seeming changes to things like templates can change the fields that\n  should be prefetched.\n* Related to that, the code that requires the ``prefetch_related()`` (e.g. the\n  template) may be quite removed from where the ``prefetch_related()`` needs to\n  be applied (e.g. the view).\n* Subsequently finding where ``prefetch_related()`` / ``select_related()``\n  calls are missing is non-trivial and needs to be done on an ongoing\n  basis.\n* Excess entries in ``prefetch_related()`` calls are even harder\n  to find and result in unnecessary database queries.\n* It is very difficult for libraries like the admin and Django Rest Framework\n  to automatically generate correct ``prefetch_related()`` clauses.\n\nOn the first iteration of the loop in the example above, when we first\naccess a choice’s question field, instead of fetching the question for\njust that choice, auto-prefetch will speculatively fetch the questions\nfor all the choices returned by the ``QuerySet``. This change results in\nthe first snippet having the same database behavior as the second while\nreducing or eliminating all of the noted usability issues.\n\nSome important points:\n\n* ``ManyToManyField``\\s are not changed at all.\n* Because these are ``ForeignKey`` and ``OneToOneField``\\s, the\n  generated queries can’t have more result rows than the original query\n  and may have less. This eliminates any concern about a multiplicative\n  query size explosion.\n* This feature will never result in more database queries as a prefetch will\n  only be issued where the ORM was already going to fetch a single related\n  object.\n* Because it is triggered by fetching missing related objects it will not at\n  all change the DB behavior of code which is fully covered by\n  ``prefetch_related()`` and/or ``select_related()`` calls.\n* This will inherently chain across relations like ``choice.question.author``.\n  The conditions above still hold under such chaining.\n* In some rare situations it may result in larger data transfer between the\n  database and Django (see below).\n\nAn example of that last point is:\n\n.. code:: python\n\n   qs = Choice.objects.all()\n   list(qs)[0].question\n\nSuch examples generally seem to be rarer and more likely to be visible\nduring code inspection (vs ``{{ choice.question }}`` in a template). And\nlarger queries are usually a better failure mode than producing hundreds\nof queries. For this to actually produce inferior behavior in practice\nyou need to:\n* fetch a large number of choices\n* filter out basically all of them\n* ...in a way that prevents garbage collection of the unfiltered ones\n\nIf any of those aren’t true then automatic prefetching will still\nproduce equivalent or better database behavior than without.\n\nSee Also\n--------\n\n*  The phabricator guide to the N+1 queries problem:\n   https://secure.phabricator.com/book/phabcontrib/article/n_plus_one/\n*  The django-developers mailing list discussion of adding the feature\n   to core Django:\n   https://groups.google.com/forum/m/#!topic/django-developers/EplZGj-ejvg\n*  The nplus package, useful for detecting the N+1 queries problem in\n   your application (but not solving it):\n   https://pypi.org/project/nplusone/\n\nP.S.\n----\n\nIf you have concerns go look at the code, it’s all in\n`auto_prefetch/__init__.py \u003chttps://github.com/tolomea/django-auto-prefetch/blob/main/src/auto_prefetch/__init__.py\u003e`__\nand is fairly short.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftolomea%2Fdjango-auto-prefetch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftolomea%2Fdjango-auto-prefetch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftolomea%2Fdjango-auto-prefetch/lists"}