Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/scrapinghub/andi

Library for annotation-based dependency injection
https://github.com/scrapinghub/andi

Last synced: about 1 month ago
JSON representation

Library for annotation-based dependency injection

Awesome Lists containing this project

README 

        
====

andi

====



.. image:: https://img.shields.io/pypi/v/andi.svg

   :target: https://pypi.python.org/pypi/andi

   :alt: PyPI Version



.. image:: https://img.shields.io/pypi/pyversions/andi.svg

   :target: https://pypi.python.org/pypi/andi

   :alt: Supported Python Versions



.. image:: https://github.com/scrapinghub/andi/workflows/tox/badge.svg

   :target: https://github.com/scrapinghub/andi/actions

   :alt: Build Status



.. image:: https://codecov.io/github/scrapinghub/andi/coverage.svg?branch=master

   :target: https://codecov.io/gh/scrapinghub/andi

   :alt: Coverage report



``andi`` makes easy implementing custom dependency injection mechanisms

where dependencies are expressed using type annotations.



``andi`` is useful as a building block for frameworks, or as a library

which helps to implement dependency injection (thus the name -

ANnotation-based Dependency Injection).



License is BSD 3-clause.



Installation

============



::



    pip install andi



andi requires Python >= 3.9.



Goal

====



See the following classes that represents parts of a car

(and the car itself):



.. code-block:: python



    class Valves:

        pass



    class Engine:

        def __init__(self, valves):

            self.valves = valves



    class Wheels:

        pass



    class Car:

        def __init__(self, engine, wheels):

            self.engine = engine

            self.wheels = wheels



The following would be the usual way of build a ``Car`` instance:



.. code-block:: python



    valves = Valves()

    engine = Engine(valves)

    wheels = Wheels()

    car = Car(engine, wheels)



There are some dependencies between the classes: A car requires

and engine and wheels to be built, as well as the engine requires

valves. These are the car dependencies and sub-dependencies.



The question is, could we have an automatic way of building instances?

For example, could we have a ``build`` function that

given the ``Car`` class or any other class would return an instance

even if the class itself has some other dependencies?



.. code-block:: python



    car = build(Car)  # Andi helps creating this generic build function



``andi`` inspect the dependency tree and creates a plan making easy creating

such a ``build`` function.



This is how this plan for the ``Car`` class would looks like:



1. Invoke ``Valves`` with empty arguments

2. Invoke ``Engine`` using the instance created in 1 as the argument ``valves``

3. Invoke ``Wheels`` with empty arguments

4. Invoke ``Cars`` with the instance created in 2 as the ``engine`` argument and with

   the instance created in 3 as the ``wheels`` argument



Type annotations

----------------



But there is a missing piece in the Car example before. How can

``andi`` know that the class ``Valves`` is required to build the

argument ``valves``? A first idea would be to use the argument

name as a hint for the class name

(as `pinject `_ does),

but ``andi`` opts to rely on arguments' type annotations instead.



The classes for ``Car`` should then be rewritten as:



.. code-block:: python



    class Valves:

        pass



    class Engine:

        def __init__(self, valves: Valves):

            self.valves = valves



    class Wheels:

        pass



    class Car:

        def __init__(self, engine: Engine, wheels: Wheels):

            self.engine = engine

            self.wheels = wheels



Note how now there is a explicit annotation stating that the

``valves`` argument is of type ``Valves``

(same for ``engine`` and ``wheels``).



The ``andi.plan`` function can now create a plan to build the

``Car`` class (ignore the ``is_injectable`` parameter by now):



.. code-block:: python



    plan = andi.plan(Car, is_injectable={Engine, Wheels, Valves})



This is what the ``plan`` variable contains:



.. code-block:: python



    [(Valves, {}),

     (Engine, {'valves': Valves}),

     (Wheels, {}),

     (Car,    {'engine': Engine,

               'wheels': Wheels})]



Note how this plan correspond exactly to the 4-steps plan described

in the previous section.



Building from the plan

----------------------



Creating a generic function to build the instances from

a plan generated by ``andi`` is then very easy:



.. code-block:: python



    def build(plan):

        instances = {}

        for fn_or_cls, kwargs_spec in plan:

            instances[fn_or_cls] = fn_or_cls(**kwargs_spec.kwargs(instances))

        return instances



So let's see putting all the pieces together. The following code

creates an instance of ``Car`` using ``andi``:



.. code-block:: python



    plan = andi.plan(Car, is_injectable={Engine, Wheels, Valves})

    instances = build(plan)

    car = instances[Car]



is_injectable

-------------



It is not always desired for ``andi`` to manage every single annotation found.

Instead is usually better to explicitly declare which types

can be handled by ``andi``. The argument ``is_injectable``

allows to customize this feature.



``andi`` will raise an error on the presence of a dependency that cannot be resolved

because it is not injectable.



Usually is desirable to declare injectabilty by

creating a base class to inherit from. For example,

we could create a base class ``Injectable`` as base

class for the car components:



.. code-block:: python



    class Injectable(ABC):

        pass



    class Valves(Injectable):

        pass



    class Engine(Injectable):

        def __init__(self, valves: Valves):

            self.valves = valves



    class Wheels(Injectable):

        pass



The call to ``andi.plan`` would then be:



.. code-block:: python



    is_injectable = lambda cls: issubclass(cls, Injectable)

    plan = andi.plan(Car, is_injectable=is_injectable)



Functions and methods

---------------------



Dependency injection is also very useful when applied to functions.

Imagine that you have a function ``drive`` that drives the ``Car``

through the ``Road``:



.. code-block:: python



    class Road(Injectable):

        ...



    def drive(car: Car, road: Road, speed):

        ... # Drive the car through the road



The dependencies has to be resolved before invoking

the ``drive`` function:



.. code-block:: python



    plan = andi.plan(drive, is_injectable=is_injectable)

    instances = build(plan.dependencies)



Now the ``drive`` function can be invoked:



.. code-block:: python



    drive(instances[Car], instances[Road], 100)



Note that ``speed`` argument was not annotated. The resultant plan just won't include it

because the ``andi.plan`` ``full_final_kwargs`` parameter is ``False``

by default. Otherwise, an exception would have been raised (see ``full_final_kwargs`` argument

documentation for more information).



An alternative and more generic way to invoke the drive function

would be:



.. code-block:: python



    drive(speed=100, **plan.final_kwargs(instances))



dataclasses and attrs

---------------------



``andi`` supports classes defined using `attrs `_

and also `dataclasses `_.

For example the ``Car`` class could have been defined as:



.. code-block:: python



    # attrs class example

    @attr.s(auto_attribs=True)

    class Car:

        engine: Engine

        wheels: Wheels



    # dataclass example

    @dataclass

    class Car(Injectable):

        engine: Engine

        wheels: Wheels



Using ``attrs`` or ``dataclass`` is handy because they avoid

some boilerplate.



Externally provided dependencies

--------------------------------



Retaining the control over object instantiation

could be desired in some cases. For example creating

a database connection could require accessing some

credentials registry or getting the connection from a pool

so you might want to control building

such instances outside of the regular

dependency injection mechanism.



``andi.plan`` allows to specify which types would be

externally provided. Let's see an example:



.. code-block:: python



    class DBConnection(ABC):



        @abstractmethod

        def getConn():

            pass



    @dataclass

    class UsersDAO:

        conn: DBConnection



        def getUsers():

           return self.conn.query("SELECT * FROM USERS")



``UsersDAO`` requires a database connection to run queries.

But the connection will be provided externally from a pool, so we

call then ``andi.plan`` using also the ``externally_provided``

parameter:



.. code-block:: python



    plan = andi.plan(UsersDAO, is_injectable=is_injectable,

                     externally_provided={DBConnection})



The build method should then be modified slightly to be able

to inject externally provided instances:



.. code-block:: python



    def build(plan, instances_stock=None):

        instances_stock = instances_stock or {}

        instances = {}

        for fn_or_cls, kwargs_spec in plan:

            if fn_or_cls in instances_stock:

                instances[fn_or_cls] = instances_stock[fn_or_cls]

            else:

                instances[fn_or_cls] = fn_or_cls(**kwargs_spec.kwargs(instances))

        return instances



Now we are ready to create ``UserDAO`` instances with ``andi``:



.. code-block:: python



    plan = andi.plan(UsersDAO, is_injectable=is_injectable,

                     externally_provided={DBConnection})

    dbconnection = DBPool.get_connection()

    instances = build(plan.dependencies, {DBConnection: dbconnection})

    users_dao = instances[UsersDAO]

    users = user_dao.getUsers()



Note that being injectable is not required for externally provided

dependencies.



Optional

--------



``Optional`` type annotations can be used in case of

dependencies that can be optional. For example:



.. code-block:: python



    @dataclass

    class Dashboard:

        conn: Optional[DBConnection]



        def showPage():

            if self.conn:

                self.conn.query("INSERT INTO VISITS ...")

            ...  # renders a HTML page



In this example, the ``Dashboard`` class generates a HTML page to be served, and

also stores the number of visits into a database. Database

could be absent in some environments, but you might want

the dashboard to work even if it cannot log the visits.



When a database connection is possible the plan call would be:



.. code-block:: python



    plan = andi.plan(UsersDAO, is_injectable=is_injectable,

                     externally_provided={DBConnection})



And the following when the connection is absent:



.. code-block:: python



    plan = andi.plan(UsersDAO, is_injectable=is_injectable,

                     externally_provided={})



It is also required to register the type of ``None``

as injectable. Otherwise ``andi.plan`` with raise an exception

saying that "NoneType is not injectable".



.. code-block:: python



    Injectable.register(type(None))



Union

-----



``Union`` can also be used to express alternatives. For example:



.. code-block:: python



    @dataclass

    class UsersDAO:

        conn: Union[ProductionDBConnection, DevelopmentDBConnection]



``DevelopmentDBConnection`` will be injected in the absence of

``ProductionDBConnection``.



Annotated

---------



``Annotated`` type annotations can be used to attach arbitrary metadata that

will be preserved in the plan. Occurrences of the same type annotated with

different metadata will not be considered duplicates. For example:



.. code-block:: python



    @dataclass

    class Dashboard:

        conn_main: Annotated[DBConnection, "main DB"]

        conn_stats: Annotated[DBConnection, "stats DB"]



The plan will contain both dependencies.



Custom builders

---------------



Sometimes a dependency can't be created directly but needs some additional code

to be built. And that code can also have its own dependencies:



.. code-block:: python



    class Wheels:

        pass



    def wheel_factory(wheel_builder: WheelBuilder) -> Wheels:

        return wheel_builder.get_wheels()



As by default ``andi`` can't know how to create a ``Wheels`` instance or that

the plan needs to create a ``WheelBuilder`` instance first, it needs to be told

this with a ``custom_builder_fn`` argument:



.. code-block:: python



    custom_builders = {

        Wheels: wheel_factory,

    }



    plan = andi.plan(Car, is_injectable={Engine, Wheels, Valves},

                     custom_builder_fn=custom_builders.get,

                     )



``custom_builder_fn`` should be a function that takes a type and returns a factory

for that type.



The build code also needs to know how to build ``Wheels`` instances. A plan step

for an object built with a custom builder uses an instance of the ``andi.CustomBuilder``

wrapper that contains the type to be built in the ``result_class_or_fn`` attribute and

the callable for building it in the ``factory`` attribute:



.. code-block:: python



    from andi import CustomBuilder



    def build(plan):

        instances = {}

        for fn_or_cls, kwargs_spec in plan:

            if isinstance(fn_or_cls, CustomBuilder):

                instances[fn_or_cls.result_class_or_fn] = fn_or_cls.factory(**kwargs_spec.kwargs(instances))

            else:

                instances[fn_or_cls] = fn_or_cls(**kwargs_spec.kwargs(instances))

        return instances



Full final kwargs mode

-------------------------



By default ``andi.plan`` won't fail if it is not able to provide

some of the direct dependencies for the given input (see the

``speed`` argument in one of the examples above).



This behaviour is desired when inspecting functions

for which is already known that some arguments won't be

injectable but they will be provided by other means

(like the ``drive`` function above).



But in other cases is better to be sure that all dependencies

are fulfilled and otherwise fail. Such is the case for classes.

So it is recommended to set ``full_final_kwargs=True`` when invoking

``andi.plan`` for classes.



Overrides

---------



Let's go back to the ``Car`` example. Imagine you want to build a car again.

But this time you want to replace the ``Engine`` because this is

going to be an electric car!. And of course, an electric engine contains a battery

and have no valves at all. This could be the new ``Engine``:



.. code-block:: python



    class Battery:

        pass



    class ElectricEngine(Engine):



        def __init__(self, battery: Battery):

            self.battery = valves



Andi offers the possibility to replace dependencies when planning,

and this is what is required to build the electric car: we need

to replace any dependency on ``Engine`` by a dependency on ``ElectricEngine``.

This is exactly what overrides offers. Let's see how ``plan`` should

be invoked in this case:



.. code-block:: python



    plan = andi.plan(Car, is_injectable=is_injectable,

                     overrides={Engine: ElectricEngine}.get)



Note that Andi will unroll the new dependencies properly. That is,

``Valves`` and ``Engine`` won't be in the resultant plan but

``ElectricEngine`` and ``Battery`` will.



In summary, overrides offers a way to override the default

dependencies anywhere in the tree, changing them with an

alternative one.



By default overrides are not recursive: overrides aren't applied

over the children of an already overridden dependency. There

is flag to turn recursion on if this is what is desired.

Check ``andi.plan`` documentation for more information.



Why type annotations?

---------------------



``andi`` uses type annotations to declare dependencies (inputs).

It has several advantages, and some limitations as well.



Advantages:



1. Built-in language feature.

2. You're not lying when specifying a type - these

   annotations still work as usual type annotations.

3. In many projects you'd annotate arguments anyways, so ``andi`` support

   is "for free".



Limitations:



1. Callable can't have two arguments of the same type.

2. This feature could possibly conflict with regular type annotation usages.



If your callable has two arguments of the same type, consider making them

different types. For example, a callable may receive url and html of

a web page:



.. code-block:: python



    def parse(html: str, url: str):

        # ...



To make it play well with ``andi``, you may define separate types for url

and for html:



.. code-block:: python



    class HTML(str):

        pass



    class URL(str):

        pass



    def parse(html: HTML, url: URL):

        # ...



This is more boilerplate though.



Why doesn't andi handle creation of objects?

--------------------------------------------



Currently ``andi`` just inspects callable and chooses best concrete types

a framework needs to create and pass to a callable, without prescribing how

to create them. This makes ``andi`` useful in various contexts - e.g.



* creation of some objects may require asynchronous functions, and it

  may depend on libraries used (asyncio, twisted, etc.)

* in streaming architectures (e.g. based on Kafka) inspection may happen

  on one machine, while creation of objects may happen on different nodes

  in a distributed system, and then actually running a callable may happen on

  yet another machine.



It is hard to design API with enough flexibility for all such use cases.

That said, ``andi`` may provide more helpers in future,

once patterns emerge, even if they're useful only in certain contexts.



Examples: callback based frameworks

-----------------------------------



Spider example

**************



Nothing better than a example to understand how ``andi`` can be useful.

Let's imagine you want to implemented a callback based framework

for writing spiders to crawl web pages.



The basic idea is that there is framework in which the user

can write spiders. Each spider is a collection of callbacks

that can process data from a page, emit extracted data or request new

pages. Then, there is an engine that takes care of downloading

the web pages

and invoking the user defined callbacks, chaining requests

with its corresponding callback.



Let's see an example of an spider to download recipes

from a cooking page:



.. code-block:: python



    class MySpider(Spider):

        start_url = "htttp://a_page_with_a_list_of_recipes"



        def parse(self, response):

            for url in recipes_urls_from_page(response)

                yield Request(url, callback=parse_recipe)



        def parse_recipe(self, response):

            yield extract_recipe(response)



It would be handy if the user can define some requirements

just by annotating parameters in the callbacks. And ``andi`` make it

possible.



For example, a particular callback could require access to the cookies:



.. code-block:: python



    def parse(self, response: Response, cookies: CookieJar):

        # ... Do something with the response and the cookies



In this case, the engine can use ``andi`` to inspect the ``parse`` method, and

detect that ``Response`` and ``CookieJar`` are required.

Then the framework will build them and will invoke the callback.



This functionality would serve to inject into the users callbacks

some components only when they are required.



It could also serve to encapsulate better the user code. For

example, we could just decouple the recipe extraction into

it's own class:



.. code-block:: python



    @dataclass

    class RecipeExtractor:

        response: Response



        def to_item():

            return extract_recipe(self.response)



The callback could then be defined as:



.. code-block:: python



        def parse_recipe(extractor: RecipeExtractor):

            yield extractor.to_item()



Note how handy is that with ``andi`` the engine can create

an instance of ``RecipesExtractor`` feeding it with the

declared ``Response`` dependency.



In definitive, using ``andi`` in such a framework

can provide great flexibility to the user

and reduce boilerplate.



Web server example

******************



``andi`` can be useful also for implementing a new

web framework.



Let's imagine a framework where you can declare your sever in a

class like the following:



.. code-block:: python



    class MyWeb(Server):



        @route("/products")

        def productspage(self, request: Request):

            ... # return the composed page



        @route("/sales")

        def salespage(self, request: Request):

            ... # return the composed page



The former case is composed of two endpoints, one for serving

a page with a summary of sales, and a second one to serve

the products list.



Connection to the database can be required

to sever these pages. This logic could be encapsulated

in some classes:



.. code-block:: python



    @dataclass

    class Products:

        conn: DBConnection



        def get_products()

            return self.conn.query("SELECT ...")



    @dataclass

    class Sales:

        conn: DBConnection



        def get_sales()

            return self.conn.query("SELECT ...")



Now ``productspage`` and ``salespage`` methods can just declare

that they require these objects:



.. code-block:: python



    class MyWeb(Server):



        @route("/products")

        def productspage(self, request: Request, products: Products):

            ... # return the composed page



        @route("/sales")

        def salespage(self, request: Request, sales: Sales):

            ... # return the composed page



And the framework can then be responsible to fulfill these

dependencies. The flexibility offered would be a great advantage.

As an example, if would be very easy to create a page that requires

both sales and products:



.. code-block:: python



        @route("/overview")

        def productspage(self, request: Request,

                         products: Products, sales: Sales):

            ... # return the composed overview page



Contributing

============



* Source code: https://github.com/scrapinghub/andi

* Issue tracker: https://github.com/scrapinghub/andi/issues



Use tox_ to run tests with different Python versions::



    tox



The command above also runs type checks; we use mypy.



.. _tox: https://tox.readthedocs.io