Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/aakropotkin/laika

Nix fetcher abstractions.
https://github.com/aakropotkin/laika

Last synced: 3 months ago
JSON representation

Nix fetcher abstractions.

Awesome Lists containing this project

README 

        
#+TITLE: Laika



Nix fetcher abstractions.



* Why?

Nix and Nixpkgs offer a wide variety of fetchers with a lot of overlapping

functionality, but seemingly minor differences between these can drastically

effect performance at scale. 



For example the use case that led to the creation of this utility was

wanting an abstract frontend that was compliant with Nixpkgs while allowing

common fetch instances to be swapped for more efficient builtins when creating

non-Nixpkgs derivations.

In practice this means you have common fetcher that supports legacy Nix

using Nixpkgs fetchers, while swapping in faster builtin alternatives in later

releases of Nix.



Additionally these abstractions allowed me to dynamically /prefetch/ SHA512

integrities to yield CA derivations that use SHA256 using IFD.

This may seem minor, but you might be surprised to learn that different

sha algos will yield duplicated store paths.

For the sources themselves this isn't a big deal, but any input addressed

derivations that depend on those sources will set off rebuilds if you try to

fetch later with a flake input or optimized builtin ( backed by SHA256 ).

Now I can avoid duplicate chains of non-CA derivations even if I use a

different hash algo to fetch.



Finally, from UX perspective, having these abstractions saves new users from

needing to know the nitty gritty implementation details of each fetcher, and

just call a generic wrapper that consumers can optimize for their use case.

If you prefer derivations vs builtins for different cache performance you

can simply toggle a config option without needing to use `scopedImport` or

complex overlays. 



* Wrapper Features

** Function Argument Reading

Laika provides wrapped forms of Nix's builtin fetchers which allow routines

such as =lib.functionArgs= to "do what they're supposed to do".



These work by pushing builtins into an attrset with a =__functor= and

=__functionArgs= field so that the routine is called normally, while

providing the argument info needed by =lib.functionArgs=.



*** Polymorphic Arguments

:PROPERTIES:

:ID:       42f15723-f75c-4f02-9938-49d9a9ebd361

:END:

Many builtin functions are polymorphic, and our =__functionArgs= field

aims to list the most permissive collection of arguments accepted.

If you want to know specific combinations of acceptable args please

refer to the type signatures defined in =__functionMeta.signature=.



Additionally, the =__functionArgs= field only lists arguments for

attrset arguments.

In cases where other/multiple types of arguments such as =string= are

accepted, you will need to refer to =__functionMeta.signature= to get

insight on acceptable arguments to be passed.

See the section [[Type Checking]] for more info.



Please note that some =__functionArgs= values are effected by runtime

context, for example =pure= mode.

See the section [[Dynamic Context]] for more info.



** Type Checking

Wrappers carry a =__functionMeta.signature= field which contains a list of

YANTS types ( provided by the =ak-nix= fork: =ytypes= ) representing the

function "prototype".



For example for the function =inc= our signature will be =[int int]=,

meaning "a function which accepts an integer and returns an integer".



Another example: =sum= would have the signature =[int int int]=, indicating

a function which accepts two integers and returns an integer.



This function signature is a useful tool, not only for type checking, but

also for documentation and "auto-calling" functions with correct

combinations of arguments.



Many of Nix's builtins are polymorphic, for example =builtins.fetchGit=

which accepts either a string or an attrset, and these signatures help us

keep track of our available argument combinations.



*** Enabling/Disabling

Runtime type-checking can be enabled/disabled across =laika= or on a per

function basis to suit the performance or strictness requirements your use

case calls for.



Personally I find these type-checks most useful in =nix flake check=

contexts or test suits, while in "production" after we already know a

locked build will succeed I turn them off to speed up evaluation for

package consumers.



Each fetcher defined in [[./src]] accepts a boolean argument =typecheck=

which can be overridden to control this feature.

Even if type checking is disabled, the field =__functionMeta.signature=

will still exist for reference by other utilities.



** Dynamic Context

Many builtin functions are effected by runtime environment settings, for

example =pure= mode, and our wrappers



These settings are always controlled by a boolean argument at the top of

each fetcher's definition. 

To make this polymorphism more opaque, any time a function is effected by

context this will be recorded in =__functionMeta.properties.*= tags.

For example an instance of a fetcher for impure mode will

set ~pure = false~.



* Wrapper Conventions

** Helper Functions as Fields

All wrappers are implemented as /functors/, meaning attrsets which can be

used as functions with =self= reference.

For folks that aren't familiar with /Functional Programming/ a useful

comparison is an /Object/ in JavaScript, or a /Class/ in C++ or Python.



Because functors have a single entry point, =__functor= which can read

arbitrary fields in =self=, we have the ability to split up routines into

helper functions stored as fields.



This is largely done for organization, but another useful advantage to

using these fields is that we can easily override or modify parts of an

existing function to repurpose it.

You can think of this as being similar to overriding =buildPhase= or adding

a =preInstall= hook in a =nixpkgs#stdenv.mkDerivation= attrset.



In practice almost all of the fetchers in =laika= use the same =__functor=

definition, and their differences in behavior is handled in helpers - this

is what allows us to easily add new "layers" or disable features such as

type checking across the codebase with a single setting.



*** Example

An example of these helpers in action is this set which handles polymorphic

arguments for =builtins.fetchGit=, and injection of a custom hook.

This is a condensed form of the real definition, but a useful reference nonetheless. 



#+BEGIN_SRC nix

  { lib }:

  let

    pp = lib.generators.toPretty {};

    fetchGitW = {

      __innerFunction = builtins.fetchGit;

      # Allows `lib.functionArgs' and `lib.canPassStrict' to work.

      __functionArgs = { url = false; allRefs = true; /* ... */ };

      # Stashes "auto-args".

      __thunk = { submodules = false; shallow = false; allRefs = false; };

      # Convert strings to attrs, and add our auto-args.

      __processArgs = self: x: let

        args = if builtins.isString x then { url = x; } else x;

        # Filter args down to what is accepted by `builtins.fetchGit'.

        # This is implemented the same way as

        # `nixpkgs#lib.callPackageWith' except we omit the `override' and

        # `overrideDerivation' additions.

      in lib.canPassStrict self ( self.__thunk // args );

      # The function entry point ends up being minimal.

      __functor = self: x:

        self.__innerFunction ( self.__processArgs self x );

    };

    # Customize the fetcher with new auto-args and add logging.

    myGitFetcher = fetchGitW // {

      __thunk = fetchGitW.__thunk // { allRefs = true; };

      # Hook the arg processor with a trace, then return the args produced

      # by the original `fetchGitW'.

      __processArgs = self: x: let

        args = fetchGitW.__processArgs self x;  # parent/super call.

      in builtins.trace ''

        myGitFetcher ${pp args};

      '' args;

    };

  in myGitFetcher "git+ssh://[email protected]/aakropotkin/laika.git"

#+END_SRC