{"id":19353415,"url":"https://github.com/foreverwintr/metafunctions","last_synced_at":"2026-02-09T23:32:11.840Z","repository":{"id":21636569,"uuid":"93481713","full_name":"ForeverWintr/metafunctions","owner":"ForeverWintr","description":"metafunctions is a function composition library for python.","archived":false,"fork":false,"pushed_at":"2023-01-10T18:41:35.000Z","size":411,"stargazers_count":52,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-02T10:11:55.879Z","etag":null,"topics":["function-composition","functional-programming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ForeverWintr.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-06T05:59:34.000Z","updated_at":"2025-04-01T07:58:40.000Z","dependencies_parsed_at":"2023-01-13T21:36:16.100Z","dependency_job_id":null,"html_url":"https://github.com/ForeverWintr/metafunctions","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ForeverWintr%2Fmetafunctions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ForeverWintr%2Fmetafunctions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ForeverWintr%2Fmetafunctions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ForeverWintr%2Fmetafunctions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ForeverWintr","download_url":"https://codeload.github.com/ForeverWintr/metafunctions/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250391272,"owners_count":21422869,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["function-composition","functional-programming"],"created_at":"2024-11-10T04:42:52.788Z","updated_at":"2026-02-09T23:32:11.810Z","avatar_url":"https://github.com/ForeverWintr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MetaFunctions\n![GithubActions Badge](https://github.com/ForeverWintr/metafunctions/actions/workflows/test.yml/badge.svg)\n[![Codecov](https://codecov.io/gh/ForeverWintr/metafunctions/coverage.svg?branch=master)](https://codecov.io/gh/ForeverWintr/metafunctions)\n\n\n\n## Metafunctions is a function composition and data pipelining library.\nIt allows for data pipeline creation separate from execution, so instead of writing:\n\n```python\nresult = step3(step2(step1(data)))\n\n#or\nresult_1 = step1(data)\nresult_2 = step2(result_1)\nresult_3 = step3(result_2)\n```\n\nYou can write:\n\n```python\npipeline = step1 | step2 | step3\nresult = pipeline(data)\n```\n\n### Why do you need new syntax for pipelining functions?\nWell you may not *need* a new syntax, but the ability to compose a data pipeline before executing it does impart some advantages, such as:\n\n* **Reuse**. Compose a pipeline once and use it in multiple places, including as part of larger pipelines:\n  ```python\n  # load, parse, clean, validate, and format are functions\n  preprocess = load | parse | clean | validate | format\n\n  # preprocess is now a MetaFunction, and can be reused\n  clean_data1 = preprocess('path/to/data/file')\n  clean_data2 = preprocess('path/to/different/file')\n\n  # Preprocess can be included in larger pipelines\n  pipeline = preprocess | step1 | step2 | step3\n  ```\n* **Readability**. `step1 | step2 | step3` is both read and executed from left to right, unlike `step3(step2(step1()))`, which is executed from innermost function outwards.\n* **Inspection**. Can't remember what your MetaFunction does? `str` will tell you:\n  ```python\n  \u003e\u003e\u003e str(preprocess)\n  \"(load | parse | clean | validate | format)\"\n* **Advanced Composition**. Anything beyond simple function chaining becomes difficult using traditional methods. What if you want to send the result of `step1` to both steps `2` and `3`, then sum the results? The traditional approach requires an intermediate variable and can quickly become unwieldy:\n  ```python\n  result1 = step1(data)\n  result2 = step2(result1) + step3(result1)\n  ```\n  Using metafunctions, you can declare a pipeline that does the same thing:\n  ```python\n  pipeline = step1 | step2 + step3\n  result = pipeline(data)\n  ```\n\n## Installation\n\nMetaFunctions supports python 3.5+ (tested to 3.10+)\n\n`pip install metafunctions`\n\n## How does it work?\n\nConceptually, a MetaFunction is a function that contains other functions. When you call a MetaFunction, the MetaFunction calls the functions it contains.\n\nYou can create a MetaFunction using the `node` decorator:\n```python\nfrom metafunctions import node\n\n@node\ndef get_name(prompt):\n    return input(prompt)\n\n@node\ndef say_hello(name):\n    return 'Hello {}!'.format(name)\n```\n\nMetaFunctions override certain operators to allow for composition. For example, the following creates a new MetaFunction that combines `get_name` and `say_hello`:\n```python\ngreet = get_name | say_hello\n```\n\nWhen we call the `greet` MetaFunction, it calls both its internal functions in turn.\n```python\n# First, `get_name` is called, which prints our prompt to the screen.\n# If we enter 'Tom' at the prompt, the second function returns the string 'Hello Tom!'\ngreeting = greet('Please enter your name ')\nprint(greeting) # Hello Tom!\n```\n\nMetaFunctions are also capable of upgrading regular functions to MetaFunctions at composition time, so we can simplify our example by composing `say_hello` directly with the builtin `input` and `print` functions:\n```python\n\u003e\u003e\u003e greet = input | say_hello | print\n\u003e\u003e\u003e greet('Please enter your name: ')\n# Please enter your name: Tom\n# Hello Tom!\n```\n\n## Features\n\n### Helpful Tracebacks\n\nErrors in composed functions can be confusing. If an exception occurs in a MetaFunction, the exception traceback will tell you which function the exception occurred in. But what if that function appears multiple times in the data pipeline?\n\nImagine this function, which downloads stringified numeric data from a web api:\n\n```python\n\u003e\u003e\u003e compute_value = (query_volume | float) * (query_price | float)\n\u003e\u003e\u003e compute_value('http://prices.com/123')\n```\n\nHere we've assumed that `query_volume` and `query_price` will return strings that convert cleanly to floats, but what if something goes wrong?\n\n```\n\u003e\u003e\u003e compute_value('http://prices.com/123')\nTraceback (most recent call last):\n  File \"\u003cstdin\u003e\", line 1, in \u003cmodule\u003e\n  ...\nbuiltins.ValueError: could not convert string to float: '$800'\n```\n\nWe can deduce that float conversion failed, but *which* float function raised the exception? MetaFunctions addresses this by adding the `locate_error` metafunction decorator, which adds a location information string to any exception raised within the pipeline:\n\n```\n\u003e\u003e\u003e from metafunctions import locate_error\n\u003e\u003e\u003e with_location = locate_error(compute_value)\n\u003e\u003e\u003e with_location('http://prices.com/123')\nTraceback (most recent call last):\n  File \"\u003cstdin\u003e\", line 1, in \u003cmodule\u003e\n  ...\nbuiltins.ValueError: could not convert string to float: '$800'\nOccured in the following function: ((query_volume | float) + (query_price | -\u003efloat\u003c-))\n```\n\n### Advanced Pipeline Construction Tools\n\nMetafunctions provides utilities for constructing advanced function pipelines.\n\n* **store** / **recall**: store the output of the preceding function, and recall it later to pass to a different function. For example:\n  \n  ```python\n  # The following pipeline sends the output of `a` to `b`, and also adds it to the output of `c`\n  p = a | store('a') | b | recall('a') + c\n  ```\n\n* **mmap**: A MetaFunction decorator that wraps a function (or MetaFunction) and calls it once per item in the input it receives. This allows you to  create loops in function pipelines:\n  \n  ```python\n  (1, 2, 3) | mmap(process) # \u003c- equivalent to (1, 2, 3) | (process \u0026 process \u0026 process)\n  ```\n  `mmap` duplicates the behaviour of the builtin [`map`](https://docs.python.org/3/library/functions.html#map) function.\n\n* **star**: Calls the wrapped MetaFunction with *args instead of args (It's analogous to `lambda args, **kwargs: metafunction(*args, **kwargs)`). This allows you to incorporate functions that accept more than one parameter into your function pipeline:\n\n  ```python\n  @node\n  def f(result1, result2):\n      ...\n\n  # When cmp is called, f will receive the results of both a and b as positional args\n  cmp = (a \u0026 b) | star(f)\n  ```\n  `star` can be combined with the above `mmap` to duplicate the behaviour of [`itertools.starmap`](https://docs.python.org/3/library/itertools.html#itertools.starmap): \n\n  ```python\n  starmap = star(mmap(f))\n  ```\n\n  For more discussion of `star`, see [this pull request](https://github.com/ForeverWintr/metafunctions/pull/9)\n\n* **concurrent**:\n  *experimental, requires an os that provides `os.fork()`*\n\n  Consider the following long running MetaFunction:\n\n  ```python\n  process_companies = get_company_data | filter | process\n  process_customers = get_customer_data | filter | process\n\n  process_all = process_companies \u0026 process_customers\n  ```\n\n  Assuming the component functions in the `process_all` MetaFunction follow good functional practices and do not have side effects, it's easy to see that `process_companies` and `process_customers` are independent of each other. If that's the case, we can safely execute them in parallel. The `concurrent` metafunction decorator allows you to specify steps in the function pipeline to execute in parallel:\n\n  ```python\n  from metafunctions import concurrent\n\n  do_large_calculation_async = concurrent(process_companies + process_customers)\n  ```\n\n  `concurrent` can be combined with `mmap` to create an asynchronous map, similar to [`multiprocessing.pool.map`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map): \n  \n  ```python\n  map_async = concurrent(mmap(f))\n  ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fforeverwintr%2Fmetafunctions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fforeverwintr%2Fmetafunctions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fforeverwintr%2Fmetafunctions/lists"}