{"id":18283791,"url":"https://github.com/cfmtech/python_pipeline_blog_post","last_synced_at":"2025-07-15T07:36:05.580Z","repository":{"id":66039484,"uuid":"435965690","full_name":"CFMTech/python_pipeline_blog_post","owner":"CFMTech","description":"How to test a Dask pipeline with Pytest","archived":false,"fork":false,"pushed_at":"2021-12-09T09:28:52.000Z","size":982,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-09T05:44:12.261Z","etag":null,"topics":["dask","fixtures","pipeline","pytest"],"latest_commit_sha":null,"homepage":"https://medium.com/capital-fund-management/advanced-testing-techniques-for-your-python-data-pipeline-with-dask-and-pytest-fixtures-622064867ef8","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CFMTech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-07T17:13:17.000Z","updated_at":"2024-06-02T18:06:30.000Z","dependencies_parsed_at":"2024-04-23T08:07:42.946Z","dependency_job_id":null,"html_url":"https://github.com/CFMTech/python_pipeline_blog_post","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CFMTech/python_pipeline_blog_post","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CFMTech%2Fpython_pipeline_blog_post","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CFMTech%2Fpython_pipeline_blog_post/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CFMTech%2Fpython_pipeline_blog_post/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CFMTech%2Fpython_pipeline_blog_post/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CFMTech","download_url":"https://codeload.github.com/CFMTech/python_pipeline_blog_post/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CFMTech%2Fpython_pipeline_blog_post/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265418438,"owners_count":23761817,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dask","fixtures","pipeline","pytest"],"created_at":"2024-11-05T13:10:55.863Z","updated_at":"2025-07-15T07:36:05.562Z","avatar_url":"https://github.com/CFMTech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Advanced Testing Techniques for your Python Data Pipeline with Dask and Pytest Fixtures\n=======================================================================================\nAchieve full coverage; Use fixtures and write tests easily; Refactor in confidence with non-regression tests\n------------------------------------------------------------------------------------------------------------\n\n![CI](https://github.com/mwouts/python_pipeline_blog_post/workflows/CI/badge.svg)\n[![codecov.io](https://codecov.io/github/mwouts/python_pipeline_blog_post/coverage.svg?branch=main)](https://codecov.io/gh/mwouts/python_pipeline_blog_post/branch/main)\n\n![](joshua-sortino-LqKhnDzSF-8-unsplash.jpg)\n(Photo by \u003ca href=\"https://unsplash.com/@sortino?utm_source=unsplash\u0026utm_medium=referral\u0026utm_content=creditCopyText\"\u003eJoshua Sortino\u003c/a\u003e on \u003ca href=\"https://unsplash.com/?utm_source=unsplash\u0026utm_medium=referral\u0026utm_content=creditCopyText\"\u003eUnsplash\u003c/a\u003e)\n\n[CFM](https://www.cfm.fr) is a Quantitative Hedge Fund with more than 30 years of experience in the domain of quantitative investing. Our daily production (and research) is done with a complex Python data pipeline.\n\nIn this post, we share our experience in writing tests for the pipeline. Our first objective was to improve the coverage of the pipeline. Then, we also wanted to improve the identification of the contributions that could break any production or research usage.\n\nThe techniques that we have used, and that we document in this article are:\n- Test the functions using fixtures\n- Generate fixtures for every node of the pipeline\n- Identify unexpected code impacts with non-regression tests\n- Put breakpoints programmatically and compare the arguments passed to a given function in two different versions of the pipeline, to refactor with confidence.\n\n# Introducing our Python Pipeline\n\nWe use a pipeline to best combine the research contributions of the different teams. For instance, we have tasks dedicated to\n- **data** streams (financial information like past prices, economic data, news, ...)\n- **signals**, derived from the data, that express numerically views on future returns, for one or more financial instruments\n- and many other tasks up to portfolio construction.\n\nA convenient way to schedule this tools is to use a pipeline. At CFM, we use an in-house Python utility for this, but for this post, I'll assume that our pipeline is implemented with [Dask](https://dask.org/) (see this video: [Next Generation Big Data Pipelines with Prefect and Dask](https://www.youtube.com/watch?v=R6z77ZNJvho) for a review of Dask and a few other alternatives). Also, for the sake of simplicity our sample pipeline is made of only four nodes:\n```python\nfrom dask.delayed import delayed, Delayed\n\nfrom .data import get_closes, get_volumes, get_yahoo_data\nfrom .signals import get_signals\n\n\ndef get_full_pipeline(tickers, start_date, end_date):\n    \"\"\"Return the full simulation pipeline\"\"\"\n    yahoo_data = delayed(get_yahoo_data)(\n        tickers, start_date, end_date, dask_key_name=\"yahoo_data\"\n    )\n    volumes = delayed(get_volumes)(yahoo_data, dask_key_name=\"volumes\")\n    closes = delayed(get_closes)(yahoo_data, dask_key_name=\"closes\")\n    signals = delayed(get_signals)(closes, volumes, dask_key_name=\"signals\")  # noqa\n\n    # Return a dict with all the nodes\n    return {name: task for name, task in locals().items() if isinstance(task, Delayed)}\n```\n\nWe can visualize the pipeline with `pipeline.visualize()`:\n\n![](https://github.com/CFMTech/python_pipeline_blog_post/raw/main/sample_pipeline/sample_pipeline/tests_2_fixtures_generated_with_the_pipeline/mydask.png)\n\nCombining the nodes into a pipeline is a great improvement compared to using a simple bash scheduler:\n- With the pipeline, running the pipeline in full is accessible to every user, since the dependencies are taken care of by the pipeline. The users do not need any more to have experience with all the tasks in the pipeline.\n- The nodes must be implemented with _[pure functions](https://en.wikipedia.org/wiki/Pure_function)_, that is, functions with an output that is deterministic given the inputs, and have no side effect. This is a very important requirement - not using pure functions will cause lots of complexity and extra maintenance work when a task must be regenerated.\n- The pipeline, being more Python-oriented, is also notebook friendly - isn't it easier to call a function than run a script in a Jupyter notebook?\n\n# Testing the pipeline and the nodes\n\nOur next objective after building our pipeline is to improve coverage. While coverage is not enough to make sure that you will detect problems with a new contribution, it is necessary. Without tests, the pipeline will often break in research.\n\nA simple way to improve coverage is to run the pipeline in full on the CI. Obviously, we don't use the full configuration - our test pipeline runs for just two portfolios, a handful of financial instruments, and extends over just a few days. Overall it runs in under one minute.\n\nBut we wanted to go a bit further. We have observed that many contributors find it difficult to write tests because preparing the inputs for the function to be tested is challenging. And many tests were long to write just because of the data preparation code:\n```python\ndef test_complex_function():\n    # complex and long code to prepare the sample inputs\n    tickers = ...\n    volumes = ...\n    closes = ...\n\n    # Call the function\n    res = complex_function(tickers, volumes, closes)\n\n    # Few asserts on the result\n    assert isinstance(res, pd.DataFrame)\n    assert set(res.columns) == set(tickers)\n    ...\n```\n\nWe first decided to separate the test input preparation from the test itself. We created `fixtures` (see below) that can be used in as many tests as we want. With fixtures at hand, writing a test becomes very easy - the example above becomes much shorter indeed:\n```python\n\ndef test_complex_function(tickers, volumes, closes):\n    # Call the function\n    res = complex_function(tickers, volumes, closes)\n\n    # Few asserts on the result\n    assert isinstance(res, pd.DataFrame)\n    assert set(res.columns) == set(tickers)\n    ...\n```\n\nAnother important point is that, within a pipeline, the inputs for a node are its parent nodes. So, we decided to generate the fixtures directly from the pipeline - but before discussing that, let us introduce the fixtures.\n\n## Introducing pytest fixtures\n\nA [pytest fixture](https://docs.pytest.org/en/6.2.x/fixture.html) is a function decorated with `pytest.fixture`. You can define the fixtures either directly in the test file, or in a [`conftest.py`](https://docs.pytest.org/en/6.2.x/fixture.html#conftest-py-sharing-fixtures-across-multiple-files) file in the same or in a parent directory.\n\nFor our first fixture example in the folder [tests_1_hand_written_fixtures](https://github.com/CFMTech/python_pipeline_blog_post/tree/main/sample_pipeline/sample_pipeline/tests_1_hand_written_fixtures), we create the fixtures by just calling the corresponding functions:\n\n```python\nimport pytest\n\nfrom sample_pipeline.data import get_closes, get_volumes, get_yahoo_data\n\n\n@pytest.fixture(scope=\"session\")\ndef start_date():\n    \"\"\"A sample start date for the pipeline\"\"\"\n    return \"2021-01-04\"\n\n\n@pytest.fixture(scope=\"session\")\ndef end_date():\n    \"\"\"A sample end date for the pipeline\"\"\"\n    return \"2021-01-29\"\n\n\n@pytest.fixture(scope=\"session\")\ndef tickers():\n    \"\"\"A sample list of tickers\"\"\"\n    return {\"AAPL\", \"MSFT\", \"AMZN\", \"GOOGL\"}\n\n\n@pytest.fixture(scope=\"session\")\ndef yahoo_data(tickers, start_date, end_date):\n    return get_yahoo_data(tickers, start_date, end_date)\n\n\n@pytest.fixture(scope=\"session\")\ndef closes(yahoo_data):\n    return get_closes(yahoo_data)\n\n\n@pytest.fixture(scope=\"session\")\ndef volumes(yahoo_data):\n    return get_volumes(yahoo_data)\n```\n\nNow we can use these fixtures in the test just by putting them as arguments to the test:\n\n```python\nfrom sample_pipeline.signals import get_signals\n\n\ndef test_get_signals(tickers, closes, volumes):\n    signals = get_signals(closes, volumes)\n    assert isinstance(signals, dict)\n    assert len(signals) \u003e= 2\n    for signal_name, signal in signals:\n        assert isinstance(signals, pd.DataFrame), signal_name\n        assert set(signal.columns) == tickers, signal_name\n```\n\n## Why use `scope=\"session\"`?\n\nOur test pipeline executes in full in under one minute, but we don't want to multiply this by the number of tests. With the `scope=\"session\"` option, we save a lot of time as the fixtures are generated just once (per worker, so they will still be generated multiple times if you use `pytest-xdist`).\n\nFor instance, if we launch the tests in `test_1_data.py`, you see that the fixtures are generated on demand, and just once for each of them (cf. the INFO logs).\n\n```\n============================= test session starts ==============================\ncollecting ... collected 3 items\n\ntest_1_data.py::test_get_yahoo_data\ntest_1_data.py::test_get_closes\ntest_1_data.py::test_get_volumes\n\n============================== 3 passed in 5.30s ===============================\n\nProcess finished with exit code 0\n\n-------------------------------- live log call ---------------------------------\nINFO     sample_pipeline.data:data.py:12 Loading price data from Yahoo finance\nPASSED                                                                   [ 33%]\n-------------------------------- live log setup --------------------------------\nINFO     sample_pipeline.data:data.py:12 Loading price data from Yahoo finance\n-------------------------------- live log call ---------------------------------\nINFO     sample_pipeline.data:data.py:33 Loading close prices\nPASSED                                                                   [ 66%]\n-------------------------------- live log call ---------------------------------\nINFO     sample_pipeline.data:data.py:40 Loading volumes\nPASSED                                                                   [100%]\n```\n\nIt is legitimate to use `scope=\"session\"` as we work with pure functions - as required by the pipeline. If you are not so sure that your functions are pure, and want to double-check that they do not modify their input by reference, you can do so in the fixture teardown:\n```python\nfrom copy import deepcopy\nfrom deepdiff import DeepDiff\n\n@pytest.fixture(scope=\"session\")\ndef closes(yahoo_data):\n    # Compute the fixture\n    value_org = get_closes(yahoo_data)\n\n    # Make a copy\n    value = deepcopy(value_org)\n\n    # Yield the copy and run all the selected tests\n    yield value\n\n    # In the fixture teardown, make sure that no test modified the value\n    assert not DeepDiff(value, value_org)\n```\n\nIn this example we have used [DeepDiff](https://zepworks.com/deepdiff/current/index.html) to show the recursive differences between two Python objects - this sounds like a great library, but I have to mention that I have no extended experience with it.\n\n## Do I need to duplicate the pipeline in the conftest.py?\n\nIn our first example ([tests_1_hand_written_fixtures](https://github.com/CFMTech/python_pipeline_blog_post/tree/main/sample_pipeline/sample_pipeline/tests_1_hand_written_fixtures)), we have actually re-implemented the pipeline in the `conftest.py` file. This causes duplication and will require specific maintenance when you change the pipeline, so we recommend this approach only for short and stable pipelines.\n\nIn the second folder [tests_2_fixtures_generated_with_the_pipeline](https://github.com/CFMTech/python_pipeline_blog_post/tree/main/sample_pipeline/sample_pipeline/tests_2_fixtures_generated_with_the_pipeline), we used another approach. We created a fixture for the evaluated pipeline - a dictionary with the value for every node - and then we exposed each node as a fixture.\n```python\n@pytest.fixture(scope=\"session\")\ndef evaluated_pipeline(tickers, start_date, end_date):\n    # The pipeline\n    full_pipeline = get_full_pipeline(tickers, start_date, end_date)\n\n    # Evaluate all the tasks\n    _compute = dask.compute(full_pipeline)\n\n    # The value returned by dask.compute is a tuple of one element\n    (_evaluated_pipeline,) = _compute\n\n    return _evaluated_pipeline\n\n\n@pytest.fixture(scope=\"session\")\ndef yahoo_data(evaluated_pipeline):\n    return evaluated_pipeline[\"yahoo_data\"]\n\n\n@pytest.fixture(scope=\"session\")\ndef closes(evaluated_pipeline):\n    return evaluated_pipeline[\"closes\"]\n\n\n@pytest.fixture(scope=\"session\")\ndef volumes(evaluated_pipeline):\n    return evaluated_pipeline[\"volumes\"]\n```\n\nThe advantages of that approach are:\n- Low maintenance: you just need to add or remove fixtures when a node is added to or removed from the pipeline. Changes on the node arguments are automatically replicated on the fixtures.\n- The pipeline is covered in full - no matter if some nodes are not used in the tests, they are evaluated\n\nBut it also has a few drawbacks:\n- The fixtures become _fragile_. Say you work on the `get_signals` function, start developing and introduce an error in the function... Because of this the pipeline cannot be evaluated anymore. So you cannot get a value for the fixtures, and you are not in a position to launch the test on `get_signals` anymore.\n- The fact that the pipeline is evaluated in full makes the creation of the fixtures a bit slower. Now the logs look like this:\n```\n============================= test session starts ==============================\ncollecting ... collected 2 items\n\ntest_2_data.py::test_get_closes\ntest_2_data.py::test_get_volumes\n\n============================== 2 passed in 3.16s ===============================\n\nProcess finished with exit code 0\n\n-------------------------------- live log setup --------------------------------\nINFO     sample_pipeline.data:data.py:12 Loading price data from Yahoo finance\nINFO     sample_pipeline.data:data.py:33 Loading close prices\nINFO     sample_pipeline.data:data.py:40 Loading volumes\nINFO     sample_pipeline.signals:signals.py:8 Computing signals\n-------------------------------- live log call ---------------------------------\nINFO     sample_pipeline.data:data.py:33 Loading close prices\nPASSED                                                                   [ 50%]\n-------------------------------- live log call ---------------------------------\nINFO     sample_pipeline.data:data.py:40 Loading volumes\nPASSED                                                                   [100%]\n```\nIn particular, we see that the signals are generated even if they are not used in the tests.\n\nAs a conclusion, this [tests_2_fixtures_generated_with_the_pipeline](https://github.com/CFMTech/python_pipeline_blog_post/tree/main/sample_pipeline/sample_pipeline/tests_2_fixtures_generated_with_the_pipeline) approach is good for the CI, but not for local development.\n\n## Generating the fixtures from a cached run of the pipeline\n\nThis is the approach that we use in practice. The pipeline is run in full on the CI, and for local developments, we save the results to a cache.\n\nOur sample implementation is available at [tests_3_fixtures_from_a_cached_pipeline](https://github.com/CFMTech/python_pipeline_blog_post/tree/main/sample_pipeline/sample_pipeline/tests_3_fixtures_from_a_cached_pipeline), and we cite a short extract here:\n```python\n@pytest.fixture(scope=\"session\")\ndef cached_pipeline_path(tickers, start_date, end_date, worker_id):\n    \"\"\"This fixture returns the path to the cached pipeline and evaluates the\n    pipeline if necessary.\n\n    worker_id: the id of the worker in pytest-xdist\n    (remove this argument if you don't use pytest-xdist)\n    \"\"\"\n    return get_cached_pipeline_path(tickers, start_date, end_date, worker_id)\n\n\n@pytest.fixture(scope=\"session\")\ndef yahoo_data(cached_pipeline_path):\n    return load_from_cache(cached_pipeline_path, \"yahoo_data\")\n```\n\nNote that the fixture `cached_pipeline_path` may not return instantly - it will evaluate and cache the full pipeline if necessary (e.g. if it executed on the CI, or if the user removed the local cache).\n\nThis approach has many advantages:\n- The fixtures are available instantaneously (they are loaded from disk, not computed)\n- We can develop freely and make breaking changes locally, that will not affect the fixtures generation (well, not until we decide to regenerate)\n- And we get full coverage of the pipeline on the CI.\n\nThe only disadvantage of this method is that the developer must be aware of the cache, and will need to know when to remove and regenerate it.\n\nThe first time we run the test suite, we see a mention that the cache is being generated, and from the second time on the log will point out to the cache being reused:\n```\n============================= test session starts ==============================\ncollecting ... collected 2 items\n\ntest_3_data.py::test_get_closes\ntest_3_data.py::test_get_volumes\n\n============================== 2 passed in 3.22s ===============================\n\nProcess finished with exit code 0\n\n-------------------------------- live log setup --------------------------------\nINFO     sample_pipeline.tests_3_fixtures_from_a_cached_pipeline:__init__.py:35 Regenerating the cached pipeline at /tmp/cached_pipeline/master at 2021-12-07 15:38:10.157451\nINFO     sample_pipeline.data:data.py:12 Loading price data from Yahoo finance\nINFO     sample_pipeline.data:data.py:33 Loading close prices\nINFO     sample_pipeline.data:data.py:40 Loading volumes\nINFO     sample_pipeline.signals:signals.py:8 Computing signals\n-------------------------------- live log call ---------------------------------\nINFO     sample_pipeline.data:data.py:33 Loading close prices\nPASSED                                                                   [ 50%]\n-------------------------------- live log call ---------------------------------\nINFO     sample_pipeline.data:data.py:40 Loading volumes\nPASSED                                                                   [100%]\n```\n\n## My pipeline has parameters. Should I write multiple conftests with different fixtures?\n\nWe recommend working with only **one** set of fixtures. Maintaining a pipeline of fixtures is an investment in code, in user training, so it is best if everyone knows what the sample pipeline is.\n\nWe do understand that some tests require specific inputs. When this is the case, we recommend to _derive_ custom fixtures from the reference ones.\n\nAssume for instance that some signal generation requires that `\"FB\"` be among the tickers. In that case, we can simply create a new fixture\n```python\n@pytest.fixture(scope=\"session\")\ndef tickers_including_fb(tickers):\n    return tickers + {\"FB\"}\n```\n\n⚠️Pay attention to not change the original fixture by reference, i.e. do `tickers + {\"FB\"}` but not `tickers.add('FB')`!\n\n## Non-regression tests\n\nWith the test fixtures documented above, we already get excellent coverage for the data pipeline. Still, this is not enough to ensure that the pipeline will work in practical applications.\n\nSo we added another kind of test to our platform, the non-regression tests. These tests are run with the complete portfolio configuration, for each portfolio that we have in production. We want a test that is not too slow (\u003c 5 minutes), so we don't cover the full data history but just one month. Also, we might not cover the full cartesian product of portfolios times tasks, but only the selection that most matters to us.\n\nOur sample non-regression test is coded at [tests_4_non_regression](https://github.com/CFMTech/python_pipeline_blog_post/tree/main/sample_pipeline/sample_pipeline/tests_4_non_regression), and the test itself is also reproduced below.\n\n```python\ndef non_regression_nodes_iterator():\n    for name, node in get_non_regression_pipeline().items():\n        # Skip the nodes for which you don't want a non-regression test\n        if \"slow\" in name:\n            continue\n        yield name, node\n\n\n@pytest.mark.parametrize(\"name,node\", non_regression_nodes_iterator())\ndef test_non_regression(name, node, non_regression_data):\n    \"\"\"For each node in the data pipeline, load the inputs from a\n    reference run, evaluate the node, and compare the new output with\n    the output from the reference run\"\"\"\n    expected = non_regression_data[name]\n\n    # Load the inputs for the given node from the reference non-reg data\n    inputs = {\n        input_name: non_regression_data[input_name]\n        for input_name in node.dask.dependencies[name]\n    }\n\n    # And evaluate the node given the inputs above\n    node_with_inputs_from_non_reg_data = Delayed(name, dict(node.dask, **inputs))\n    actual = node_with_inputs_from_non_reg_data.compute()\n\n    # ######################################\n    # ### There should be no difference! ###\n    # ######################################\n\n    diff = DeepDiff(actual, expected)\n    if diff:\n        raise ValueError(\n            f\"The value for {name} has changed. \"\n            f\"You can either revert the change, or, if you understand the new values, \"\n            f\"you can delete the non-regression file `non_regression_data.pickle` \"\n            f\"and regenerate it by running `test_regenerate_non_regression_data`.\\n\"\n            f\"Differences: {diff}\"\n        )\n```\n\nThe difference between the non-regression test and the previous test suite, are that\n- The non-regression test has better coverage of the production use cases (since it uses the production configuration)\n- It will detect any impact on the outputs of the nodes. Unlike the simple tests that we wrote before, we don't only check the shape of the outputs, but also their value.\n- It also takes more time to run but gives much more confidence in the updated code.\n\nIn our example, we saved the non-regression data into a simple file. When a non-regression occurs and is expected, that file must be updated with the new outputs (i.e. deleted, the framework will regenerate it). It is possible to save the non-regression data outside the project repository (i.e. on disk/url) if it is too big. In that case, make sure the non-regression data sets are incremental (i.e. use a new file name or URL for each new non-regression run), otherwise the non-regression tests on existing branches will break.\n\n## Refactor and test that the arguments passed to a certain function don't change\n\nWe will conclude this article with one last technique that we have found useful in the context of large refactorings. The objective is to guarantee that, after the refactoring, a given function is called with the exact same arguments as before (so, in particular, it will have the same outputs).\n\nOur technique is a bit comparable to a breakpoint that we would set programmatically, and that would export the arguments at that point of the program. We have implemented this with a _context manager_. The context manager intercepts the (first) call to the target function, stops the computation, without evaluating the target function, and returns the arguments of the call.\n\nWith this `intercept_function_arguments` context manager we can write tests like [test_same_arguments.py](https://github.com/CFMTech/python_pipeline_blog_post/tree/main/sample_pipeline/sample_pipeline/tests_5_intercept_function_arguments/test_same_arguments.py):\n```python\ndef test_same_arguments(new_pipeline, old_pipeline):\n    \"\"\"We test that the two versions of the pipeline result in identical\n    parameters passed to get_signals.\n\n    Note that we need to pass the path to the get_signals function that is\n    actually used by the pipelines (i.e. sample_pipeline.pipeline.get_signals\n    for the Dask pipeline)\n    \"\"\"\n    fun_path = \"sample_pipeline.signals.get_signals\"\n    args_old = {}\n    with intercept_function_arguments(fun_path, args_old):\n        old_pipeline()\n\n    fun_path = \"sample_pipeline.pipeline.get_signals\"\n    args_new = {}\n    with intercept_function_arguments(fun_path, args_new):\n        new_pipeline()\n\n    assert not DeepDiff(args_new, args_old)\n```\n\nA subtlety in the above is that the target function is patched using `mock.patch`, so you will have to be careful with imports. If you import the target function before entering the `intercept_function_arguments`, then `fun_path` should be the path where the function is imported, see the section on [where to patch](https://docs.python.org/3/library/unittest.mock.html#where-to-patch) in the standard library.\n\n# Conclusion\n\nWe hope this post will help you keep your data pipeline under control! As we have seen, creating a fixture for each task in the pipeline makes the writing of tests very easy. The non-regression tests are also super useful to identify unexpected impacts before a contribution gets accepted. And if you want to go further and guarantee that the inputs of a certain function don't change, then the `intercept_function_arguments` technique is all yours!\n\n# Acknowledgments\n\nThis article was written by [Marc Wouts](https://github.com/mwouts), a researcher at CFM, and the author of [Jupytext](https://github.com/mwouts/jupytext). Marc would like to thank the Portfolio team for the collaboration on the pipeline, the Open Source Program Office at CFM for the support on this article, [Florent Zara](https://twitter.com/flzara) for the time spent reading the many draft versions of this post, [Emmanuel Serie](https://github.com/eserie) and the [Dask Discourse Group](https://dask.discourse.group) for advice on Dask.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcfmtech%2Fpython_pipeline_blog_post","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcfmtech%2Fpython_pipeline_blog_post","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcfmtech%2Fpython_pipeline_blog_post/lists"}