{"id":17047609,"url":"https://github.com/residentmario/checkpoints","last_synced_at":"2025-04-12T15:31:35.353Z","repository":{"id":62561856,"uuid":"72119327","full_name":"ResidentMario/checkpoints","owner":"ResidentMario","description":"Partial result caching for pandas in Python.","archived":false,"fork":false,"pushed_at":"2019-04-26T04:16:41.000Z","size":63,"stargazers_count":19,"open_issues_count":2,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-26T10:11:25.654Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ResidentMario.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-27T15:00:18.000Z","updated_at":"2024-12-06T23:18:45.000Z","dependencies_parsed_at":"2022-11-03T15:15:23.571Z","dependency_job_id":null,"html_url":"https://github.com/ResidentMario/checkpoints","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fcheckpoints","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fcheckpoints/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fcheckpoints/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fcheckpoints/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ResidentMario","download_url":"https://codeload.github.com/ResidentMario/checkpoints/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248589567,"owners_count":21129636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T09:49:50.197Z","updated_at":"2025-04-12T15:31:35.041Z","avatar_url":"https://github.com/ResidentMario.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# checkpoints [![PyPi version](https://img.shields.io/pypi/v/checkpoints.svg)](https://pypi.python.org/pypi/checkpoints/) ![t](https://img.shields.io/badge/status-stable-green.svg)\n\n![demo](http://i.imgur.com/paxQ51Y.gif)\n\n`checkpoints` is a tiny module which imports new `pandas.DataFrame.safe_apply` and `pandas.Series.safe_map`\nexpressions, stop-and-start versions of the `pandas.DataFrame.apply` and `pandas.Series.map` operations which caches\npartial results in between runtimes in case an exception is thrown.\n\nThis means that the next time these functions are called, the operation will pick up back where it failed, instead\nof all the way back at the beginning of the map. After all, there's nothing more aggrevating than waiting ages for a\nprocess to complete, only to lose all of your data on the last iteration!\n\nJust `pip install checkpoints` to get started.\n\n## Why?\n\nFor a writeup with a practical example of what `checkpoints` can do for you see [this post on my personal blog](http://www.residentmar.io/2016/10/29/saving-progress-pandas.html).\n\n## Quickstart\n\nTo start, import `checkpoints` and enable it:\n\n    \u003e\u003e\u003e from checkpoints import checkpoints\n    \u003e\u003e\u003e checkpoints.enable()\n\nThis will augment your environment with `pandas.Series.safe_map` and `pandas.DataFrame.safe_apply` methods. Now\nsuppose we create a `Series` of floats, except for one invalid entry smack in the middle:\n\n    \u003e\u003e\u003e import pandas as pd; import numpy as np\n    \u003e\u003e\u003e rand = pd.Series(np.random.random(100))\n    \u003e\u003e\u003e rand[50] = \"____\"\n\nSuppose we want to remean this data. If we apply a naive `map`:\n\n    \u003e\u003e\u003e rand.map(lambda v: v - 0.5)\n\n        TypeError: unsupported operand type(s) for -: 'str' and 'float'\n\nNot only are the results up to that point lost, but we're also not actually told where the failure occurs! Using\n`safe_map` instead:\n\n    \u003e\u003e\u003e rand.safe_map(lambda v: v - 0.5)\n\n        \u003cROOT\u003e/checkpoint/checkpoints/checkpoints.py:96: UserWarning: Failure on index 50\n        TypeError: unsupported operand type(s) for -: 'str' and 'float'\n\nAll of the prior results are cached, and we can retrieve them at will with `checkpoints.results`:\n\n    \u003e\u003e\u003e checkpoints.results\n\n        0    -0.189003\n        1     0.337332\n        2    -0.143698\n        3    -0.312296\n        ...\n        47   -0.188995\n        48   -0.286550\n        49   -0.258107\n        dtype: float64\n\n`checkpoints` will store the partial results until either the process fully completes or it is explicitly told to get\n rid of them using `checkpoints.flush()`:\n\n    \u003e\u003e\u003e checkpoints.flush()\n    \u003e\u003e\u003e checkpoints.results\n        None\n\nYou can also induce this by passing a `flush=True` argument to `safe_map`.\n\n`pd.DataFrame.safe_apply` is similar:\n\n    \u003e\u003e\u003e rand = pd.DataFrame(np.random.random(100).reshape((20,5)))\n    \u003e\u003e\u003e rand[2][10] = \"____\"\n    \u003e\u003e\u003e rand.apply(lambda srs: srs.sum())\n\n        TypeError: unsupported operand type(s) for +: 'float' and 'str'\n\n    \u003e\u003e\u003e rand.safe_apply(lambda srs: srs.sum())\n\n        \u003cROOT\u003e/checkpoint/checkpoints/checkpoints.py:49: UserWarning: Failure on index 2\n        TypeError: unsupported operand type(s) for +: 'float' and 'str'\n\n    \u003e\u003e\u003e checkpoints.results\n\n        0    9.273607\n        1    8.259637\n        2    8.359239\n        3    7.873243\n        dtype: float64\n\nFinally, the disable checkpoints:\n\n    \u003e\u003e\u003e checkpoints.disable()\n\n## Performance\n\nMaintaining checkpoints introduces some overhead, but really not that much. `DataFrame` performance differs by a\nreasonably small constant factor, while `Series` performance is one-to-one:\n\n![Performance charts](http://i.imgur.com/jFIgXOG.png)\n\n## Technicals\n\nUnder the hood, `checkpoints` implements a [state machine](https://en.wikipedia.org/wiki/Finite-state_machine),\n`CheckpointStateMachine`, which uses a simple list to keep track of which entries have and haven't been mapped yet.\nThe function fed to `safe_*` is placed in a `wrapper` which redirects its output to a `results` list. When a map is\ninterrupted midway, then rerun, `safe_*` partitions the input, using the length of `results` to return to the first\nnon-outputted entry, and continues to run the `wrapper` on that slice.\n\nAn actual `pandas` object isn't generated until **all** entries have been mapped. At that point `results` is\nrepackaged into a `Series` or `DataFrame`, wiped, and a `pandas` object is returned, leaving `CheckpointStateMachine`\n ready to handle the next set of inputs.\n\n## Limitations\n\n* Another feature useful for long-running methods are progress bars, but as of now there is no way to integrate\n`checkpoints` with e.g. [`tqdm`](https://github.com/tqdm/tqdm). The workaround is to estimate the time cost of your\nprocess beforehand.\n* `pandas.DataFrame.safe_apply` jobs on functions returning `DataFrame` are not currently implemented, and will\nsimply return `None`. This means that e.g. the following will silently fail:\n\n    `\u003e\u003e\u003e pd.DataFrame({'a': [1, 2], 'b': [3, 4]}).safe_apply(lambda v: pd.DataFrame({'a': [1, 2], 'b': [2, 3]}))`\n\n\n* The `Series.map` `na_action` parameter is not implemented; nor are any of `broadcast`, `raw`, or `reduce` for\n`DataFrame.apply`.\n\n## See also\n\n`checkpoints` provides a form of [defensive programming](https://en.wikipedia.org/wiki/Defensive_programming). If\nyou're a fan of this sort of thing, you should also check out [`engarde`](https://github.com/TomAugspurger/engarde).\n\n## Contributing\n\nBugs? Thoughts? Feature requests? [Throw them at the bug tracker and I'll take a look](https://github.com/ResidentMario/checkpoints/issues).\n\nAs always I'm very interested in hearing feedback\u0026mdash;reach out to me at `aleksey@residentmar.io`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresidentmario%2Fcheckpoints","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fresidentmario%2Fcheckpoints","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresidentmario%2Fcheckpoints/lists"}