{"id":13467593,"url":"https://github.com/alex-sherman/deco","last_synced_at":"2025-10-25T15:52:16.133Z","repository":{"id":37580100,"uuid":"55726144","full_name":"alex-sherman/deco","owner":"alex-sherman","description":null,"archived":false,"fork":false,"pushed_at":"2021-11-03T22:59:55.000Z","size":83,"stargazers_count":1574,"open_issues_count":6,"forks_count":50,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-03-02T06:04:11.050Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alex-sherman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-04-07T20:33:20.000Z","updated_at":"2025-02-17T16:46:23.000Z","dependencies_parsed_at":"2022-08-29T10:11:13.753Z","dependency_job_id":null,"html_url":"https://github.com/alex-sherman/deco","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-sherman%2Fdeco","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-sherman%2Fdeco/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-sherman%2Fdeco/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-sherman%2Fdeco/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alex-sherman","download_url":"https://codeload.github.com/alex-sherman/deco/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245584443,"owners_count":20639549,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T15:00:58.308Z","updated_at":"2025-10-25T15:52:16.052Z","avatar_url":"https://github.com/alex-sherman.png","language":"Python","funding_links":[],"categories":["Python","Python decorator in the wild"],"sub_categories":[],"readme":"Decorated Concurrency\n===========\n\nA simplified parallel computing model for Python.\nDECO automatically parallelizes Python programs, and requires minimal modifications to existing serial programs.\n\nInstall using pip:\n\n```\npip install deco\n```\n\nDocumentation\n--------------\nYou can reference the [Wiki on Github](https://github.com/alex-sherman/deco/wiki) for slightly more in-depth documentation.\n\nGeneral Usage\n---------------\n\nUsing DECO is as simple as finding, or creating, two functions in your Python program.\nThe first function is the one we want to run in parallel, and is decorated with `@concurrent`.\nThe second function is the function which calls the `@concurrent` function and is decorated with `@synchronized`.\nDecorating the second function is optional, but provides some very cool benefits.\nLet's take a look at an example.\n\n\n```python\n@concurrent # We add this for the concurrent function\ndef process_lat_lon(lat, lon, data):\n  #Does some work which takes a while\n  return result\n\n@synchronized # And we add this for the function which calls the concurrent function\ndef process_data_set(data):\n  results = defaultdict(dict)\n  for lat in range(...):\n    for lon in range(...):\n      results[lat][lon] = process_lat_lon(lat, lon, data)\n  return results\n```\n\nThat's it, two lines of changes is all we need in order to parallelize this program.\nNow this program will make use of all the cores on the machine it's running on, allowing it to run significantly faster.\n\nWhat it does\n-------------\n\n  - The `@concurrent` decorator uses multiprocessing.pool to parallelize calls to the target function\n  - Indexed based mutation of function arguments is handled automatically, which pool cannot do\n  - The `@synchronized` decorator automatically inserts synchronization events \n  - It also automatically refactors assignments of the results of `@concurrent` function calls to happen during synchronization events\n\nLimitations\n-------------\n  - The `@concurrent` decorator will only speed up functions that take longer than ~1ms\n    - If they take less time your code will run slower!\n  - By default, `@concurrent` function arguments/return values must be pickleable for use with `multiprocessing`\n  - The `@synchronized` decorator only works on 'simple' functions, make sure the function meets the following criteria\n    - Only calls, or assigns the result of `@concurrent` functions to indexable objects such as:\n      - concurrent(...)\n      - result[key] = concurrent(...)\n    - Never indirectly reads objects that get assigned to by calls of the `@concurrent` function\n\nHow it works\n-------------\n\nFor an in depth discussion of the mechanisms at work, we wrote a paper for a class\nwhich [can be found here](https://drive.google.com/file/d/0B_olmC0u8E3gWTBmN3pydGxHdEE/view?usp=sharing\u0026resourcekey=0-9aUctXy9Hn5g9SIul4kbVw).\n\nAs an overview, DECO is mainly just a smart wrapper for Python's multiprocessing.pool.\nWhen `@concurrent` is applied to a function it replaces it with calls to pool.apply_async.\nAdditionally when arguments are passed to pool.apply_async, DECO replaces any index mutable objects with proxies, allowing it to detect and synchronize mutations of these objects.\nThe results of these calls can then be obtained by calling wait() on the concurrent function, invoking a synchronization event.\nThese events can be placed automatically in your code by using the `@synchronized` decorator on functions that call `@concurrent` functions.\nAdditionally while using `@synchronized`, you can directly assign the result of concurrent function calls to index mutable objects.\nThese assignments get refactored by DECO to automatically occur during the next synchronization event.\nAll of this means that in many cases, parallel programming using DECO appears exactly the same as simpler serial programming.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falex-sherman%2Fdeco","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falex-sherman%2Fdeco","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falex-sherman%2Fdeco/lists"}