{"id":13795979,"url":"https://github.com/bakwc/PySyncObj","last_synced_at":"2025-05-13T00:30:37.281Z","repository":{"id":6574870,"uuid":"50359688","full_name":"bakwc/PySyncObj","owner":"bakwc","description":"A library for replicating your python class between multiple servers, based on raft protocol","archived":false,"fork":false,"pushed_at":"2023-10-27T08:54:06.000Z","size":594,"stargazers_count":681,"open_issues_count":31,"forks_count":106,"subscribers_count":42,"default_branch":"master","last_synced_at":"2024-05-06T20:44:57.152Z","etag":null,"topics":["distributed-systems","fault-tolerance","python","raft","raft-protocol","replication"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakwc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-25T15:26:03.000Z","updated_at":"2024-06-18T13:59:01.071Z","dependencies_parsed_at":"2024-06-18T13:59:00.675Z","dependency_job_id":"fbe01036-9a69-4d28-9b99-887441e53fdc","html_url":"https://github.com/bakwc/PySyncObj","commit_stats":{"total_commits":336,"total_committers":20,"mean_commits":16.8,"dds":"0.25595238095238093","last_synced_commit":"ee0a3a8128fee9770ba4173a557bb276a718caaa"},"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakwc%2FPySyncObj","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakwc%2FPySyncObj/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakwc%2FPySyncObj/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakwc%2FPySyncObj/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakwc","download_url":"https://codeload.github.com/bakwc/PySyncObj/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225159843,"owners_count":17430190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","fault-tolerance","python","raft","raft-protocol","replication"],"created_at":"2024-08-03T23:01:04.816Z","updated_at":"2025-05-13T00:30:37.263Z","avatar_url":"https://github.com/bakwc.png","language":"Python","readme":"# PySyncObj\n\n[![Build Status][tests-image]][tests] [![Windows Build Status][appveyor-image]][appveyor] [![Coverage Status][coverage-image]][coverage] [![Release][release-image]][releases] [![License][license-image]][license] [![gitter][gitter-image]][gitter] [![docs][docs-image]][docs]\n\n[tests-image]: https://github.com/bakwc/PySyncObj/actions/workflows/tests.yaml/badge.svg\n[tests]: https://github.com/bakwc/PySyncObj/actions/workflows/tests.yaml\n\n[appveyor-image]: https://ci.appveyor.com/api/projects/status/github/bakwc/pysyncobj?branch=master\u0026svg=true\n[appveyor]: https://ci.appveyor.com/project/bakwc/pysyncobj\n\n[coverage-image]: https://coveralls.io/repos/github/bakwc/PySyncObj/badge.svg?branch=master\n[coverage]: https://coveralls.io/github/bakwc/PySyncObj?branch=master\n\n[release-image]: https://img.shields.io/badge/release-0.3.14-blue.svg?style=flat\n[releases]: https://github.com/bakwc/PySyncObj/releases\n\n[license-image]: https://img.shields.io/badge/license-MIT-blue.svg?style=flat\n[license]: LICENSE.txt\n\n[gitter-image]: https://badges.gitter.im/bakwc/PySyncObj.svg\n[gitter]: https://gitter.im/bakwc/PySyncObj?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge\n\n[docs-image]: https://readthedocs.org/projects/pysyncobj/badge/?version=latest\n[docs]: http://pysyncobj.readthedocs.io/en/latest/\n\nPySyncObj is a python library for building fault-tolerant distributed systems. It provides the ability to replicate your application data between multiple servers. It has following features:\n\n- [raft protocol](http://raft.github.io/) for leader election and log replication\n- Log compaction - it use fork for copy-on-write while serializing data on disk\n- Dynamic membership changes - you can do it with [syncobj_admin](https://github.com/bakwc/PySyncObj/wiki/syncobj_admin) utility or [directly from your code](https://github.com/bakwc/PySyncObj/wiki/Dynamic-membership-change)\n- [Zero downtime deploy](https://github.com/bakwc/PySyncObj/wiki/Zero-downtime-deploy) - no need to stop cluster to update nodes\n- In-memory and on-disk serialization - you can use in-memory mode for small data and on-disk for big one\n- Encryption - you can set password and use it in external network\n- Python2 and Python3 on linux, macos and windows - no dependencies required (only optional one, eg. cryptography)\n- Configurable event loop - it can works in separate thread with it's own event loop - or you can call onTick function inside your own one\n- Convenient interface - you can easily transform arbitrary class into a replicated one (see example below).\n\n## Content\n * [Install](#install)\n * [Basic Usage](#usage)\n * [\"Batteries\"](#batteries)\n * [API Documentation](http://pysyncobj.readthedocs.io)\n * [Performance](#performance)\n * [Publications](#publications)\n\n## Install\nPySyncObj itself:\n```bash\npip install pysyncobj\n```\nCryptography for encryption (optional):\n```bash\npip install cryptography\n```\n\n## Usage\nConsider you have a class that implements counter:\n```python\nclass MyCounter(object):\n\tdef __init__(self):\n\t\tself.__counter = 0\n\n\tdef incCounter(self):\n\t\tself.__counter += 1\n\n\tdef getCounter(self):\n\t\treturn self.__counter\n```\nSo, to transform your class into a replicated one:\n - Inherit it from SyncObj\n - Initialize SyncObj with a self address and a list of partner addresses. Eg. if you have `serverA`, `serverB` and `serverC` and want to use 4321 port, you should use self address `serverA:4321` with partners `[serverB:4321, serverC:4321]` for your application, running at `serverA`; self address `serverB:4321` with partners `[serverA:4321, serverC:4321]` for your application at `serverB`; self address `serverC:4321` with partners `[serverA:4321, serverB:4321]` for app at `serverC`.\n - Mark all your methods that modifies your class fields with `@replicated` decorator.\nSo your final class will looks like:\n```python\nclass MyCounter(SyncObj):\n\tdef __init__(self):\n\t\tsuper(MyCounter, self).__init__('serverA:4321', ['serverB:4321', 'serverC:4321'])\n\t\tself.__counter = 0\n\n\t@replicated\n\tdef incCounter(self):\n\t\tself.__counter += 1\n\n\tdef getCounter(self):\n\t\treturn self.__counter\n```\nAnd thats all! Now you can call `incCounter` on `serverA`, and check counter value on `serverB` - they will be synchronized.\n\n## Batteries\nIf you just need some distributed data structures - try built-in \"batteries\". Few examples:\n### Counter \u0026 Dict\n```python\nfrom pysyncobj import SyncObj\nfrom pysyncobj.batteries import ReplCounter, ReplDict\n\ncounter1 = ReplCounter()\ncounter2 = ReplCounter()\ndict1 = ReplDict()\nsyncObj = SyncObj('serverA:4321', ['serverB:4321', 'serverC:4321'], consumers=[counter1, counter2, dict1])\n\ncounter1.set(42, sync=True) # set initial value to 42, 'sync' means that operation is blocking\ncounter1.add(10, sync=True) # add 10 to counter value\ncounter2.inc(sync=True) # increment counter value by one\ndict1.set('testKey1', 'testValue1', sync=True)\ndict1['testKey2'] = 'testValue2' # this is basically the same as previous, but asynchronous (non-blocking)\nprint(counter1, counter2, dict1['testKey1'], dict1.get('testKey2'))\n```\n### Lock\n```python\nfrom pysyncobj import SyncObj\nfrom pysyncobj.batteries import ReplLockManager\n\nlockManager = ReplLockManager(autoUnlockTime=75) # Lock will be released if connection dropped for more than 75 seconds\nsyncObj = SyncObj('serverA:4321', ['serverB:4321', 'serverC:4321'], consumers=[lockManager])\nif lockManager.tryAcquire('testLockName', sync=True):\n  # do some actions\n  lockManager.release('testLockName')\n```\nYou can look at [batteries implementation](https://github.com/bakwc/PySyncObj/blob/master/pysyncobj/batteries.py), [examples](https://github.com/bakwc/PySyncObj/tree/master/examples) and [unit-tests](https://github.com/bakwc/PySyncObj/blob/master/test_syncobj.py) for more use-cases. Also there is an [API documentation](http://pysyncobj.readthedocs.io). Feel free to create proposals and/or pull requests with new batteries, features, etc. Join our [gitter chat](https://gitter.im/bakwc/PySyncObj) if you have any questions.\n\n\n## Performance\n![15K rps on 3 nodes; 14K rps on 7 nodes;](http://pastexen.com/i/Ge3lnrM1OY.png \"RPS vs Cluster Size\")\n![22K rps on 10 byte requests; 5K rps on 20Kb requests;](http://pastexen.com/i/0RIsrKxJsV.png \"RPS vs Request Size\")\n\n## Publications\n- [Adventures in fault tolerant alerting with Python](https://blog.hostedgraphite.com/2017/05/05/adventures-in-fault-tolerant-alerting-with-python/)\n- [Строим распределенную систему c PySyncObj](https://habrahabr.ru/company/wargaming/blog/301398/)\n","funding_links":[],"categories":["Django","DevOps Utilities"],"sub_categories":["Repositories"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakwc%2FPySyncObj","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakwc%2FPySyncObj","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakwc%2FPySyncObj/lists"}