{"id":13662108,"url":"https://github.com/grofit/persistity","last_synced_at":"2026-03-16T16:31:29.566Z","repository":{"id":19031371,"uuid":"84936135","full_name":"grofit/persistity","owner":"grofit","description":"A persistence framework for game developers","archived":false,"fork":false,"pushed_at":"2025-01-10T15:14:22.000Z","size":6453,"stargazers_count":36,"open_issues_count":2,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-27T11:22:02.500Z","etag":null,"topics":["binary","etl","json","serialization","unity","unity3d","xml"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grofit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-14T10:27:18.000Z","updated_at":"2025-01-10T15:14:26.000Z","dependencies_parsed_at":"2022-09-13T10:51:42.336Z","dependency_job_id":null,"html_url":"https://github.com/grofit/persistity","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grofit%2Fpersistity","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grofit%2Fpersistity/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grofit%2Fpersistity/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grofit%2Fpersistity/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grofit","download_url":"https://codeload.github.com/grofit/persistity/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243982141,"owners_count":20378605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","etl","json","serialization","unity","unity3d","xml"],"created_at":"2024-08-02T05:01:49.370Z","updated_at":"2026-03-16T16:31:29.532Z","avatar_url":"https://github.com/grofit.png","language":"C#","funding_links":[],"categories":["C\\#"],"sub_categories":[],"readme":"# Persistity\n\nA pipeline based data persistence framework (a bit like ETL) for game developers for use with Unity, Monogame or other .net based engines/frameworks.\n\n[![Build Status][build-status-image]][build-status-url]\n[![Nuget Version][nuget-image]][nuget-url]\n[![Join Discord Chat][discord-image]][discord-url]\n\n## What is it?\n\nIts an async pipeline for handling complex extract/transform/load interactions for your models in any format (xml/json/binary out the box).\n\nAt its heart it has:\n\n- `Transformers` (Transforms one statically typed model to another)\n- `Serializers` (Extracts and serializes models)\n- `Processors` (Does stuff on the data before it reaches its destination)\n- `Endpoints` (Sends or Recieves data from an end point)\n\nThese are basically steps in a workflow or pipeline, so you can chain together steps to do stuff like converting your model to json, url encoding it and then sending it to a web service, or converting it to binary and then pumping it into a file.\n\n## Example \n\nHere are a few examples covering incoming and outgoing data pipelines.\n\n### Setting Up Models For persistence\n```csharp\n[Persist]\npublic class SomeClass\n{\n    [PersistData]\n    public float SomeValue { get; set; }\n}\n```\n\nThe `[Persist]` attribute indicates this is a root class for persisting, and `[PersistData]` indicates that the property should be mapped. In nested classes or collections you can omit the `[Persist]` attribute, its just the root type that needs this.\n\n### Outgoing Pipeline\n\nSo for example lets cover a quick use case for saving your game state data to a binary file:\n\n```csharp\n// This would normally be setup once in your app or via DI etc\nvar mappingRegistry = new MappingRegistry(new DefaultTypeMapper());\nvar binarySerializer = new BinarySerializer(mappingRegistry);\nvar fileEndpoint = new FileEndpoint(\"savegame.sav\");\n\n// Create the pipeline which wraps the underlying steps\nvar saveToBinaryFilePipeline = new PipelineBuilder()\n    .StartWithInput()\n    .SerializeWith(binarySerializer)\n    .ThenSendTo(fileEndpoint)\n    .Build();\n\n// Execute the pipeline with your game data\nawait saveToBinaryFilePipeline.Execute(myGameData);\n```\n\nNow lets imagine we decide we wanted to encrypt our game data before we spat it out, we would make changes like so:\n\n\n```csharp\n// Same setup as before but we add an encryption processor\nvar encryptor = new AesEncryptor(\"some-pass-phrase\");\nvar encryptionProcessor = new EncryptDataProcessor(encryptor);\n\n// Same as before but we now add the processor into the mix\nvar saveToBinaryFilePipeline = new PipelineBuilder()\n    .StartWithInput()\n    .SerializeWith(binarySerializer)\n    .ProcessWith(encryptionProcessor)\n    .ThenSendTo(fileEndpoint)\n    .Build();\n\n// Execute the pipeline with your game data\nawait saveToBinaryFilePipeline.Execute(myGameData);\n```\n\nThis then will encrypt your data after it has been transformed and will pass it over to be persisted in a file. You could easily decide to change from using a file to using `PlayerPrefs` by just changing the `FileWriter` to `PlayerPrefWriter` and using that `.SendTo(playerPrefsEndpoint)` which would send your data to player prefs rather than a flat file.\n\n### Incoming Pipeline\n\nSo in the previous example we covered sending data to the file system, but lets imagine we wanted that encrypted saved data back in at some point, we could do:\n\n```csharp\n// This would normally be setup once in your app or via DI etc\nvar mappingRegistry = new MappingRegistry(new DefaultTypeMapper());\nvar binaryDeserializer = new BinaryDeserializer(mappingRegistry);\nvar encryptor = new AesEncryptor(\"some-pass-phrase\");\nvar decryptionProcessor = new DecryptDataProcessor(encryptor);\nvar fileEndpoint = new FileEndpoint(\"savegame.sav\");\n\n// Create the pipeline which wraps the underlying steps\nvar loadBinaryFilePipeline = new PipelineBuilder()\n    .StartFrom(fileEndpoint)\n    .ProcessWith(decryptionProcessor)\n    .DeserializeWith(binaryDeserializer)\n    .Build();\n\n// Execute the pipeline to get your game data\nvar myGameData = await loadBinaryFilePipeline.Execute\u003cMyGameData\u003e();\n```\n\nThis will decrypt and convert your lovely binary data back into a statically typed object for you to do as you wish with.\n\n### Creating Pipelines Without Builder\n\nSo the builder offers a simple and flexible way to create pipelines wherever you want, however in most complex situations you may just want to create your own pipeline implementation and pass that around, which can be done for the above like so:\n\n```csharp\npublic class ABespokePipeline : IPipeline\n{\n    private ISerializer _serializer;\n    private ITransformer _dataTransformer;\n    private ISendDataEndpoint _endpoint;\n\n    public SaveEncryptedBinaryFilePipeline(ISerializer serializer, IToEntityDatabaseDataTransformer dataTransformer, ISendDataEndpoint endpoint)\n    {\n        _serializer = serializer;\n        _dataTransformer = dataTransformer;\n        _endpoint = endpoint;\n    }\n\n    public Task\u003cobject\u003e Execute(object input = null, object state = null)\n    {\n        var transformedData = _dataTransformer.Transform(input);\n        var output = _serializer.Serialize(transformedData, true);\n        return _endpoint.Send(output);\n    }\n}\n```\n\n### Creating Strongly Typed Pipeline via Builder\n\n```csharp\npublic class MyStronglyTypedPipeline : BuiltPipeline\n{\n    protected override IEnumerable\u003cIPipelineStep\u003e BuildSteps()\n    {\n        return new PipelineBuilder()\n            .StartFromInput()                \n            // Build the pipeline\n            .BuildSteps();\n    }\n}\n```\n\n## Using The Framework\n\nAll you need to do is download the stable release and install the unity package.\n\n### Docs\n\nThere are a load of documents covering the various classes and how to use them within the docs folder.\n\n### Advised Setup\n\nIt is HIGHLY recommended you use some sort of DI system to setup your high level pipelines and just inject them in via IoC to your objects. This will make your setup and configuration far more sane.\n\nFor those using unity something like [Zenject](https://github.com/modesttree/Zenject) works wonders here.\n\n### Dependencies\n\n- `LazyData` (There are specific LazyData libs for each serialization format)\n\nHistorically **LazyData** used to be part of this project, but realistically it could be consumed outside of here without any problem, so it made sense to make it into its own library.\n\n## Docs\n\nCheck out the docs directory for docs which go into more detail.\n\n## More Waffle\n\nThe general idea here is that this framework provides a basis for you to add your own transformers, processors and endpoints. So you may be fine with the default transformers and processors out of the box, but you will probably want to add some of your own for your own workflows.\n\nYou can just use pieces of this as well if you want, like if you just want to use the serialization, just take that and use it without the transformers and the other endpoints etc.\n\nIt was originally started as a branch in the [EcsRx](https://github.com/grofit/ecsrx) repository but as it was not tied to that was brought out into its own repo. \n\nThe overall goal is to attempt to build simple building blocks for consumers to create larger workflows of data, such as scenarios like extracting data as JSON and posting to a web service, or extracting data as binary and putting into a flat file.\n\n[build-status-image]: https://ci.appveyor.com/api/projects/status/wuthq2w1oavx24tf/branch/master?svg=true\n[build-status-url]: https://ci.appveyor.com/project/grofit/persistity/branch/master\n[nuget-image]: https://img.shields.io/nuget/v/persistity.svg\n[nuget-url]: https://www.nuget.org/packages/persistity/\n[discord-image]: https://img.shields.io/discord/488609938399297536.svg\n[discord-url]: https://discord.gg/bS2rnGz","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrofit%2Fpersistity","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrofit%2Fpersistity","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrofit%2Fpersistity/lists"}