{"id":22725422,"url":"https://github.com/ovotech/comms-deduplication","last_synced_at":"2025-04-13T20:33:12.162Z","repository":{"id":37894091,"uuid":"161629933","full_name":"ovotech/comms-deduplication","owner":"ovotech","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-11T04:30:33.000Z","size":765,"stargazers_count":3,"open_issues_count":2,"forks_count":3,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-04-11T05:50:33.529Z","etag":null,"topics":["company-kaluza","team-comms"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovotech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-12-13T11:34:38.000Z","updated_at":"2024-04-15T06:09:59.537Z","dependencies_parsed_at":"2023-02-17T11:05:25.980Z","dependency_job_id":"6574d85f-991f-4909-b3f6-f87d87a60e69","html_url":"https://github.com/ovotech/comms-deduplication","commit_stats":null,"previous_names":[],"tags_count":453,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fcomms-deduplication","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fcomms-deduplication/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fcomms-deduplication/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fcomms-deduplication/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovotech","download_url":"https://codeload.github.com/ovotech/comms-deduplication/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229089204,"owners_count":18018390,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["company-kaluza","team-comms"],"created_at":"2024-12-10T16:10:41.727Z","updated_at":"2024-12-10T16:10:42.376Z","avatar_url":"https://github.com/ovotech.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mnemosyne\n\n\u003e Mnemosyne (mnɛːmosýːnɛː) is the Greek god of memory. \"Mnemosyne\" is derived\n\u003e from the same source as the word mnemonic, that being the Greek word mnēmē,\n\u003e which means \"remembrance, memory\"\n\nGiven an effectful operation `F[A]`, this library can be used to wrap the\ncomputation so that it's only executed once[*](#limitations). It achieves this\nby remembering which operations have already been executed and what their result\n`A` was.\n\nIt is based on two main concepts:\n\n- `id`: An identifier of the operation, this can be any value that is both\n  consistent and unique for each operation. _Only one[*](#limitations)_\n  operation with the same `id` will be allowed to run.\n- `contextId`: A context identifier, usually the type of operation being\n  performed. This is used to keep different types of operation separate within a\n  service, so that they can all use the same `id` (eg: they are all processing\n  the same entity)\n\nThe library is able to work across multiple nodes with the same `contextId`. The\npersistence is based on [DynamoDb](https://aws.amazon.com/dynamodb/) and its\nstrong write consistency capability. The same concept can be applied to [Apache\nCassandra](http://cassandra.apache.org/) or any other similar database that\nprovides these two features:\n\n- Strong write consistency\n- Upsert with the previous record values returned\n\n## Versions\n\n| Version | Description              |\n|---------|--------------------------|\n| 0.x     | Old logic                |\n| 1.x     | Old logic, Cats Effect 3 |\n| 2.x     | New logic                |\n | 3.x     | New logic, Cats Effect 3 |\n\n\n## Add the dependency to your project\n\nYou'll need to add our public Maven repository:\n\n```scala\nresolvers += \"Artifactory maven\" at \"https://kaluza.jfrog.io/artifactory/maven\"\n\n```\n\nThen add this snippet to your `build.sbt`\n\n```scala\nlibraryDependencies += \"com.ovoenergy.comms\" %% \"deduplication\" % \"$VERSION\"\n```\n\nAn [example terraform file](example.tf) is provided for provisioning the backing\ndatabase with DynamoDB.\n\n## How to use it\n\nThe main two objects in the library are `Deduplication` and\n`DeduplicationContext`.\n\n`Deduplication` is the entrypoint.\nIt holds the global configuration and a reference to the data storage\nand would usually be instantiated once at the beginning of your program.\n\n`DeduplicationContext` is used to deduplicate a single effectful operation.\nYou can have as many contexts as you like, as long as they all have a different\ncontext ID.\nNew contexts can be instantiated by calling `.context(\u003ccontextId\u003e)` on a\n`Deduplication` instance.\n\nOnce you have an instance of `DeduplicationContext`, you can use\n`context.protect(id: ID, fa: F[A])` to wrap your side effects.\n\n#### DynamoDB backend\n\nIn order to create a new `Deduplication` instance you will need an\nimplementation of `ProcessRepo`, which is the data backend that is responsible\nto store information around which operations have been executed and what their\nresults were.\n\nThe library has a built-in implementation of `ProcessRepo` that uses DynamoDB\nand [Meteor](https://d2a4u.github.io/meteor/) under the hood. You can use it by\ncreating an instance of `Deduplication` with the\n`com.ovoenergy.comms.deduplication.meteor.MeteorDeduplication` factory object.\n\nIf you use the meteor implementation you will need to make sure the meteor\ncodecs for the return type of your operations are in scope when you instantiate\nnew contexts.\n\n#### Example\n\nIn the following example a service is consuming a stream of events and\nperforming 2 effectful operations on top of them.\n\nUsing the library we make sure[*](#limitations) that, if the same event is\nconsumed more than once, the side effects will not be re-executed.\n\n```scala\nimport cats.effect._\nimport com.ovoenergy.comms.deduplication\nimport com.ovoenergy.comms.deduplication.meteor.MeteorDeduplication\nimport com.ovoenergy.comms.deduplication.meteor.codecs._\nimport meteor.CompositeKeysTable\nimport meteor.syntax._\n\n// A stream of events.\n// This could contain duplicates itself or the same event could be present\n// in different streams across service instances\nval events: Stream[IO, MyEvent] = ???\n\n// Effectful operations that need to be executed for each event\ndef sendEmail(evt: MyEvent): IO[String] = ???\ndef storeEmail(evt: MyEvent, sendId: String): IO[Unit] = ???\n\n// Global configuration\nval dedupConf: deduplication.Config = ???\nval dedupTable: CompositeKeysTable[String, String] = ???\n\nval dedupResource = for {\n  client \u003c- meteor.Client.resource[IO]\n  dedup \u003c- MeteorDeduplication.resource[IO, String, String](\n    client,\n    dedupTable,\n    dedupConf\n  )\n} yield dedup\n\ndedupResource.use { deduplication =\u003e\n\n  // Create two contexts to deduplicate the operations separately\n  val sendEmailCtx = deduplication.context[String](\"sendEmail\")\n  val storeEmailCtx = deduplication.context[Unit](\"storeEmail\")\n\n  events\n    .evalMap { evt =\u003e\n      for {\n        // Wrap the operations in a protect call\n        // returns the stored result if sendEmail(evt) was already executed\n        // in a different thread or process\n        sendId \u003c- sendEmailCtx.protect(evt.id, sendEmail(evt))\n        _ \u003c- storeEmailCtx.protect(evt.id, storeEmail(evt, sendId))\n      } yield ()\n    }\n    .compile\n    .drain\n}\n```\n\n## How does it work\n\nThe library is based on the two phase commit strategy. It records when the\nan operation starts being executed within a context and when it's completed.\n It provides a `protect` method that wraps an effectful operation to\n guarantee[*](#limitations) that it will happen only once for each `contextId`.\n\nThe DynamoDb table has this structure:\n\n- `id`: S - The unique identifier of the execution\n- `contextId`: S - The unique identifier of the context\n- `startedAt`: N - The datetime the signal has started to be processed\n- `result`: M - An object containing the result of the execution\n- `expiresOn`: N - The datetime when the process result will expire\n\nEach time a context with a given `contextId` attempts to execute an operation\nidentified by `id`, it updates or writes on the table a record with `id`,\n`contextId`, `startedAt`. If a record with the given `id` and `contextId` is\nalready present, its value is returned to the library, otherwise nothing is\nreturned. After the operation has run successfully, the library marks it as\ncompleted by storing the `result` and the `expiresOn` fields.\n\nThe `expiresOn` allows clean up of old data and the same operation to re-run\nafter some time.\n\nWhen the library attempts to start a process, these scenarios can happen:\n\n1. The signal has never been processed previously (no previous record found)\n2. The signal has timed out processing (`result` is absent and `startedAt +\nprocessingTime` is in the past)\n3. The signal has already been processed previously (`result` is present)\n4. The signal is still being processed (`result` is absent and `startedAt +\nprocessingTime` is in the future)\n\nIn cases (1) and (2) the library allows the signal to be processed. In case (3)\nthe library does not allow the signal to be processed again and returns the\nstored result. In case (4) the library waits for the process to either complete\nor timeout before making any decision.\n\n## How to configure it\n\nA `deduplication.Config` is required in order to create an instance of\n`Deduplication`. The following parameters are available:\n\n- `maxProcessingTime: FiniteDuration`: The time after which a pending operation\n  will be considered stale and a new one will be allowed to take over.\n- `ttl: Option[FiniteDuration]`: The time after which a successful operation\n  will be considered expired and allowed to run again. If `None` the operation\n  will never expire.\n- `pollStrategy: Config.PollStrategy`: The delay strategy to use when polling\n  for the status of a running operation. The `PollStrategy` object provides\n  helper methods for creating one easily.\n\nA `contextId` needs to be assigned to each instance of `DeduplicationContext`.\nIt will uniquely identify the type of operation being run.  If more than one\nservice instance uses the same `contextId` the library will\nensure[*](#limitations) that each operation is only executed once across all of\nthem.\n\n## Limitations\n\nUnsurprisingly, this library doesn't achieve perfect _exactly once_ executions,\nbut it makes a best effort at it.\n\nIts purpose is to __limit__ duplication as much as possible while making sure\nthat all operations are executed _at least once_.\n\nYou should still make sure your system is resilient to duplicates.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovotech%2Fcomms-deduplication","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovotech%2Fcomms-deduplication","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovotech%2Fcomms-deduplication/lists"}