{"id":19522191,"url":"https://github.com/captaincodeman/datastore-locker","last_synced_at":"2025-02-26T00:47:02.966Z","repository":{"id":57506398,"uuid":"63973730","full_name":"CaptainCodeman/datastore-locker","owner":"CaptainCodeman","description":"A lock / lease mechanism to prevent duplicate execution of AppEngine tasks","archived":false,"fork":false,"pushed_at":"2017-03-08T20:33:37.000Z","size":21,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-08T14:15:27.353Z","etag":null,"topics":["appengine","appengine-tasks","datastore-locker","duplicate-tasks","lease","lock"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CaptainCodeman.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-07-22T18:09:28.000Z","updated_at":"2018-09-11T07:53:07.000Z","dependencies_parsed_at":"2022-08-29T20:31:54.897Z","dependency_job_id":null,"html_url":"https://github.com/CaptainCodeman/datastore-locker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fdatastore-locker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fdatastore-locker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fdatastore-locker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fdatastore-locker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CaptainCodeman","download_url":"https://codeload.github.com/CaptainCodeman/datastore-locker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240771906,"owners_count":19854982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appengine","appengine-tasks","datastore-locker","duplicate-tasks","lease","lock"],"created_at":"2024-11-11T00:37:37.544Z","updated_at":"2025-02-26T00:47:02.783Z","avatar_url":"https://github.com/CaptainCodeman.png","language":"Go","readme":"# Datastore Locker\nProvides a lock / lease mechanism to prevent duplicate execution of\nappengine tasks and an easy way to continue long-running processes. Used\nby [datastore-mapper](https://github.com/CaptainCodeman/datastore-mapper).\n\nSometimes it's extremely difficult if not impossible to make tasks truly\nidempotent. e.g. if your task sends an email or charges a credit card\nthen executing it twice could be 'a bad thing'. Actually preventing a task\nfrom running more than once can be challenging though ...\n\nAlthough named tasks can be used to prevent duplicate tasks being *enqueued*\nthey cannot be used together with datastore transactions which leaves you\nwith the possibility that either the task scheduling or the datastore write\ncould fail.\n\nAn unnamed task *can* be enqueued within a transaction to ensure that both\nhappen or neither happen but that then leaves the possibility of *that*\noperation being repeated (if running in it's own task) which could cause\nduplicate tasks.\n\nWhichever approach is used, the taskqueue only promises *at-least-once* \ndelivery so there is also the chance that appengine will execute a task more\nthan once.\n\nThe solution is to coordinate execution using information in the task with\ninformation in a datastore entity. This package aims to make it easy to\nrestrict task execution by using a lease / lock mechanism.\n\nBy obtaining a lock on an entity within a datastore transaction we ensure\nthat only a single instance of any task will be executed at once and, once\nprocessed, that duplicate execution will be prevented.\n\nIt can be used with single tasks or to chain a series of tasks in sequence\nwith the sequence number used to prevent any old tasks being re-executed.\n\nAn exceptional situation can occur if a failure happens during processing\nof a task and the result cannot be communicated back to appengine (this is\na platform issue). In this case the lock / lease is already held but the\nsystem cannot determine if the task completed or maybe it just failed to \nclear the lock. The locker will allow a timeout before querying the appengine\nlogs to determine the task status. In the case of a complete failure with\nno log information, a timeout will prevent deadlock by overwriting the\nexpired lock / lease.\n\nBoth overwritten locks and permanently failing tasks (past a configurable\nnumber of retries) can be alerted by email as needing further investigation.\n\n## Usage\nSee the example project for a simple demonstration of locker being used.\n\nEmbed the `locker.Lock` field within the struct you want lock on.\n\n    Foo struct {\n        locker.Struct\n        Value string `datastore:\"value\"`  \n    }\n\nCreate an instance of the locker and configure as required:\n\n    l := locker.NewLocker()\n\n    l := locker.NewLocker(\n      locker.LeaseDuration(time.Duration(5)*time.Minute),\n      locker.LeaseTimeout(time.Duration(15)*time.Minute),\n      locker.AlertOnOverwrite,\n    )\n\n    l := locker.NewLocker(locker.LogVerbose)\n\nSchedule a task to be executed once:\n\n    key := datastore.NewKey(c, \"foo\", \"\", 1, nil)\n    entity := \u0026Foo{Value:\"bar\"}\n    err := l.Schedule(c, key, entity, \"/task/handler/url\", nil)\n    if err != nil {\n      // operation failed (entity not saved and task not enqueued)\n    }\n\nHandle the task execution:\n\n    func init() {\n      http.Handle(\"/task/handler/url\", locker.Handle(fooHandler, fooFactory)\n    }\n\n    // the task handler needs a factory to construct an instance of our entity\n    func fooFactory() interface{} {\n      return new(Foo)\n    }\n\n    // the handler for the task will be passed the appengine context, request, datastore key and entity\n    func foohandler(c context.Context, r *http.Request, key *datastore.Key, entity locker.Lockable) error {\n      foo := entity.(*Foo)\n\n      switch foo.Sequence {\n        case 1:\n          // step 1 processing, e.g. charge credit card\n          // schedule another task to follow this one:\n          return l.Schedule(c, key, entity, \"/task/handler/url\", nil)\n        case 2:\n          // step 2 processing, e.g. send confirmation email\n          // mark the task as completed (to prevent the last task re-executing)\n          return l.Complete(c, key, entity)\n      }\n\n      // returning an error would cause the task to be failed and retried (normal task semantics)\n      // a configurable number of retries can be set to prevent endless attempts from happening\n      return nil\n    }\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaptaincodeman%2Fdatastore-locker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaptaincodeman%2Fdatastore-locker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaptaincodeman%2Fdatastore-locker/lists"}