{"id":19522224,"url":"https://github.com/captaincodeman/go-appengine-mapper","last_synced_at":"2025-02-26T00:47:02.183Z","repository":{"id":66357883,"uuid":"51462881","full_name":"CaptainCodeman/go-appengine-mapper","owner":"CaptainCodeman","description":"Example of using go to iterate and process datastore entities","archived":false,"fork":false,"pushed_at":"2016-07-28T13:12:28.000Z","size":13,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-08T14:15:28.632Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CaptainCodeman.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-10T18:42:09.000Z","updated_at":"2016-02-10T19:10:35.000Z","dependencies_parsed_at":"2023-02-21T18:15:24.084Z","dependency_job_id":null,"html_url":"https://github.com/CaptainCodeman/go-appengine-mapper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fgo-appengine-mapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fgo-appengine-mapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fgo-appengine-mapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainCodeman%2Fgo-appengine-mapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CaptainCodeman","download_url":"https://codeload.github.com/CaptainCodeman/go-appengine-mapper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240771906,"owners_count":19854982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T00:37:49.412Z","updated_at":"2025-02-26T00:47:02.158Z","avatar_url":"https://github.com/CaptainCodeman.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go AppEngine Mapper\n\nNOTE: Superseded by the more feature complete [datastore-mapper](https://github.com/CaptainCodeman/datastore-mapper).\n\nExample of using Go (Golang) to iterate over entities to process or aggregate them.\n\nIteration handling is reused so new processor just involves implementing a struct to match the interface.\nSee aggregate and log for examples.\n\nCould do with some tidfying up around the range / parameter settings and error handling but it works.\n\n## Strategies\n\nThe basic idea is that the a query is defined for the collection and slices as required (in this case from / to date range)\n\nThe processor defines the batch size to use and the timeout. The timeout should allow for the last batch to be handled.\n\nEntities are processed in batches using a keys only query for efficiency. Only one entity is ever loaded at a time. This\ncould be speeded up by having a pipeline using goroutines and channels.\n\nAlso for performance and atomicity, it could use named tasks to process set batch sizes and schedule a continuation before\nprocessing the entities in a batch. See talks by Brett Slatkin for details of doing that.\n\n## Running\n\nInstall go app dependencies:\n\n    go get ./...\n\nRun with dev server:\n\n    goapp serve\n\n### Example requests\n\nLog all entities from 2015 on ...\n\n    http://localhost:8080/_ah/cron/process/logPhotos?from=2015-01-01\n\nLog all entities for Jan 2015 only ...\n\n    http://localhost:8080/_ah/cron/process/logPhotos?from=2015-01-01\u0026to=2015-02-01\n\nAggregate all entities from 2015 on ...\n\n    http://localhost:8080/_ah/cron/process/aggregatePhotos?from=2015-01-01\n\n## Notes for demo\n\nDefault cron task without params is designed to process previous days entries only\n\nRemember to change the app id in app.yaml if deploying\n\nCron tasks can be restructed for production\n\nWarmup task creates dummy data for demo\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaptaincodeman%2Fgo-appengine-mapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaptaincodeman%2Fgo-appengine-mapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaptaincodeman%2Fgo-appengine-mapper/lists"}