{"id":21655905,"url":"https://github.com/koltyakov/spsync","last_synced_at":"2025-04-11T21:30:55.013Z","repository":{"id":73405077,"uuid":"489668673","full_name":"koltyakov/spsync","owner":"koltyakov","description":"🪢 Go library for robust cloud native SharePoint Lists synchronization or backup","archived":false,"fork":false,"pushed_at":"2024-03-24T20:08:49.000Z","size":50,"stargazers_count":9,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-25T17:22:57.926Z","etag":null,"topics":["go","golang","library","sharepoint","sharepoint-online","synchronization"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koltyakov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-05-07T12:34:11.000Z","updated_at":"2024-12-30T22:26:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"cf23453a-5ee5-4d33-8db6-47041f8fd736","html_url":"https://github.com/koltyakov/spsync","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fspsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fspsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fspsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fspsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koltyakov","download_url":"https://codeload.github.com/koltyakov/spsync/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248482859,"owners_count":21111393,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","library","sharepoint","sharepoint-online","synchronization"],"created_at":"2024-11-25T08:37:44.391Z","updated_at":"2025-04-11T21:30:54.977Z","avatar_url":"https://github.com/koltyakov.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spsync\n\n\u003e Go library for robust cloud native SharePoint Lists synchronization or backup\n\n\u003c!--suppress HtmlDeprecatedAttribute --\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003cimg alt=\"Gosip\" src=\"https://raw.githubusercontent.com/koltyakov/gosip-docs/master/.gitbook/assets/gosip.png\" /\u003e\n\u003c/div\u003e\n\nThe library implements optimal synchronization of SharePoint Lists taking care of pagination, large lists thresholds, incremental sync via changes API, deletion and items restore tracking, and more.\n\nIt is designed to run as a scheduled job within serverless scenarios. The incremental mode could be adapted for Webhooks signals.\n\nIn a scheduler mode, let's assume the Azure Functions timer job, after single or multiple executions (which could be terminated with a timeout) full sync is completed, then incremental mode is effectively catching up with only changed items in seconds.\n\nHere are the design and core principles of how it works.\n\n## Entity sync metadata persistent state\n\n| Entity     | Change Token  | Page Token           | Mode | Stage  | Sync Date | Failed Sessions |\n| ---------- | ------------- | -------------------- | ---- | ------ | --------- | --------------- |\n| List/ListA | 1;3;b9...6db9 | Paged=TRUE\u0026p_ID=6387 | Full | Upsert |           | 0               |\n\n### Synchronization Modes\n\n- `Full`\n- `Incr`\n\n### Synchronization Stages\n\n- Upsert\n- Delete\n\n## Full synchronization mode\n\n### Full synchronization mode condition(s)\n\n`Mode` == `Full` or a `Change token` is not set.\n\n### Full synchronization mode logic\n\n- On a blank session:\n  - save current change token and timestamp (sync start not completion)\n  - change mode to `Full`\n- On a continue session:\n  - start from a previous page skip token (if any)\n- Default sort (by ID ascending)\n- Paginate by N items\n- Save a previous page skip token (on page successfully processed)\n- Bulk processing upserts (in a custom hook handler)\n- Control time (for serverless scenarios)\n- Fail aware and continue logic (continue from page token)\n- On completion:\n\n  - update sync date after a successful session\n  - clear session page skip token\n  - change mode to `Incr`\n\n- Tracking deletions\n  - Paginate through all items with only ID selected\n  - Using max page size of 5000 items\n  - Detect gaps in ID sequence\n  - Bulk process deletions (in a custom hook handler)\n\n## Incremental synchronization mode\n\n### Incremental synchronization mode condition(s)\n\n`Mode` == `Incr` and a `Change token` is not empty.\n\n### Incremental synchronization mode logic\n\n- Use Change Token and change API\n- Track: Adds, Updates, Deletes, Restores\n- Paginate through by N items\n- Save a previous page skip token (on page successfully processed)\n- Process changes list\n  - For added/updated/restored items: request by IDs, and process (in a custom hook handler)\n  - Bulk process deletions (in a custom hook handler)\n- Save the latest change token from a previously processed page\n- Control time (for serverless scenarios)\n- By design, a failure is amended on a next incremental run\n  - however, failed sessions should be tracked\n  - with too many fails - create an incident\n- On completion:\n  - update sync date after a successful session\n  - reset failed sessions counter\n\n## Usage sample\n\n```golang\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"os\"\n\n\t\"github.com/joho/godotenv\"\n\t\"github.com/koltyakov/gosip\"\n\t\"github.com/koltyakov/gosip/api\"\n\tstrategy \"github.com/koltyakov/gosip/auth/saml\"\n\t\"github.com/koltyakov/gosip/cpass\"\n\t\"github.com/koltyakov/spsync\"\n)\n\nfunc main() {\n\t// Load environment variables needed for sync processing\n\t_ = godotenv.Load()\n\n\tctx := context.Background()\n\n\t// Setup sync configuration per a specific entity.\n\t// Usually you need to sync multiple Lists so most of a logic from this sample\n\t// iterates for synced entities.\n\t// In a serverless scenarios, sync session timeout is controlled and terminated\n\t// by the caller after timeout. It's OK, the resumed session is designed\n\t// to continue since a previous successful incremental state.\n\n\topts := \u0026spsync.Options{\n\t\t// Create and bind SharePoint API client, see more https://go.spflow.com\n\t\t// For multiple entities sync initiate the client externally\n\t\tSP: NewSP(),\n\t\t// Pass current state from a persistent storage (e.g. database or SharePoint list)\n\t\tState: \u0026spsync.State{\n\t\t\tEntID:    \"Lists/MyList\",\n\t\t\tSyncMode: spsync.Full,\n\t\t},\n\t\t// Pass configuration for a specific entity, usually stored in config file\n\t\tEnt: \u0026spsync.Ent{\n\t\t\tSelect: []string{\"Id\", \"Title\"},\n\t\t},\n\t\t// Provide a handler to deal with created and updated items\n\t\tUpsert: func(ctx context.Context, items []spsync.Item) error {\n\t\t\t// Implement your logic here for your target system:\n\t\t\t// Bulk create or update if a target system supports batch processing\n\t\t\t// alternatively, check if an item's Id exists and update otherwise create\n\t\t\tfmt.Printf(\"Upsert %d items\\n\", len(items))\n\t\t\treturn nil\n\t\t},\n\t\t// Provide a handler to deal with deleted items\n\t\tDelete: func(ctx context.Context, ids []int) error {\n\t\t\t// Implement your logic here for your target system:\n\t\t\t// Bulk delete if a target system supports batch processing\n\t\t\t// alternatively, delete item by Id and ignore NotFound error types\n\t\t\tfmt.Printf(\"Delete %d items\\n\", len(ids))\n\t\t\treturn nil\n\t\t},\n\t\t// Save the updated entity sync state to a persistent storage\n\t\tPersist: func(s *spsync.State) {\n\t\t\t// The state is used to resume sync session from a previous state.\n\t\t\t// Even when a sync session ended up with errors we need to save it.\n\t\t\t// Some scenarious, e.g. sync in serverless jobs might be designed\n\t\t\t// to terminate in 10-15 minutes and resume on a next run.\n\t\t},\n\t\t// Hook to sync events and use logging middleware of your choice\n\t\tEvents: \u0026spsync.Events{},\n\t}\n\n\t// Run sync session\n\tstate, err := spsync.Run(ctx, opts)\n\tif err != nil {\n\t\tstate.Fails += 1\n\t\t// Persist state here as well\n\t\tfmt.Println(err)\n\t}\n}\n\n// NewSP constructs SharePoint authenticated API client instance\nfunc NewSP() *api.SP {\n\t// Simplest strategy for testing\n\t// for prod the AzureAD auth is recommended:\n\t// https://go.spflow.com/auth/strategies/azure-certificate-auth\n\n\t// Create .env with the following variables: SP_SITE_URL, SP_USERNAME, SP_PASSWORD\n\t// or export the variables in your shell\n\n\tc := cpass.Cpass(\"\")\n\tpassword, _ := c.Decode(os.Getenv(\"SP_PASSWORD\"))\n\n\tauth := \u0026strategy.AuthCnfg{\n\t\tSiteURL:  os.Getenv(\"SP_SITE_URL\"),\n\t\tUsername: os.Getenv(\"SP_USERNAME\"),\n\t\tPassword: password,\n\t}\n\n\tclient := \u0026gosip.SPClient{AuthCnfg: auth}\n\treturn api.NewSP(client)\n}\n```\n\nPowered by [gosip](https://github.com/koltyakov/gosip).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoltyakov%2Fspsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoltyakov%2Fspsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoltyakov%2Fspsync/lists"}