{"id":24309400,"url":"https://github.com/mgroves/sqlservertocouchbase","last_synced_at":"2025-09-26T14:31:32.008Z","repository":{"id":46221065,"uuid":"322403562","full_name":"mgroves/SqlServerToCouchbase","owner":"mgroves","description":"Library to automatically best effort move and remodel data from relational databases (like SQL Server) to Couchbase","archived":false,"fork":false,"pushed_at":"2021-11-11T21:03:58.000Z","size":590,"stargazers_count":11,"open_issues_count":3,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2023-03-11T10:56:49.995Z","etag":null,"topics":["couchbase","data","json","migration","sql","sql-server","tables"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mgroves.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-17T20:18:45.000Z","updated_at":"2022-10-25T21:03:08.000Z","dependencies_parsed_at":"2022-09-25T05:22:05.900Z","dependency_job_id":null,"html_url":"https://github.com/mgroves/SqlServerToCouchbase","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgroves%2FSqlServerToCouchbase","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgroves%2FSqlServerToCouchbase/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgroves%2FSqlServerToCouchbase/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgroves%2FSqlServerToCouchbase/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mgroves","download_url":"https://codeload.github.com/mgroves/SqlServerToCouchbase/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234314426,"owners_count":18812697,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["couchbase","data","json","migration","sql","sql-server","tables"],"created_at":"2025-01-17T05:15:11.916Z","updated_at":"2025-09-26T14:31:31.448Z","avatar_url":"https://github.com/mgroves.png","language":"C#","readme":"# SQL Server to Couchbase migration library\n\nA library to help guide your SQL Server to Couchbase migration/sync efforts. This library represents a best effort at mapping SQL Server concepts to Couchbase Server concepts *automatically*. It may or may not fulfill every one of your requirements. In the worst case, it can at least be an educational tool. Think of it like Google Translate.\n\n## Use\n\nCreate a `SqlToCb` object with config (and logger of your choice). Execute the `Migrate` method with various flags to limit/expand what you want to try migrating.\n\n`SqlToCbConfig` contains SQL Server connection string, Couchbase connection string/credentials, and other settings.\n\n## Migrations\n\n_Catalog to Bucket_ - For the given *database* (aka \"catalog\") in SQL Server, create a *bucket* in Couchbase Server (named according to `SqlToCbConfig::TargetBucket`).\n\n_Tables to Collections_ - For each *table* in the given SQL Server database catalog, create a corresponding *collection* in a Couchbase Server bucket. This will also involve the *schema* which you may choose to correspond to a *scope* in Couchbase Server.\n\nExamples:\n\n`Config::UseSchemaForScope = true`\n\nSchema | Table | \u003c-\u003e | Scope | Collection\n--- | --- | --- | --- | ---\nPerson | Address | | Person | Address\nSales | Contact | | Sales | Contact\ndbo | Invoice | | _default** | Invoice\n\n`Config::UseSchemaForScope = false`\n\nSchema | Table | \u003c-\u003e | Scope | Collection\n--- | --- | --- | --- | ---\nPerson | Address | | _default | Person_Address\nSales | Contact | | _default | Sales_Contact\ndbo | Invoice | | _default** | Invoice\n\n** The rough equivalent of `dbo`  in Couchbase Server is _default.\n\n_Why would I ever want `UserSchemaForScope = false`?_ Scope can be used in Couchbase Server for multi-tenancy. In which case, you'd likely want to reserve scopes to correspond to tenants.\n\n_Indexes to Indexes_ - Indexes designed for relational access may not always be optimal for Couchbase access. However, Indexes will be created that are roughly equivalent.\n\nExample:\n\nSQL Server: `CREATE INDEX [AK_Person_rowguid] ON [Person].[Person] ([rowguid])`\n\nbecomes\n\nCouchbase: `CREATE INDEX sql_AK_Person_rowguid ON Person_Person(rowguid)`\n\nBefore using these indexes blindly, I strongly recommend trying your query in Couchbase Server Query Workbench to see if there's a better way to write it and/or to see if Couchbase advises creating a different index.\n\n_Rows to documents_ - As with indexes, the translation here is going to be literal. But to truly maximize Couchbase data, you may want to (eventually) consolidate the \"translated\" documents. (e.g. consolidate Invoice row with PK 123 and corresponding InvoiceItems rows into a single Invoice document with key 123).\n\nExamples:\n\nStarting with a `dbo.Person` Schema/Table in SQL Server:\nId | Name | ShoeSize\n--- | --- | ---\n55 | Matt D | 14\n77 | Emma R | 6\n\nThose rows of data will be created in the `_default.Person` Scope/Collection in Couchbase:\n```\nkey: 55\n{ Id: 55, Name: 'Matt D', ShoeSize: 14}\n\nkey: 77\n{ Id: 77, Name: 'Emma R', ShoeSize: 6}\n```\n\nNotice that Id remains intact in the document body. This is to ease the transition, but in the long run it may not be a good idea to keep this \"duplicated\" data that's already in the document key.\n\nIf the primary key in the SQL Server table is a compound key, the values will be combined into a single document key in Couchbase, separated by \"::\".\n\nExample:\n\nKey1Id | Key2Id | Name\n--- | --- | ---\n55 | 14 | Matt D\n77 | 6 | Emma R\n\nThe documentss for the equivalent data in Couchbase will be:\n```\nkey: 55::14\n{ Key1Id: 55, Key2Id: 14, Name: 'Matt D' }\n\nkey: 77::6\n{ Key1Id: 77, Key2Id: 6, Name: 'Emma R' }\n```\n\nAlso note that *most* data in SQL Server translates to JSON in a relatively straightforward way. You may want to pay special attention to dates and SQL Server specific types (like geometry). Again, a best effort translation is made. Improvements or suggestions are welcome :)\n\n_Users to users_\n\nSQL Server has multiple kinds of users. This program will look at all users in sysusers that have DB access and create a user in Couchbase by the same name.\n\nIt will also give this user *roughly* the same permissions:\n\nSQL Server Permission | Couchbase Permission(s)\n--- | ---\nSELECT | Query Select, Data Reader\nINSERT | Query Insert, Data Writer\nUPDATE | Query Update, Data Writer\nDELETE | Query Delete, Data Writer\n\nIf the user doesn't have any specific permissions, it will give that user Bucket Admin access (which gives the user unlimited access to all features of the bucket).\n\nFor complex auth scenarios, users are an area that require a manual audit; user migration is definitely intended as a learning tool only.\n\n## Filters/Transform Pipelines\n\nBy default, _all_ data is copied over _as is_ from SQL Server to Couchbase.\n\nYou can add \"pipelines\" to your migration if you want to filter and/or transform data that's being copied over.\n\n_Filter_: Logic to decide whether or not to include a given piece of data. E.g. \"only copy a row of data if 'createDate' is later than '2020-08-18'\".\n\n_Transform_: How to change data as its being copied. E.g. \"when copying a row of data, change the state value from the 2-letter abbreviation 'OH' to the full state name 'Ohio'\"\n\nTo create a filter and/or transform, create a class that inherits from SqlPipelineBase. You can override one or more of the methods to add filtering/transforming behavior:\n\n* **Query**: The query that will pull the data from SQL Server. By default, this is `SELECT * FROM [SchemaName].[TableName]`, but you can override it. By using a query with more specificity than `SELECT *`, you can transform the data. By using a query with `WHERE`, you can filter the data.\n\n* **IsIncluded**: When overriding this method for filtering, you can supply logic that returns true to include the data, or false to exclude the data.\n\n* **Transform**: When overriding this method for transforming, you can supply logic that transforms the given piece of data and return whatever you'd like.\n\nOnce you've created your class, instatiate it and associate it with a schema+table. See **SamplePipelines.cs** for some examples of pipeline classes. Create a `SqlPipelines` object and add your filters to it. Then give that pipelines object to the `Migrate` method.\n\n```\nvar pipelines = new SqlPipelines();\npipelines.Add(new ModifiedDateSqlFilter(new DateTime(2014, 05, 27), \"Person\", \"Address\"));\n\n// ... snip ...\n\nawait convert.MigrateAsync(copyData: true, pipelines: pipelines);\n```\n\nYou may only add one filter/transform object per SQL Server table.\n\n_Why override `IsIncluded` or `Transform` instead of overriding `Query`? Isn't changing the query going to be more efficient?_ Yes, giving a custom SQL query is going to be the most efficient way to filter/transform. However, if you have complex logic better expressed in C# and/or have need to call out to services that SQL Server doesn't have access to, you can use these functions instead.\n\n## Denormalization\n\nAfter the data is copied over, you can run one or more denormalizations. The `SqlToCbConfig` object you use has a `DenormalizeMaps` property. Denormalization definitions are applied **in order** to the documents. So if you want to denormalize at multiple levels, make sure that the farthest branches of denormalization run first. The documents will be then cascaded up to the root.\n\nTwo denormalization objects that are built-in are `ManyToOneDenormalizer` and `OneToOneDenormalizer`. (You can define your own normalizer by implementing `IDenormalizer`).\n\n### ManyToOneDenormalizer\n\nA many-to-one denormalizer means that you will be embedding documents in one collection into their corresponding \"parent\" documents, using their foreign key values. (Couchbase does not support foreign key constraints, but all the values needed are present in the converted documents).\n\nFor example, if you have a ShoppingCart collection and a ShoppingCartItems collection, you can use ManyToOneDenormalizer to embed all the ShoppingCartItems documents into their \"parent\" ShoppingCart documents.\n\nThis object contains two properties: `From` and `To`.\n\n`From` defines the \"child\" data: SchemaName, TableName, and ForeignKeyNames.\n\n`To` defined the \"parent\" data: SchemaName and TableName.\n\nAfter embeddeding, the embedded values will live in an array with a pluralized version of the table name.\n\nFor instance, a shopping cart example:\n\n```\nFrom.SchemaName = \"Shopping\"\nFrom.TableName = \"ShoppingCartItems\"\nFrom.PrimaryKeyNames = [ \"ShoppingCartId\" ]\n\nTo.SchemaName = \"Shopping\"\nTo.TableName = \"ShoppingCart\"\n```\n\nThe end result will be documents in the ShoppingCart collection like:\n\n```\n{\n    \"ShoppingCartId\" : 123,\n    \"CartCreatedDt\" : \"2021-07-15\",\n    \"ShoppingCartItems\" : [\n        { \"ShoppingCartId\" : 123, \"Desc\" : \"Widget\", \"Qty\" : 1, \"Price\" : 9.99 },\n        { \"ShoppingCartId\" : 123, \"Desc\" : \"Doohickey\", \"Qty\" : 3, \"Price\" : 0.99 },\n    ]\n}\n```\n\nNotice the vestigial `ShoppingCartId` field in the array of nested objects.\n\nThe documents in the ShoppingCartItems collection ALSO remain as vestigial data, though it can be easily removed with the Couchbase SDK directly if you want.\n\n### OneToOneDenormalizer\n\nA one-to-one denormalizer means you will be embedding a document from one collection into a document in another collection by using the foreign key value from one document. (Couchbase does not support foreign key constraints, but all the values needed are present in the converted documents).\n\nFor example, if you have a document in a Person collection with a `PhoneNumberId` field which corresponds to a document in the PhoneNumber document, the PhoneNumber document will be embedded into the \"parent\" document.\n\nThis object contains two properties: `From` and `To`.\n\n`From` defines the \"child\" data: SchemaName and TableName.\n\n`To` defined the \"parent\" data, and some optional transforms: SchemaName, TableName, RemoveForeignKey, Unnest, UnnestSeparator, ForeignKeyNames\n\nFor instance, the phone number example:\n\n```\nFrom.SchemaName = \"Person\"\nFrom.TableName = \"PhoneNumber\"\n\nTo.SchemaName = \"Person\"\nTo.TableName = \"Person\"\nTo.ForeignKeyNames = [ \"HomePhoneNumberId\" ]\nTo.RemoveForeignKey = false\nTo.Unnest = false\nTo.UnnestSeparator = \"_\"\n```\n\nThe end result will be documents in the Person collection like:\n\n```\n{\n    \"PersonId\" : 123,\n    \"Name\" : \"Matt\",\n    \"PhoneNumberId\" : 456,\n    \"PhoneNumber\" : {\n        \"PhoneNumberId\" : 456,\n        \"Number\" : \"111-222-3333\"\n    }\n}\n```\n\nNotice the vestigial PhoneNumberId field: if you set RemoveForeignKey to true, this will be automatically removed.\n\nNotice that PhoneNumber is a nested object: if you set Unnest to true, all the fields will be nested up one level. The field names will be prepended with the old table name and whatever unnest seperator you define.\n\n## Known limitations:\n\n* SQL Server views are not translated at all. [Couchbase M/R Views](https://docs.couchbase.com/server/current/learn/views/views-intro.html) are *roughly* equivalent.\n\n* SQL Server stored procedures (sprocs) and UDFs are not translated at all. [Couchbase UDFs](https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/userfun.html) are *roughly* equivalent.\n\n* SQL Server triggers are not translated at all. [Couchbase Eventing](https://docs.couchbase.com/server/current/eventing/eventing-overview.html) is the closest equivalent.\n\n* Collection names in Couchbase are limited to 30 characters. So if a table name (or schema+table name depending on your config) is longer than 30 characters, you'll need to provider a mapping in `Config::TableNameToCollectionMapping`\n\n* HierarchyId data type - this SQL Server specific data type can *possibly* be translated to JSON, but for now it doesn't translate to anything meaningful\n\n* Temporal tables - Currently the \"Archive\" tables are not being copied over, only the snapshots. Temporal data capabilities are not built into Couchbase, they would need to be added with a combination of client code and/or Couchbase Eventing.\n\n* Denormalization: if you run denormalization on the data multiple times, it will likely result in duplicates. You should rerun the data copy BEFORE denormalization every time.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgroves%2Fsqlservertocouchbase","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmgroves%2Fsqlservertocouchbase","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgroves%2Fsqlservertocouchbase/lists"}