{"id":15419292,"url":"https://github.com/pensivebrian/catalogstore","last_synced_at":"2026-04-19T02:03:19.649Z","repository":{"id":145174232,"uuid":"155261327","full_name":"pensivebrian/catalogstore","owner":"pensivebrian","description":"An experimental .NET Core library to more efficiently reverse engineer a SQL Server database.","archived":false,"fork":false,"pushed_at":"2020-06-18T13:23:50.000Z","size":1947,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-02T05:14:01.016Z","etag":null,"topics":["mssql","sql-server","sqlclient","sqlserver"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pensivebrian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-29T18:29:50.000Z","updated_at":"2024-07-09T18:06:31.000Z","dependencies_parsed_at":"2023-03-23T18:18:31.975Z","dependency_job_id":null,"html_url":"https://github.com/pensivebrian/catalogstore","commit_stats":{"total_commits":14,"total_committers":2,"mean_commits":7.0,"dds":0.0714285714285714,"last_synced_commit":"0a6b1591b5c67309b1884b0d59df40d41a1132bf"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pensivebrian%2Fcatalogstore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pensivebrian%2Fcatalogstore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pensivebrian%2Fcatalogstore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pensivebrian%2Fcatalogstore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pensivebrian","download_url":"https://codeload.github.com/pensivebrian/catalogstore/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245966929,"owners_count":20701759,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mssql","sql-server","sqlclient","sqlserver"],"created_at":"2024-10-01T17:24:39.220Z","updated_at":"2025-10-06T12:33:53.670Z","avatar_url":"https://github.com/pensivebrian.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CatalogStore\nAn experimental .NET Core library to more efficiently reverse engineer a SQL Server database.\n\n## Overview\nAll database tooling needs to reverse engineer a database.  Tooling does this by querying the system catalog about the objects\nin the database (e.g. tables, views, triggers, store procedures, etc).  Here we compare DacFx and a new strategy for reverse \nengineering a database: CatalogStore.\n\n## DacFx Reverse Engineering\nToday, when DacFx reverse engineers a database, it issues a brutal query that punches SQL Server in the face.  The query is \n380K in size, and does all kinds of complicated joins and fancy TSQL manipulation.  See file \n[DacFxReverseEngineeringQuery2016.sql](ReverseEngineeringTableDump/DacFxReverseEngineeringQuery2016.sql).\nAfter viewing that file and changing your pants, you can probably guess that executing this query is very inefficient. Executing \nthis query against an Azure 'Basic SKU' database, pegs the DTU quota, making the database unusable for a few seconds after \nexecution. \n\nNot only are these queries inefficient, they are expensive to maintain.  All of the queries are handcrafted against a SQL \nversion.  When we release a new version of SQL, we cut and paste the query text, along with the SqlClient processing code,\nand then update the copy to handle the new system catalog tables and columns.\n\n## CatalogStore Reverse Engineering\nThis project take a different approach.  SQL Server exposes rich meta data over the system catalog.  We leverage this, and code\ngenerate a library that can execute simple 'select *' statements for each system catalog table.  We take the results, and \nplace them all in a sqlite database.  Then, we code generate an entity framework model over the sqlite system \ncatalog tables.  This give us a nice type-safe query experience over the system catalog, with all processing occurring on \nthe client side.  Because everything is code generated, maintenance should be lower.\n\nThis design can potentially have other benefits\n* Opens the door to implement an efficient global search over all database objects.\n* The sqlite system catalog database can be persisted and cached on disk.\n* Intellisense cache\n\n## Benchmarks\nInitial benchmarks indicate CatalogStore is in-line with DacFx when reverse engineering small databases.  However as object\ncounts grow, CatalogStore scales better than DacFx.\n\n| Engine              | Database                   | Object Count  | Time      |\n| ------------------- | -------------------------- |:-------------:| ---------:|\n| DacFx               | AdventureWorks             | 1603          | 1 sec     |\n| CatalogStore        | AdventureWorks             | 1603          | 1 sec     |\n| DacFx               | WideWorldImporters         | 2618          | 1 sec     |\n| CatalogStore        | WideWorldImporters         | 2618          | 1 sec     |\n| DacFx               | Dynamics                   | 5230          | 6 sec     |\n| CatalogStore        | Dynamics                   | 5230          | 2 sec     |\n| DacFx               | Dynamics x 2               | 7871          | 18 sec    |\n| CatalogStore        | Dynamics x 2               | 7871          | 4 sec     |\n| DacFx               | Dynamics x 5               | 15794         | 1 min     |\n| CatalogStore        | Dynamics x 5               | 15794         | 11 sec    |\n| DacFx               | Dynamics x 10              | 28999         | 3 min     |\n| CatalogStore        | Dynamics x 10              | 28999         | 21 sec    |\n| DacFx               | Dynamics x 20              | 55409         | 14 min    |\n| CatalogStore        | Dynamics x 20              | 55409         | 43 sec    |\n| DacFx               | Azure Basic AdventureWorks | 1601          | 6 sec     |\n| CatalogStore        | Azure Basic AdventureWorks | 1601          | 8 sec     |\n| DacFx               | Azure S3 AdventureWorks    | 1601          | 2 sec     |\n| CatalogStore        | Azure S3 AdventureWorks    | 1601          | 2 sec     |\n| DacFx               | Azure P1 AdventureWorks    | 1603          | 1 sec     |\n| CatalogStore        | Azure P1 AdventureWorks    | 1603          | 1 sec     |\n\n## Projects\n\n#### Microsoft.SqlServer.CatalogStore\n* The code generated CatalogStore\n\n#### CatalogStoreCodeGenerator\n* Code generates the files in the [Microsoft.SqlServer.CatalogStore\\CodeGen](Microsoft.SqlServer.CatalogStore/CodeGen) folder\n* Uses the [Catalog.Whitelist.txt](CatalogStoreCodeGenerator/Catalog.Whitelist.txt) to get all system catalog tables used for reverse engineering.\n* Generates and caches the system catalog list (e.g. [Catalog.2005.txt](https://raw.githubusercontent.com/pensivebrian/catalogstore/master/CatalogStoreCodeGenerator/Catalog.2005.txt)) and catalog xml meta data files (e.g [Catalog.2005.xml](CatalogStoreCodeGenerator/Catalog.2005.xml)) for each SQL Server version.\n\n\n#### Microsoft.SqlServer.CatalogStore.Tests\n* Has tests to compare the execution of the DacFx query with the CatalogStore version.\n\n#### ReverseEngineeringTableDump\n* Uses ScriptDom to parse the DacFx reverse engineering queries for Azure and SQL 2016, and generates the \n[Catalog.Whitelist.txt](CatalogStoreCodeGenerator/Catalog.Whitelist.txt) file, which is input into the code generation process.  This way, both DacFx and CatalogStore query the same tables.\n\n## TODO\n*  Explicit mapping of primary keys\n* Figure out how to handle sql_variants - currently not mapped\n* Handle timespan column types - currently not mapped\n* Introduce loading profiles: lazy, object explorer, reverse engineer\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpensivebrian%2Fcatalogstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpensivebrian%2Fcatalogstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpensivebrian%2Fcatalogstore/lists"}