{"id":23149666,"url":"https://github.com/munchy-bytes/schemamapper","last_synced_at":"2025-08-17T19:31:43.782Z","repository":{"id":40882733,"uuid":"173005506","full_name":"munchy-bytes/SchemaMapper","owner":"munchy-bytes","description":"A .NET class library that allows you to import data from different sources into a unified destination","archived":false,"fork":false,"pushed_at":"2023-05-30T19:15:56.000Z","size":6188,"stargazers_count":60,"open_issues_count":3,"forks_count":16,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-08-15T07:23:37.668Z","etag":null,"topics":["csharp","csv","data-import","data-integration","databases","excel","html","json","msaccess","mysql","oracle","powerpoint","schema-mapping","schema-matching","sql-server","sqlce","sqlite","tabular-data","vcard","xml"],"latest_commit_sha":null,"homepage":"https://munchy-bytes.github.io/SchemaMapper/","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/munchy-bytes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-02-27T23:18:53.000Z","updated_at":"2025-03-27T13:38:31.000Z","dependencies_parsed_at":"2023-01-25T06:45:43.770Z","dependency_job_id":"9d691aee-5134-458f-b161-6035a8e285a6","html_url":"https://github.com/munchy-bytes/SchemaMapper","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/munchy-bytes/SchemaMapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/munchy-bytes%2FSchemaMapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/munchy-bytes%2FSchemaMapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/munchy-bytes%2FSchemaMapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/munchy-bytes%2FSchemaMapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/munchy-bytes","download_url":"https://codeload.github.com/munchy-bytes/SchemaMapper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/munchy-bytes%2FSchemaMapper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270898221,"owners_count":24664697,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-17T02:00:09.016Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","csv","data-import","data-integration","databases","excel","html","json","msaccess","mysql","oracle","powerpoint","schema-mapping","schema-matching","sql-server","sqlce","sqlite","tabular-data","vcard","xml"],"created_at":"2024-12-17T18:14:46.256Z","updated_at":"2025-08-17T19:31:42.450Z","avatar_url":"https://github.com/munchy-bytes.png","language":"C#","readme":"# SchemaMapper\n\n\u003cimg src=\"sm_logo.png\" height=\"200pt\" width =\"325pt\"/\u003e\n\nSchemaMapper is a data integration class library that facilitates data import process from external sources having different schema definitions. It replaces creating many integration services packages by writing few lines of codes.\n\nIt imports tabular data from different data sources such as into a destination table with a user defined table schema after mapping columns between source and destination.\n\nSchemaMapper has the ability to read data from:\n\n- Excel worksheets *(.xls, .xlsx)*\n- Flat files *(.csv, .txt)*\n- Access databases *(.mdb, .accdb)*\n- Web pages *(.htm, .html)* \n- JSON files *(.json)*\n- XML files *(.xml)* \n- Powerpoint presentations *(.ppt, .pptx)*\n- Word documents *(.doc, .docx)*\n- Relational databases *(SQL Server, Oracle, MySQL, SQLite, SQL Server compact)*\n\nAnd it can export data into different destination types:\n\n- Flat files *(.csv, .txt)*\n- XML files *(.xml)* \n- Relational databases *(SQL Server, Oracle, MySQL)*\n\nIn addition, it allows users to add new computed and fixed valued columns.\n\n------------------------\n\n## Used technologies\n\nSchemaMapper utilizes from many technologies to read data from different source such as:\n\n- [Microsoft Office Interop libraries to import tables from Word and Powerpoint / Clean Excel files](https://www.microsoft.com/en-us/download/details.aspx?id=3508)\n- [Json.Net library](https://www.newtonsoft.com/json/help/html/Introduction.htm) to import JSON\n- [HtmlAgilityPack](https://html-agility-pack.net/) to import tables from HTML\n- [Microsoft Access database engine](https://www.microsoft.com/en-us/download/details.aspx?id=13255) to import data from Excel worksheets and Access databases. \n- [.NET framework 4.5.2](https://dotnet.microsoft.com/download/dotnet-framework/net452)\n- [MySQL .NET connector](https://dev.mysql.com/downloads/connector/net/8.0.html) to import and export data to MYSQL databases\n- [Oracle Data Provider for .NET](https://www.oracle.com/technetwork/cn/topics/dotnet/index-085163.html) to import and export data to Oracle databases\n- [System.Data.SQLite](https://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) to import data from SQLite databases\n- [SQL Server Compact 4.0 SP1 redistributable](https://www.microsoft.com/en-us/download/details.aspx?id=30709) to import data from SQL Server CE databases\n\n*Note: The needed Assemblies are found in the Assemblies folder but it is recommended to install the assemblies related to the providers that you want to connect to from the official links (mentioned above) since they can requires more related assemblies*\n\n------------------------\n\n## Project details\n\nSchemaMapper is composed of four main namespaces:\n\n- **Converters:**  It reads data from external files into DataSet\n- **DataCleaners:** Cleans files before importing\n- **SchemaMapping:** Changes the imported data structure to a unified schema\n- **Exporters:** export tables to external sources\n-------------------------\n\n## Wiki\n\n- [Import data from multiple files into one SQL table step by step guide](https://github.com/munchy-bytes/SchemaMapper/wiki/Import-data-from-multiple-files-into-one-SQL-table-step-by-step-guide)\n- [Storing SchemaMapper information into XML\n](https://github.com/munchy-bytes/SchemaMapper/wiki/Storing-SchemaMapper-information-into-XML)\n- [Using FileCleaner namespace to ignore reading additional empty cells in Excel](https://github.com/munchy-bytes/SchemaMapper/wiki/Using-FileCleaner-namespace-to-ignore-reading-additional-empty-cells-in-Excel)\n\n-------------------------\n\n## NuGet Package\n\n- [NuGet Gallery | SchemaMapper](https://www.nuget.org/packages/SchemaMapper/)\n\n-------------------------\n\n## External links\n\n- [Adding MySQL, SQLite, and Oracle support to SchemaMapper](https://medium.com/munchy-bytes/schemamapper-1-1-0-release-notes-9b1314825669)\n- [Announcing SchemaMapper a C# data integration class library](https://medium.com/munchy-bytes/announcing-schemamapper-a-c-data-integration-class-library-541dcfad4e2b)\n\n-------------------------\n\n## Credits\n\nCredits are for Munchy Bytes™\n\n-------------------------\n\n## Examples\n\n### (1) Converters:\n\n**Import data from Excel file (first worksheet)**\n\n```cs\nusing (SchemaMapperDLL.Classes.Converters.MsExcelImport smExcel = new SchemaMapperDLL.Classes.Converters.MsExcelImport(@\"U:\\Passwords.xlsx\",\"\",false))\n{\n\n   //Read Excel\n   smExcel.BuildConnectionString();\n   var lst = smExcel.GetSheets();\n   DataTable dt = smExcel.GetTableByName(lst.First(), true, 0);\n   return dt;\n}\n```\n\n**Import data from Excel file using paging**\n\n```cs\nusing (SchemaMapperDLL.Classes.Converters.MsExcelImport smExcel = new SchemaMapperDLL.Classes.Converters.MsExcelImport(@\"U:\\Passwords.xlsx\", \"\", false)){\n\n   //Read Excel with pagging\n   smExcel.BuildConnectionString();\n   var lst = smExcel.GetSheets();\n\n   int result = 1;\n   int PagingStart = 1, PagingInterval = 10;\n\n   while (result != 0){\n\n      DataTable dt = smExcel.GetTableByNamewithPaging(lst.First(), PagingStart, PagingInterval, out result, true, 0);\n\n      PagingStart = PagingStart + PagingInterval;\n\n   }\n\n}\n```\n\n**Import data from flat file (.txt, .csv)**\n\n```cs\nusing (SchemaMapperDLL.Classes.Converters.FlatFileImportTools smFlat = new SchemaMapperDLL.Classes.Converters.FlatFileImportTools(@\"U:\\Passwords.csv\",true,0))\n{\n\n   //Read flat file structure\n   smFlat.BuildDataTableStructure();\n   //Import data from flat file\n   DataTable dt = smFlat.FillDataTable();\n   int Result = dt.Rows.Count;\n\n}\n```\n\n**Import data from word document**\n\n```cs\nusing (SchemaMapperDLL.Classes.Converters.MsWordImportTools smWord = new SchemaMapperDLL.Classes.Converters.MsWordImportTools(@\"U:\\DocumentTable.docx\", true, 0))\n{\n\n   smWord.ImportWordTablesIntoList(\";\");\n   DataSet ds = smWord.ConvertListToTables(\";\");\n\n   int ct = ds.Tables.Count;\n}\n```\n\n### (2) SchemaMapping\n\n**Initiate a SchemaMapper class**\n\nFirst you have to imports `SchemaMapperDLL.Classes.SchemaMapping` namespace.\n\n```cs\nusing SchemaMapperDLL.Classes.SchemaMapping;\n\npublic SchemaMapper InitiateTestSchemaMapper(string schema, string table){\n\n   SchemaMapper smResult = new SchemaMapper();\n\n   smResult.TableName = table;\n   smResult.SchemaName = schema;\n\n   //Add variables\n   smResult.Variables.Add(new Variable(\"@Today\", DateTime.Now.ToString(\"yyyy-MM-dd HH:mm:ss\")));\n\n\n   //Define Columns\n   SchemaMapper_Column smServerCol = new SchemaMapper_Column(\"Server_Name\", SchemaMapper_Column.ColumnDataType.Text);\n   SchemaMapper_Column smUserCol = new SchemaMapper_Column(\"User_Name\", SchemaMapper_Column.ColumnDataType.Text);\n   SchemaMapper_Column smPassCol = new SchemaMapper_Column(\"Password\", SchemaMapper_Column.ColumnDataType.Text);\n\n   //Define a column with Fixed Value\n   SchemaMapper_Column smFixedValueCol = new SchemaMapper_Column(\"AddedDate\", SchemaMapper_Column.ColumnDataType.Text,\"@Today\");\n   \n   //Define a Column with Expression\n   SchemaMapper_Column smExpressionCol = new SchemaMapper_Column(\"UserAndPassword\",SchemaMapper_Column.ColumnDataType.Text,true,\"[User_Name] + '|' + [Password]\");\n\n   //Add columns to SchemaMapper\n   smResult.Columns.Add(smServerCol);\n   smResult.Columns.Add(smUserCol);\n   smResult.Columns.Add(smPassCol);\n   smResult.Columns.Add(smFixedValueCol);\n   smResult.Columns.Add(smExpressionCol);\n\n   //Add all possible input Columns Names for each Column\n   smServerCol.MappedColumns.AddRange(new[] {\"server\",\"server name\",\"servername\",\"Server\",\"Server Name\",\"ServerName\"});\n   smUserCol.MappedColumns.AddRange(new[] { \"UserName\", \"User\", \"login\", \"Login\", \"User name\" });\n   smPassCol.MappedColumns.AddRange(new[] { \"Password\",\"pass\", \"Pass\", \"password\" });\n\n   //Added columns to ignore if found\n   //Sys_SheetName and Sys_ExtraFields is an auto generated column when reading Excel file\n   smResult.IgnoredColumns.AddRange(new[] { \"Column1\", \"Sys_Sheetname\", \"Sys_ExtraFields\", \"Center Name\" });\n\n   //Save Schema Mapper into xml\n   smResult.WriteToXml(Environment.CurrentDirectory + \"\\\\SchemaMapper\\\\1.xml\",true);\n\n   return smResult;\n\n}\n```\n\n**Change DataTable schema and insert into SQL using stored procedure with Table variable parameter**\n\n\n```cs\nDataTable dt = ReadExcel();\n\nusing (SchemaMapper SM = InitiateTestSchemaMapper(\"dbo\",\"PasswordsTable\"))\n{\n\n   bool result  = SM.ChangeTableStructure(ref dt);\n   string con = @\"Data Source=.\\SQLINSTANCE;Initial Catalog=tempdb;integrated security=SSPI;\";\n \n   using (SchemaMapperDLL.Classes.Exporters.SqlServerExport exp = new SchemaMapperDLL.Classes.Exporters.SqlServerExport(con))\n   {\n       exp.CreateDestinationTable(SM);\n       exp.InsertToSQLUsingStoredProcedure(SM, dtExcel);\n   }\n\n}\n\n```\n**Change DataTable schema and insert into SQL using BULK Insert**\n\n```cs\nDataTable dt = ReadExcel();\n\nusing (SchemaMapper SM = new SchemaMapper(Environment.CurrentDirectory + \"\\\\SchemaMapper\\\\1.xml\"))\n{\n\n   bool result  = SM.ChangeTableStructure(ref dt);\n   string con = @\"Data Source=.\\SQLINSTANCE;Initial Catalog=tempdb;integrated security=SSPI;\";\n\n   using (SchemaMapperDLL.Classes.Exporters.SqlServerExport exp = new SchemaMapperDLL.Classes.Exporters.SqlServerExport(con))\n   {\n       exp.CreateDestinationTable(SM );\n       exp.InsertUsingSQLBulk(SM, dtExcel);\n   }\n   \n}\n```\n\n**Read SchemaMapper class from Saved XML**\n\n```cs\nusing (SchemaMapper SM = new SchemaMapper(\"Environment.CurrentDirectory + \"\\\\SchemaMapper\\\\1.xml\")){\n\n   //write your code here\n\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmunchy-bytes%2Fschemamapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmunchy-bytes%2Fschemamapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmunchy-bytes%2Fschemamapper/lists"}