{"id":25870728,"url":"https://github.com/the-tech-idea/beepdm","last_synced_at":"2025-08-08T12:03:11.217Z","repository":{"id":37865142,"uuid":"353611013","full_name":"The-Tech-Idea/BeepDM","owner":"The-Tech-Idea","description":"A Library for Managing your Connection to Different DataSources . Still in Alpha.please be patient","archived":false,"fork":false,"pushed_at":"2025-07-23T15:38:37.000Z","size":93450,"stargazers_count":1,"open_issues_count":4,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-08-01T00:55:13.789Z","etag":null,"topics":["data-analysis","data-management","data-management-platform","data-science","database","dataset","information"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/The-Tech-Idea.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-04-01T07:19:55.000Z","updated_at":"2025-07-23T15:38:41.000Z","dependencies_parsed_at":"2023-10-21T18:23:40.400Z","dependency_job_id":"ee924434-0207-4b11-9262-e4a6899b5f93","html_url":"https://github.com/The-Tech-Idea/BeepDM","commit_stats":{"total_commits":288,"total_committers":3,"mean_commits":96.0,"dds":"0.23611111111111116","last_synced_commit":"6c416c258dcd4ba1c2c27cde730c76019b44609d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/The-Tech-Idea/BeepDM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-Tech-Idea%2FBeepDM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-Tech-Idea%2FBeepDM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-Tech-Idea%2FBeepDM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-Tech-Idea%2FBeepDM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/The-Tech-Idea","download_url":"https://codeload.github.com/The-Tech-Idea/BeepDM/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-Tech-Idea%2FBeepDM/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269417453,"owners_count":24413379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","data-management","data-management-platform","data-science","database","dataset","information"],"created_at":"2025-03-02T06:31:55.943Z","updated_at":"2025-08-08T12:03:11.152Z","avatar_url":"https://github.com/The-Tech-Idea.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BeepDM: Beep Data Management Engine\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)  \n**Current Status: Alpha** - Actively developed, expect bugs, contributions welcome!\n\nBeepDM is a modular, extensible data management engine designed to streamline connecting, managing, and synchronizing data across diverse sources. It provides a robust framework for developers, supporting databases, files, and in-memory stores with programmatic control over connections, data movement, and entity management.\n\n## Key Features\n- **Modular Architecture**: Flexible components for data sources, ETL, workflows, and add-ins.\n- **Wide Data Source Support**: Connect to databases (e.g., SQLite, SQL Server), files (e.g., CSV, XLS), APIs, and in-memory stores via `IDataConnection` and `IRDBSource`.\n- **Data Synchronization**: `DataSyncManager` for real-time or scheduled sync with metrics and logging.\n- **Entity Management**: `UnitofWork\u003cT\u003e` for CRUD operations, change tracking, and transactional commits.\n- **ETL \u0026 Import**: `DataImportManager` for transforming and importing data with batch processing.\n- **Dependency Injection**: Supports **Microsoft.Extensions.DependencyInjection** and **Autofac**.\n- **Configuration Management**: Centralized settings via `IConfigEditor`.\n- **Extensibility**: Add custom functionality with `IDM_Addin` and extend connections/data types.\n\n## Core Components (Main Interfaces)\nThese are the primary interfaces driving BeepDM’s functionality:\n1. **`IDMEEditor`**: The mother class, orchestrating all components below. Acts as the central hub for data management operations.\n2. **`IConfigEditor` (`ConfigEditor`)**: Manages framework configurations (e.g., `DataDriversClasses`, `DataTypesMap`, `QueryList`, `DataConnections`), persisting them to JSON files.\n3. **`IDataSource`**: Defines the contract for all data source implementations (e.g., SQLite, XLS), providing methods like `GetEntity`, `CreateEntityAs`, and `UpdateEntities`.\n4. **`IETL`**: Handles Extract, Transform, and Load operations for data integration.\n5. **`IDataTypesHelper`**: Manages data type mappings and configurations, supporting `IDataSource` type translation.\n6. **`IUtil`**: Provides common utility functions used across the engine.\n7. **`IAssemblyHandler`**: Loads assemblies and extracts implementations (e.g., `IDataSource`, drivers, `IDM_Addin`, extensions).\n8. **`IErrorsInfo`**: Handles error reporting and management.\n9. **`IDMLogger`**: Manages logging across the framework.\n10. **`IJsonLoader`**: Handles loading and saving JSON configuration files.\n11. **`IClassCreator`**: Generates classes/types for data source entities.\n12. **`IWorkFlowEditor`**: Manages data workflows.\n13. **`IWorkFlowStepEditor`**: Manages individual steps/stages within workflows.\n14. **`IRuleParser`**: Parses data rules used in workflows.\n15. **`IRulesEditor`**: Manages data rules configuration.\n\n## Directory Structure\nEvery BeepDM project follows this directory structure:\n1. **Addin**: Stores DLLs implementing the `IDM_Addin` interface (e.g., user controls, forms, classes).\n2. **AI**: Stores AI scripts (for future use).\n3. **Config**: Contains configuration files:\n   - `QueryList.json`: Defines query types for retrieving metadata from data sources.\n   - `ConnectionConfig.json`: Defines drivers, data source classes, and metadata (e.g., icons).\n   - `DataTypeMapping.json`: Maps data types between data sources.\n   - `DataConnections.json`: Stores data source connection details.\n4. **ConnectionDrivers**: Holds data source driver DLLs (e.g., Oracle, SQLite, SQL Server).\n5. **DataFiles**: Primary storage for project data files.\n6. **DataViews**: Stores JSON files for federated views of data source entities.\n7. **Entities**: Temporary storage for data source entity descriptions.\n8. **GFX**: Stores graphics and icons used by the application.\n9. **LoadingExtensions**: Contains classes implementing `ILoaderExtention` to dynamically load additional functionality.\n10. **Mapping**: Stores mapping definitions between data sources.\n11. **OtherDLL**: Holds miscellaneous DLLs required by the application.\n12. **ProjectClasses**: Primary folder for loading custom implementations (e.g., `IDataSource`, add-ins).\n13. **ProjectData**: Stores project-specific files.\n14. **Scripts**: Stores scripts and logs.\n15. **WorkFlow**: Stores workflow definitions.\n\n## Getting Started\nBeepDM is in alpha and offers programmatic control over data operations. Below are examples using **Autofac** to demonstrate core functionality with `IDataSource`-based data sources.\n\n### Prerequisites\n- .NET Framework or .NET Core (specific version TBD).\n- NuGet packages: `Autofac`.\n- Database drivers (e.g., SQLite) or file access for your data sources.\n\n### Installation\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/The-Tech-Idea/BeepDM.git\n   ```\n2. Open the solution in Visual Studio.\n3. Restore NuGet packages.\n4. Build the project.\n\n### Initialization\nAfter bootstrapping with `IBeepService`, perform these steps to populate `ConfigEditor` with defaults, enabling `IDataSource` operations:\n\n#### 1. Add Connection Configurations\nPopulates `ConfigEditor.DataDriversClasses` with default drivers.\n```csharp\nusing TheTechIdea.Beep.Container;\nusing TheTechIdea.Beep.Helpers;\n\nbeepService.AddAllConnectionConfigurations();\n```\n\n#### 2. Add Data Type Mappings\nPopulates `ConfigEditor.DataTypesMap` with default type mappings.\n```csharp\nusing TheTechIdea.Beep.Container;\nusing TheTechIdea.Beep.Helpers;\n\nbeepService.AddAllDataSourceMappings();\n```\n\n#### 3. Add Query Configurations\nPopulates `ConfigEditor.QueryList` with default SQL queries for RDBMS.\n```csharp\nusing TheTechIdea.Beep.Container;\n\nbeepService.AddAllDataSourceQueryConfigurations();\n```\n\n#### Example Initialization\n```csharp\nusing Autofac;\nusing TheTechIdea.Beep.Container;\nusing TheTechIdea.Beep.Container.Services;\n\nstatic void Main()\n{\n    var builder = new ContainerBuilder();\n    BeepServicesRegisterAutFac.RegisterServices(builder);\n    var container = builder.Build();\n    BeepServicesRegisterAutFac.ConfigureServices(container);\n    var beepService = BeepServicesRegisterAutFac.beepService;\n\n    // Initialize for IDataSource support\n    beepService.AddAllConnectionConfigurations();\n    beepService.AddAllDataSourceMappings();\n    beepService.AddAllDataSourceQueryConfigurations();\n}\n```\n\n### Requirements\nEnsure data source configurations are registered in `ConfigEditor`. The initialization steps add defaults for common `IDataSource` implementations (e.g., SQLite, XLS). Custom data sources require additional setup (see \"Extending BeepDM\").\n\n### Basic Usage\n#### Bootstrapping with Autofac (Mother Class: `DMEEditor`)\n`IDMEEditor` must be initialized to use BeepDM. Here’s an implementation using Autofac:\n```csharp\nusing Autofac;\nusing TheTechIdea.Beep;\nusing TheTechIdea.Beep.ConfigUtil;\nusing TheTechIdea.Beep.Container.Services;\nusing TheTechIdea.Logger;\nusing TheTechIdea.Util;\n\nstatic void Main()\n{\n    var builder = new ContainerBuilder();\n    builder.RegisterType\u003cDMEEditor\u003e().As\u003cIDMEEditor\u003e().SingleInstance();\n    builder.RegisterType\u003cConfigEditor\u003e().As\u003cIConfigEditor\u003e().SingleInstance();\n    builder.RegisterType\u003cDMLogger\u003e().As\u003cIDMLogger\u003e().SingleInstance();\n    builder.RegisterType\u003cUtil\u003e().As\u003cIUtil\u003e().SingleInstance();\n    builder.RegisterType\u003cErrorsInfo\u003e().As\u003cIErrorsInfo\u003e().SingleInstance();\n    builder.RegisterType\u003cJsonLoader\u003e().As\u003cIJsonLoader\u003e().SingleInstance();\n    builder.RegisterType\u003cAssemblyHandler\u003e().As\u003cIAssemblyHandler\u003e().SingleInstance();\n\n    var container = builder.Build();\n    BeepServicesRegisterAutFac.ConfigureServices(container);\n    var beepService = BeepServicesRegisterAutFac.beepService;\n\n    // Initialize framework\n    beepService.AddAllConnectionConfigurations();\n    beepService.AddAllDataSourceMappings();\n    beepService.AddAllDataSourceQueryConfigurations();\n}\n```\n- All features are pluggable; replace implementations (e.g., `DMLogger` with `YourLogger`) as needed.\n\n#### Connecting to SQLite\n```csharp\nusing TheTechIdea.Beep.ConfigUtil;\nusing TheTechIdea.Beep.DataBase;\n\nvar config = beepService.DMEEditor.ConfigEditor.DataDriversClasses\n    .FirstOrDefault(p =\u003e p.DatasourceType == DataSourceType.SqlLite);\nif (config == null)\n    throw new Exception(\"SQLite config not found in ConfigEditor.DataDriversClasses.\");\n\nvar connProps = new ConnectionProperties\n{\n    ConnectionString = \"Data Source=./Beep/dbfiles/northwind.db\",\n    ConnectionName = \"northwind.db\",\n    DriverName = config.PackageName,\n    DriverVersion = config.version,\n    DatabaseType = DataSourceType.SqlLite,\n    Category = DatasourceCategory.RDBMS\n};\n\nbeepService.DMEEditor.ConfigEditor.AddDataConnection(connProps);\nvar sqliteDB = (SQLiteDataSource)beepService.DMEEditor.GetDataSource(\"northwind.db\");\nsqliteDB.Openconnection();\n\nif (sqliteDB.ConnectionStatus == ConnectionState.Open)\n    Console.WriteLine(\"SQLite connection opened successfully\");\n```\n\n## Extending BeepDM\n### Creating a New Data Source\n1. **Implement `IDataSource`**:\n   ```csharp\n   using System;\n   using System.Collections.Generic;\n   using System.Data;\n   using TheTechIdea.Beep;\n   using TheTechIdea.Beep.DataBase;\n\n   [AddinAttribute(Category = DatasourceCategory.CLOUD, DatasourceType = DataSourceType.WebService)]\n   public class AzureCosmosDataSource : IDataSource\n   {\n       public string GuidID { get; set; } = Guid.NewGuid().ToString();\n       public event EventHandler\u003cPassedArgs\u003e PassEvent;\n       public DataSourceType DatasourceType { get; set; } = DataSourceType.WebService;\n       public DatasourceCategory Category { get; set; } = DatasourceCategory.CLOUD;\n       public IDataConnection Dataconnection { get; set; }\n       public string DatasourceName { get; set; }\n       public IErrorsInfo ErrorObject { get; set; }\n       public string Id { get; set; }\n       public IDMLogger Logger { get; set; }\n       public List\u003cstring\u003e EntitiesNames { get; set; }\n       public List\u003cEntityStructure\u003e Entities { get; set; } = new List\u003cEntityStructure\u003e();\n       public IDMEEditor DMEEditor { get; set; }\n       public ConnectionState ConnectionStatus { get; set; }\n       public string ColumnDelimiter { get; set; } = \"''\";\n       public string ParameterDelimiter { get; set; } = \":\";\n\n       public AzureCosmosDataSource(string name, IDMEEditor editor)\n       {\n           DatasourceName = name;\n           DMEEditor = editor;\n       }\n\n       public ConnectionState Openconnection() { /* Implement */ return ConnectionState.Open; }\n       public ConnectionState Closeconnection() { /* Implement */ return ConnectionState.Closed; }\n       public bool CheckEntityExist(string EntityName) { /* Implement */ return false; }\n       public bool CreateEntityAs(EntityStructure entity) { /* Implement */ return false; }\n       public object GetEntity(string EntityName, List\u003cAppFilter\u003e filter) { /* Implement */ return null; }\n       public IErrorsInfo UpdateEntities(string EntityName, object UploadData, IProgress\u003cPassedArgs\u003e progress) { /* Implement */ return null; }\n       // Implement other IDataSource methods...\n\n       public void Dispose() { /* Implement cleanup */ }\n   }\n   ```\n\n2. **Add to `ConfigEditor`**:\n   - Place driver DLLs (if needed) in `ConnectionDrivers`.\n   - Update `ConnectionConfig.json` or use the Beep Enterprize Winform app:\n     ```csharp\n     var driver = new ConnectionDriversConfig\n     {\n         GuidID = \"azure-cosmos-guid\",\n         PackageName = \"AzureCosmos\",\n         DriverClass = \"AzureCosmos\",\n         version = \"1.0.0\",\n         DbConnectionType = \"AzureCosmosConnection\",\n         ConnectionString = \"AccountEndpoint={Host};AccountKey={Password};Database={Database};\",\n         classHandler = \"AzureCosmosDataSource\",\n         DatasourceCategory = DatasourceCategory.CLOUD,\n         DatasourceType = DataSourceType.WebService,\n         ADOType = false\n     };\n     beepService.DMEEditor.ConfigEditor.DataDriversClasses.Add(driver);\n     ```\n   - Place the DLL in `ProjectClasses`.\n\n### Creating a New Add-in\n1. **Implement `IDM_Addin`**:\n   ```csharp\n   using TheTechIdea.Beep;\n   using TheTechIdea.Beep.Addin;\n\n   [AddinAttribute(Caption = \"Copy Entity Manager\", Name = \"CopyEntityManager\", misc = \"ImportDataManager\", addinType = AddinType.Class)]\n   public class CopyEntityManager : IDM_Addin\n   {\n       public string AddinName =\u003e \"CopyEntityManager\";\n       public IDMEEditor DMEEditor { get; set; }\n       public IPassedArgs Passedarg { get; set; }\n       public IDMLogger Logger { get; set; }\n       public IErrorsInfo ErrorObject { get; set; }\n\n       public void Run(IPassedArgs pPassedarg)\n       {\n           var ds = DMEEditor.GetDataSource(Passedarg.DatasourceName);\n           if (ds != null) ds.Openconnection();\n           // Implement logic\n       }\n\n       public void SetConfig(IDMEEditor pbl, IDMLogger plogger, IUtil putil, string[] args, IPassedArgs e, IErrorsInfo per)\n       {\n           DMEEditor = pbl;\n           Passedarg = e;\n           Logger = plogger;\n           ErrorObject = per;\n       }\n   }\n   ```\n\n2. **Deploy**: Place the DLL in `Addin` or `ProjectClasses`. It will appear in the add-in tree.\n\n### Creating an Extension\n1. **Implement `ILoaderExtention`**:\n   ```csharp\n   using TheTechIdea.Beep;\n   using TheTechIdea.Util;\n\n   public class CustomExtension : ILoaderExtention\n   {\n       public IAssemblyHandler Loader { get; set; }\n\n       public CustomExtension(IAssemblyHandler ploader) { Loader = ploader; }\n\n       public IErrorsInfo LoadAllAssembly()\n       {\n           var er = new ErrorsInfo();\n           // Custom loading logic\n           return er;\n       }\n\n       public IErrorsInfo Scan()\n       {\n           var er = new ErrorsInfo();\n           LoadAllAssembly();\n           er.Flag = Errors.Ok;\n           return er;\n       }\n   }\n   ```\n\n2. **Deploy**: Place the DLL in `LoadingExtensions`.\n\n## Project Status\n- **Alpha Phase**: Core features functional, APIs may evolve.\n- **Contributions**: Welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) (TBD).\n\n## License\nBeepDM is licensed under the [MIT License](LICENSE).\n\n## Learn More\n- [Wiki](https://github.com/The-Tech-Idea/BeepDM/wiki/Beep-Data-Management-Engine-(BeepDM))\n- [Issues](https://github.com/The-Tech-Idea/BeepDM/issues)\n- [Beep Data Sources](https://github.com/The-Tech-Idea/BeepDataSources)\n- [Beep Enterprize Winform](https://github.com/The-Tech-Idea/BeepEnterprize.winform)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthe-tech-idea%2Fbeepdm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthe-tech-idea%2Fbeepdm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthe-tech-idea%2Fbeepdm/lists"}