{"id":13515094,"url":"https://github.com/mrpaulandrew/procfwk","last_synced_at":"2025-04-08T03:17:52.370Z","repository":{"id":37712359,"uuid":"243524080","full_name":"mrpaulandrew/procfwk","owner":"mrpaulandrew","description":"A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.","archived":false,"fork":false,"pushed_at":"2024-02-13T14:25:19.000Z","size":42886,"stargazers_count":185,"open_issues_count":38,"forks_count":115,"subscribers_count":24,"default_branch":"master","last_synced_at":"2025-03-27T15:31:34.004Z","etag":null,"topics":["adf","adfprocfwk","azure","azure-functions","azure-sql-database","data-engineering","data-factory","framework","metadata","pipelines","processing","procfwk"],"latest_commit_sha":null,"homepage":"https://mrpaulandrew.com/category/azure/data-factory/adf-procfwk/","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrpaulandrew.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["mrpaulandrew"]}},"created_at":"2020-02-27T13:19:18.000Z","updated_at":"2024-12-30T22:24:51.000Z","dependencies_parsed_at":"2022-09-08T18:01:30.651Z","dependency_job_id":"54b14f3c-f9c0-4154-9311-3e12b9d8061d","html_url":"https://github.com/mrpaulandrew/procfwk","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrpaulandrew%2Fprocfwk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrpaulandrew%2Fprocfwk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrpaulandrew%2Fprocfwk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrpaulandrew%2Fprocfwk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrpaulandrew","download_url":"https://codeload.github.com/mrpaulandrew/procfwk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247767237,"owners_count":20992548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adf","adfprocfwk","azure","azure-functions","azure-sql-database","data-engineering","data-factory","framework","metadata","pipelines","processing","procfwk"],"created_at":"2024-08-01T05:01:06.230Z","updated_at":"2025-04-08T03:17:52.348Z","avatar_url":"https://github.com/mrpaulandrew.png","language":"C#","funding_links":["https://github.com/sponsors/mrpaulandrew"],"categories":["Repositories"],"sub_categories":["Tools \u0026 Samples"],"readme":"# Read Me - Orchestrate.[procfwk](http://procfwk.com/)\n\nFor complete documentation on this solution see [procfwk.com](http://procfwk.com/).\n\n## ProcFwk Has Become CF.Cumulus.Control\n\nSee blog: [mrpaulandrew.com](https://mrpaulandrew.com/2024/01/07/procfwk-is-getting-an-upgrade-to-cf-cumulus/)\n\nSee new product page: [cloudformations.org/cumulus](https://www.cloudformations.org/cumulus?utm_source=pa\u0026utm_medium=github\u0026utm_campaign=cumulus\u0026utm_content=l2)\n\n[ ![](https://mrpaulandrew.github.io/procfwk/procfwk-to-cumulus.png) ](https://mrpaulandrew.github.io/procfwk/procfwk-to-cumulus.png)\n\nProcFwk will receive no further development beyond December 2023.\n\n## Framework Capabilities\n\n * Granular metadata control.\n * Metadata integrity checking.\n * Global properties.\n * Complete pipeline dependency chains.\n * Concurrent batch executions (hourly/daily/monthly).\n * Execution restart-ability.\n * Parallel pipeline execution.\n * Full execution and error logs.\n * Operational dashboards.\n * Low cost orchestration.\n * Disconnection between framework and Worker pipelines.\n * Cross Tenant/Subscription/Data Factory control flows.\n * Pipeline parameter support.\n * Simple troubleshooting.\n * Easy deployment.\n * Email alerting.\n * Automated testing.\n * Azure Key Vault integration.\n * Is pipeline already running checks.\n\n## Complete Data Factory Activity Chain\n\n[ ![](https://mrpaulandrew.github.io/procfwk/activitychain-full.png) ](https://mrpaulandrew.github.io/procfwk/activitychain-full.png)\n\n## Issues\n\nIf you've found a bug or have a new feature request please log the details using the repository issues.\n\nGo to... [Issues](https://github.com/mrpaulandrew/procfwk/issues)\n\n## Projects\nGo to... [External Requests](https://github.com/mrpaulandrew/procfwk/projects/2)\n\nGo to... [Internal Backlog](https://github.com/mrpaulandrew/procfwk/projects/1)\n\n## Release Details\n\n| Version | Overview | Version Details \u0026 Release Notes |\n|:----:|--------------|--------|\n| 2.0 |Azure Synapse Analytics fully supported as an interchangeable orchestrator of pipelines within the procfwk.|GitHub Pages: \u003c/br\u003e[Orchestrators](https://mrpaulandrew.github.io/procfwk/orchestrators)\u003c/br\u003e[Orchestrator Types](https://mrpaulandrew.github.io/procfwk/orchestratortypes) \u003c/br\u003e\u003c/br\u003eRelease Summary Video: \u003c/br\u003e[YouTube - procfwk Playlist](https://www.youtube.com/c/mrpaulandrew)\u003cbr/\u003e\u003cbr/\u003eGitHub Issues:\u003c/br\u003e[procfwk #95](https://github.com/mrpaulandrew/procfwk/issues/95) |\n| 2.0-beta |\u003cu\u003eAzure Synapse Analytics **Beta** support\u003c/u\u003e added.\u003cbr/\u003e\u003cbr/\u003eDevelopment of Azure Functions App completed using the Synapse namespace: _Azure.Analytics.Synapse.Artifacts_ with version **1.0.0-beta.1** of the NuGet package.|GitHub Issues:\u003c/br\u003e[procfwk #21](https://github.com/mrpaulandrew/procfwk/issues/21) |\n| 1.9.2 |\u003cu\u003eBatch Executions\u003c/u\u003e added, plus:\u003cul\u003e\u003cli\u003eException Pipeline\u003c/li\u003e\u003cli\u003eRunning Pipeline Check\u003c/li\u003e\u003cli\u003ePipeline Parameter Last Values\u003c/li\u003e\u003cli\u003eWorker Pipeline Validation\u003c/li\u003e\u003c/ul\u003e|GitHub Pages: [Batch Executions](https://mrpaulandrew.github.io/procfwk/executionbatches) \u003c/br\u003e\u003c/br\u003eRelease Demo Summary Video: [YouTube - procfwk Playlist](https://www.youtube.com/c/mrpaulandrew)\u003cbr/\u003e\u003cbr/\u003eGitHub Issues:\u003c/br\u003e[procfwk #78](https://github.com/mrpaulandrew/procfwk/issues/78)\u003cbr/\u003e[procfwk #77](https://github.com/mrpaulandrew/procfwk/issues/77)\u003cbr/\u003e[procfwk #71](https://github.com/mrpaulandrew/procfwk/issues/71)\u003cbr/\u003e[procfwk #73](https://github.com/mrpaulandrew/procfwk/issues/73)\u003cbr/\u003e[procfwk #80](https://github.com/mrpaulandrew/procfwk/issues/80)\u003cbr/\u003e[procfwk #72](https://github.com/mrpaulandrew/procfwk/issues/72) |\n| 1.9.1 |\u003cu\u003eActivity Policy Update\u003c/u\u003e, plus:\u003cul\u003e\u003cli\u003eSecure Activity Inputs/Outputs.\u003c/li\u003e\u003cli\u003eExecution Wrapper Hardening.\u003c/li\u003e\u003cli\u003eNew Activity Icons and Framework Factory Cosmetics.\u003c/li\u003e\u003c/ul\u003e |GitHub Issues:\u003c/br\u003e[procfwk #65](https://github.com/mrpaulandrew/procfwk/issues/65)\u003cbr/\u003e[procfwk #66](https://github.com/mrpaulandrew/procfwk/issues/66)\u003cbr/\u003e[procfwk #67](https://github.com/mrpaulandrew/procfwk/issues/67)\u003cbr/\u003e[procfwk #69](https://github.com/mrpaulandrew/procfwk/issues/69) |\n| 1.9.0 |\u003cu\u003eCross Tenant \u0026 Subscription Support\u003c/u\u003e added, plus:\u003cul\u003e\u003cli\u003eNew integration tests created.\u003c/li\u003e\u003cli\u003eInfant pipeline refactoring.\u003c/li\u003e\u003cli\u003etSQLt project added.\u003c/li\u003e\u003c/ul\u003e |GitHub Issues:\u003c/br\u003e[procfwk #34](https://github.com/mrpaulandrew/procfwk/issues/34)\u003cbr/\u003e[procfwk #35](https://github.com/mrpaulandrew/procfwk/issues/35)\u003cbr/\u003e[procfwk #46](https://github.com/mrpaulandrew/procfwk/issues/46)\u003cbr/\u003e[procfwk #55](https://github.com/mrpaulandrew/procfwk/issues/55)\u003cbr/\u003e[procfwk #56](https://github.com/mrpaulandrew/procfwk/issues/56)\u003cbr/\u003e[procfwk #59](https://github.com/mrpaulandrew/procfwk/issues/59) |\n| 1.8.6 |\u003cu\u003ePipeline Expressions Refactored to Use Variables\u003c/u\u003e added, plus:\u003cul\u003e\u003cli\u003eNew integration tests created.\u003c/li\u003e\u003cli\u003eComplete activity chain redrawn in Visio.\u003c/li\u003e\u003c/ul\u003e |GitHub Issues:\u003c/br\u003e[procfwk #51](https://github.com/mrpaulandrew/procfwk/issues/51)\u003cbr/\u003e[procfwk #52](https://github.com/mrpaulandrew/procfwk/issues/52) |\n| 1.8.5 |\u003cu\u003eExecution Precursor\u003c/u\u003e added, plus:\u003cul\u003e\u003cli\u003ePowerShell helper to add initial Worker metadata.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.8.5 - Execution Precursor](https://mrpaulandrew.com/2020/08/17/adf-procfwk-v1-8-5-execution-precursor/) |\n| 1.8.4 |\u003cu\u003eDatabase Schema Reorganise and Restructuring\u003c/u\u003e |[procfwk v1.8.4 - Database Schema Reorganise and Restructuring](https://mrpaulandrew.com/2020/07/23/adf-procfwk-v1-8-4-database-schema-reorganise-and-restructuring/) |\n| 1.8.3 |\u003cu\u003eBug Fixes from the Community\u003c/u\u003e, including:\u003cul\u003e\u003cli\u003eEmail alerts sent to blank email addresses due to wrong flow in Child pipeline.\u003c/li\u003e\u003cli\u003eWorker pipelines cancelled during an execution fail when the framework is restarted due to missing Parent pipeline clean up condition.\u003c/li\u003e\u003c/ul\u003e |GitHub Issues:\u003c/br\u003e[procfwk #38](https://github.com/mrpaulandrew/procfwk/issues/38)\u003cbr/\u003e[procfwk #37](https://github.com/mrpaulandrew/procfwk/issues/37) |\n| 1.8.2 |\u003cu\u003eOptionally Store SPN Details in Azure Key Vault\u003c/u\u003e |[procfwk v1.8.2 - Optionally Store SPN Details in Azure Key Vault](https://mrpaulandrew.com/2020/07/22/adf-procfwk-v1-8-2-optionally-store-spn-details-in-azure-key-vault/) |\n| 1.8.1 |\u003cu\u003eAutomated Framework Pipeline Testing\u003c/u\u003e added, including tests for:\u003cul\u003e\u003cli\u003eA simple grandparent run.\u003c/li\u003e\u003cli\u003eAll types of failure dependency handling.\u003c/li\u003e\u003cli\u003eMetadata checks when pipelines and staged are disabled.\u003c/li\u003e\u003cli\u003eNo pipeline parameters provided.\u003c/li\u003e\u003c/ul\u003e |Blog Series: \u003cbr/\u003e\u003col\u003e\u003cli\u003e[Set up automated testing for Azure Data Factory](https://richardswinbank.net/adf/set_up_automated_testing_for_azure_data_factory)\u003c/li\u003e\u003cli\u003e[Automate integration tests in Azure Data Factory](https://richardswinbank.net/adf/automate_integration_tests_in_azure_data_factory)\u003c/li\u003e\u003cli\u003e[Isolated functional tests for Azure Data Factory](https://richardswinbank.net/adf/isolated_functional_tests_for_azure_data_factory)\u003c/li\u003e\u003cli\u003e[Testing Azure Data Factory in your CI/CD pipeline](https://richardswinbank.net/adf/testing_azure_data_factory_in_your_cicd_pipeline)\u003c/li\u003e\u003cli\u003e[Unit testing Azure Data Factory pipelines](https://richardswinbank.net/adf/unit_testing_azure_data_factory_pipelines)\u003c/li\u003e\u003cli\u003e[Calculating Azure Data Factory test coverage](https://richardswinbank.net/adf/calculating_azure_data_factory_test_coverage)\u003c/li\u003e\u003c/ol\u003e |\n| 1.8.0 |\u003cu\u003eComplete Pipeline Dependency Chains For Failure Handling\u003c/u\u003e added, plus:\u003cul\u003e\u003cli\u003eClean up of a previous execution run if Workers appear as running.\u003c/li\u003e\u003cli\u003eNew metadata integrity checks.\u003c/li\u003e\u003cli\u003eInternal get property value function added.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.8 - Complete Pipeline Dependency Chains For Failure Handling](https://mrpaulandrew.com/2020/07/01/adf-procfwk-v1-8-complete-pipeline-dependency-chains-for-failure-handling/) |\n| 1.7.3 |Data Factory Deployment Updated To Use azure.datafactory.tools  PowerShell Module |[SQLPlayer/azure.datafactory.tools](https://github.com/SQLPlayer/azure.datafactory.tools) |\n| 1.7.2 |\u003cu\u003ePipeline Parameter NULL Handling\u003c/u\u003e added, plus:\u003cul\u003e\u003cli\u003eWorker pipelines with a status of 'Running' protected from a new execution start/restart.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.7.2 - NULL Pipeline Parameters Handled](https://mrpaulandrew.com/2020/06/22/adf-procfwk-v1-7-2-null-pipeline-parameters-handled/) |\n| 1.7.1 |\u003cu\u003eAlerting Check Bug Fix\u003c/u\u003e added, plus:\u003cul\u003e\u003cli\u003ePipeline parameter value size limit removed.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.7.1 - Alerting Bug Fix And Pipeline Parameter Size Limit Removed](https://mrpaulandrew.com/2020/06/12/adf-procfwk-v1-7-1-alerting-bug-fix-and-pipeline-parameter-size-limit-removed/) |\n| 1.7.0 |\u003cu\u003ePipleline EMail Alerting\u003c/u\u003e added, plus:\u003cul\u003e\u003cli\u003eSend email Function implemented and hardened.\u003c/li\u003e\u003cli\u003eHandy Notebook updates.\u003c/li\u003e\u003cli\u003eActivity failure paths improved.\u003c/li\u003e\u003cli\u003eMIT license and code of conduct added.\u003c/li\u003e\u003cli\u003eError table bug fix. Error code attribute; INT to VARCHAR\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.7 - Pipeline Email Alerting](https://mrpaulandrew.com/2020/06/08/adf-procfwk-v1-7-pipeline-email-alerting/) |\n| 1.6.0 |\u003cu\u003eError Details for Failed Activities Captured\u003c/u\u003e, plus:\u003cul\u003e\u003cli\u003ePipeline parameters used at runtime captured in execution logs.\u003c/li\u003e\u003cli\u003eEmailing Function added, not yet implemented.\u003c/li\u003e\u003cli\u003eUnknown Worker outcomes optionally blocks downstream stages.\u003c/li\u003e\u003cli\u003eSolution housekeeping.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.6 - Error Details for Failed Activities Captured](https://mrpaulandrew.com/2020/05/19/adf-procfwk-v1-6-error-details-for-failed-activities-captured/) |\n| 1.5.0 |\u003cu\u003ePower BI Dashboard for Framework Executions\u003c/u\u003e, plus:\u003cul\u003e\u003cli\u003eWorker Parallelism View.\u003c/li\u003e\u003cli\u003ePipeline Run ID now logged.\u003c/li\u003e\u003cli\u003eLogging Attributes Bug Fix.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.5 - Power BI Dashboard for Framework Executions](https://mrpaulandrew.com/2020/05/01/adf-procfwk-v1-5-power-bi-dashboard-for-framework-executions/) |\n| 1.4.0 |\u003cu\u003eEnhancements for Long Running Pipelines\u003c/u\u003e, plus:\u003cul\u003e\u003cli\u003ePipeline check status function added.\u003c/li\u003e\u003cli\u003eFunction Data Factory client moved to internal class.\u003c/li\u003e\u003cli\u003eSQL GETDATE() changed to GETUTCDATE().\u003c/li\u003e\u003cli\u003eGlossary created, [here](https://github.com/mrpaulandrew/procfwk/blob/master/Glossary.md).\u003c/li\u003e\u003cli\u003eUpdated database views.\u003c/li\u003e\u003c/ul\u003e  |[procfwk v1.4 - Enhancements for Long Running Pipelines](https://mrpaulandrew.com/2020/04/20/adf-procfwk-v1-4-enhancements-for-long-running-pipelines/) |\n| 1.3.0 |\u003cu\u003eMetadata Integrity Checks\u003c/u\u003e, plus: \u003cul\u003e\u003cli\u003eLogical pipeline predecessors.\u003c/li\u003e\u003cli\u003eData Factory Powershell deployment script.\u003c/li\u003e\u003cli\u003eHelper Notebook.\u003c/li\u003e\u003cli\u003eDatabase objects renames and solution tidy up.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.3 - Metadata Integrity Checks](https://mrpaulandrew.com/2020/04/07/adf-procfwk-v1-3-metadata-integrity-checks/)  |\n| 1.2.0 |\u003cu\u003eExecution Restartability\u003c/u\u003e, plus: \u003cul\u003e\u003cli\u003eData Factory annotations and descriptions.\u003c/li\u003e\u003cli\u003eDatabase covering indexes.\u003c/li\u003e\u003cli\u003ePipeline log status changed from 'Started' to 'Preparing'.\u003c/li\u003e\u003cli\u003ePipeline log start date/time now set in child pipeline.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.2 - Execution Restartability](https://mrpaulandrew.com/2020/03/24/adf-procfwk-v1-2-execution-restartability/)  |\n| 1.1.0 |\u003cu\u003eService Principal Handling\u003c/u\u003e via Metadata, plus: \u003cul\u003e\u003cli\u003eData Factory table.\u003c/li\u003e\u003cli\u003eProperties table and view.\u003c/li\u003e\u003cli\u003eFunction body bug fix.\u003c/li\u003e\u003cli\u003eNew sample data.\u003c/li\u003e\u003c/ul\u003e |[procfwk v1.1 - Service Principal Handling via Metadata](https://mrpaulandrew.com/2020/03/17/adf-procfwk-v1-1-service-principal-handling-via-metadata/) |\n| 1.0.0 |Simple \u003cu\u003eframework designed and base compontents built\u003c/u\u003e.\u003cbr/\u003e\u003cul\u003e\u003cli\u003ePart 1 - Design, concepts, service coupling, caveats, problems.\u003c/li\u003e\u003cli\u003ePart 2 - Database build and metadata.\u003c/li\u003e\u003cli\u003ePart 3 - Data Factory build.\u003c/li\u003e\u003cli\u003ePart 4 - Execution, conclusions, enhancements.\u003c/li\u003e\u003c/ul\u003e|Blog Series: \u003cbr/\u003e[Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines](https://mrpaulandrew.com/2020/02/25/creating-a-simple-staged-metadata-driven-processing-framework-for-azure-data-factory-pipelines-part-1-of-4/) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrpaulandrew%2Fprocfwk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrpaulandrew%2Fprocfwk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrpaulandrew%2Fprocfwk/lists"}