{"id":19733936,"url":"https://github.com/ivan-the-terrible/mssqlserver-adf-mapping","last_synced_at":"2026-05-15T10:04:27.854Z","repository":{"id":250066491,"uuid":"807260074","full_name":"ivan-the-terrible/mssqlserver-adf-mapping","owner":"ivan-the-terrible","description":"An automated tool to analyze your Azure Data Factory Pipelines and SQL Stored Procedures, Views, and Tables to build a diagram to display the dependencies and their relationships.","archived":false,"fork":false,"pushed_at":"2024-06-25T14:10:26.000Z","size":65,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-25T22:14:12.480Z","etag":null,"topics":["adf","mssql-server","sql","sql-server","stored-procedures"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivan-the-terrible.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-28T18:58:46.000Z","updated_at":"2024-08-01T15:24:30.000Z","dependencies_parsed_at":"2024-07-25T01:53:56.324Z","dependency_job_id":null,"html_url":"https://github.com/ivan-the-terrible/mssqlserver-adf-mapping","commit_stats":null,"previous_names":["ivan-the-terrible/mssqlserver-adf-mapping"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivan-the-terrible%2Fmssqlserver-adf-mapping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivan-the-terrible%2Fmssqlserver-adf-mapping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivan-the-terrible%2Fmssqlserver-adf-mapping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivan-the-terrible%2Fmssqlserver-adf-mapping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivan-the-terrible","download_url":"https://codeload.github.com/ivan-the-terrible/mssqlserver-adf-mapping/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241059010,"owners_count":19902296,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adf","mssql-server","sql","sql-server","stored-procedures"],"created_at":"2024-11-12T00:34:34.076Z","updated_at":"2026-05-15T10:04:27.848Z","avatar_url":"https://github.com/ivan-the-terrible.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SQL Server  Dependency Graphing w/ADF Pipelines\n\nThis tool is meant to run and output a count of all tables, views, stored procedures, and pipelines and how they're interlinked.\nCheck out the example output here: \u003chttps://github.com/ivan-the-terrible/mssqlserver-adf-mapping/blob/master/images/test_db/pdf/Test_Pipeline.pdf\u003e\n\n## Logic\n\nBy running the queries within `queries.sql`, you can retrieve and export CSV data that lists the MS SQL Server's tables, views, and stored procedures.\n\nThese values are cross referenced, since stored procedures can reference tables and views, and views can reference tables.\n\nAzure Data Factory can have pipelines that call tables, stored procedures, and other pipelines.\nADF thankfully has version control so the repo can be downloaded locally and the JSON files of the `pipelines` directory can be parsed.\n\n## Structure\n\n```mermaidjs\nproject\n├── data\n│   └── MSSQL_SERVER_DATA_DIR\n│       ├── Tables.csv\n│       ├── View.csv\n│       └── StoredProcedures.csv\n├── images\n│   └── OUTPUT_DIR\n│       ├── json\n│       ├── mermaid\n│       └── pdf\n└── reports\n    └── OUTPUT_DIR\n        ├── table-report.txt\n        ├── view-report.txt\n        ├── stored-procedures-report.txt\n        └── pipeline-report.txt\n```\n\nThe key directories here are: data, images, and reports.\nThe subdirectories within them are named after values within your `.env`.\n\n- `data` contains the source information. It is expected that the user create this directory and another within that is then set to the `MSSQL_SERVER_DATA_DIR` env value. This subdirectory contains the CSVs saved output from `queries.sql`. Please use those queries, select all, and copy.\n  - Tables.csv        -\u003e just Copy\n  - View.csv          -\u003e Copy with Headers\n  - Stored Procedures -\u003e Copy with Headers\n\n- `images` contains the result of taking the Mermaid format of the built trees and outputing it to PDF and JSON. This directory will be created, as will the subdirectory named after the value `OUTPUT_DIR` specified in your `.env`. The raw Mermaid output can also be output if `DEBUG` is set to `True` within your `.env`.\n\n- `reports` contains the text file reports. This directory will be created, as will the subdirectory named after the value `OUTPUT_DIR` specified in your `.env`. The text files contain reference counts and a summary of the linked references.\n\n## Requirements\n\nThe below will require Python, NodeJS, but optionally Azure CLI.\n(Last used with Python 3.12.3 and Node 22.2.0)\n\nYou can use the `install.ps1` script, which will install a Python Virtual Environment and the Mermaid CLI npm package.\nOther than activating the environment, these commands are OS agnostic as well.\n\n- `pip install requirements.txt`\n- `npm install -g @mermaid-js/mermaid-cli`\n\nMermaid CLI repo: \u003chttps://github.com/mermaid-js/mermaid-cli\u003e\nAzure CLI: \u003chttps://learn.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli\u003e\n\nAzure CLI is required to fetch all the JSON files of the Azure Data Factory pipelines IF you do not have access to the source controlled list of pipelines.\nIf you're using/configuring the Azure CLI, you'll basically want to `az login` in the terminal and head to portal.azure.com to view whatever Data Factory you'd like to analyze.\nYou'll need to grab the factory name, resource group, and subscription values and add them to the `.env`, as mentioned in the below section.\n\nAgain, it is expected that the user create the `data` directory here where the script lives and another within that is then set to the `MSSQL_SERVER_DATA_DIR` env value. This subdirectory contains the CSVs saved output from `queries.sql`. Please use those queries, select all, and copy.\n\n- Tables.csv        -\u003e just Copy\n- View.csv          -\u003e Copy with Headers\n- Stored Procedures -\u003e Copy with Headers\n\nInvoking the script is as easy as `python report.py`.\n\n## ENV Values\n\nCreate an `.env` file with the following:\n\n- `PIPELINE_DIR` needs to be set to the directory of whatever ADF pipelines you'd like to parse.\nTypically ADF's version control has the name of the repo and then a directory called pipeline. This directory is used to list all the JSON files within, so whatever you specify as the directory just ensure the JSON files are within it.\n\n- `MSSQL_SERVER_DATA_DIR` is the name of the folder that exists within the `data` directory that contains Tables.csv, Views.csv, and StoredProcedures.csv. The user is expected to create this folder and copy the name to this env value.\n\n- `OUTPUT_DIR` is the name of the folder that exists within the `images` and also the `reports` directory that will be generated to contain images and reports.\n\n- `ADF_FACTORY_NAME` is the Azure Data Factory name.\n\n- `ADF_RESOURCE_GROUP` is the Azure Data Factory resource group.\n\n- `ADF_SUBSCRIPTION` is the Azure Data Factory subscription.\n\n- If you need to create extra output, set `DEBUG=\"True\"` and a `debug` directory will be made with raw class values. See also the raw Mermaid output within the `images` directory as mentioned above.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivan-the-terrible%2Fmssqlserver-adf-mapping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivan-the-terrible%2Fmssqlserver-adf-mapping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivan-the-terrible%2Fmssqlserver-adf-mapping/lists"}