{"id":27648195,"url":"https://github.com/nathanthorell/sql-tools","last_synced_at":"2025-04-24T02:34:53.414Z","repository":{"id":280902557,"uuid":"940831673","full_name":"nathanthorell/sql-tools","owner":"nathanthorell","description":"A collection of utility tools for working with various dialects of SQL databases","archived":false,"fork":false,"pushed_at":"2025-04-22T02:43:31.000Z","size":73,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T03:24:30.807Z","etag":null,"topics":["mssql","python","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nathanthorell.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-28T21:55:48.000Z","updated_at":"2025-04-22T02:43:33.000Z","dependencies_parsed_at":"2025-04-01T02:29:45.774Z","dependency_job_id":"12efa6a2-a0b6-43b8-97fb-5e09e9744138","html_url":"https://github.com/nathanthorell/sql-tools","commit_stats":null,"previous_names":["nathanthorell/mssql-tools","nathanthorell/sql-tools"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fsql-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fsql-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fsql-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fsql-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nathanthorell","download_url":"https://codeload.github.com/nathanthorell/sql-tools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250548320,"owners_count":21448643,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mssql","python","sql"],"created_at":"2025-04-24T02:34:48.249Z","updated_at":"2025-04-24T02:34:53.404Z","avatar_url":"https://github.com/nathanthorell.png","language":"Python","readme":"# sql-tools\n\nA collection of utility tools for working with various dialects of SQL databases.\n\n## Features\n\n- **Object Comparison Tool** (`object_compare`): Compare definitions of stored procedures, views, functions, tables, triggers, and sequences across different environments (DEV, QA, UAT, PROD)\n\n  - Identify exclusive objects that exist in only one environment\n  - Check for definition differences in objects across environments\n\n- **Stored Procedure Tester** (`usp_tester`): Batch test execution of stored procedures with configurable parameters\n  - Support for default parameter values\n  - Execution time tracking\n  - Different logging levels (summary, verbose)\n\n- **View Tester** (`view_tester`): Batch test queries against views\n  - Runs a \"TOP 1 *\" for each view to ensure output is valid\n  - Execution time tracking\n  - Different logging levels (summary, verbose)\n\n- **Schema Size** (`schema_size`): Analyzes storage across databases by measuring schema sizes.\n  - The tool connects to multiple servers, calculates data and index space consumption in megabytes, and generates formatted tabular reports comparing schema sizes.\n  - Results are displayed with customizable detail levels based on logging preferences.\n\n- **Data Compare** (`data_compare`): Compare data across different database platforms\n  - Support for MSSQL and PostgreSQL databases\n  - Compare data using custom SQL queries\n  - Option to use query files for complex comparisons\n  - Detailed reporting on differences between datasets\n\n## Installation\n\n### Requirements\n\n- Python 3.13+\n- Appropriate database drivers:\n\n    - ODBC Driver for SQL Server (for MSSQL databases)\n    - More drivers to be added for other database types\n\n### Setup\n\n1. Clone the repository:\n\n   ```bash\n   git clone https://github.com/nathanthorell/sql-tools.git\n   cd sql-tools\n   ```\n\n2. Create and activate a virtual environment, then install the package:\n\n   ```bash\n   make install\n   ```\n\n   Or manually:\n\n   ```bash\n   python -m venv .venv\n   source .venv/bin/activate  # On Windows: .venv\\Scripts\\activate\n   pip install -e \".[dev]\"\n   ```\n\n3. Create a `.env` file based on the provided `.env.example`:\n\n   ```bash\n   cp .env.example .env\n   ```\n\n   Then update the connection strings with your database details.\n\n4. Create a `config.toml` file for each tool you want to use.\n\n## Configuration\n\n### Environment Variables\n\nAn `.env.example` file is provided with the repository. Copy this to create your own `.env` file:\n\n```bash\ncp .env.example .env\n```\n\nThen adjust the connection strings and other settings according to your environment.\n\nThe tools will read these environment variables to establish connections to the various SQL instances.\n\n### Tool Configurations\n\nCreate a `config.toml` file in the project root directory based on the provided `config-example.toml`:\n\n```bash\ncp config-example.toml config.toml\n```\n\n- **Object Compare**: Set the schema name to compare across environments\n- **USP Tester**: Configure the schema, logging level, and default parameter values for stored procedures\n- **View Tester**: Configure the schema and logging level\n- **Schema Size**: Configure the server connections, databases to compare, and logging level\n- **Data Compare**: Configure named comparison pairs with left/right database connections, database types (MSSQL/PostgreSQL), and queries or query files to compare\n\n## Usage\n\n### Object Comparison Tool\n\n```bash\nobject_compare\n```\n\nThis will:\n\n1. Connect to each configured environment using the specified connection strings\n1. Compare object definitions across environments for each object type\n1. Report differences in object definitions using checksums\n1. Highlight objects that exist in one environment but not others\n\n### Stored Procedure Tester\n\n```bash\nusp_tester\n```\n\nThis will:\n\n1. Connect to the configured test database\n1. Execute all stored procedures in the specified schema\n1. Apply default parameter values\n1. Report execution status and timing\n\n### View Tester\n\n```bash\nview_tester\n```\n\nThis will:\n\n1. Connect to the configured test database\n1. Execute all views in the specified schema\n1. Report execution status and timing\n\n### Schema Size\n\n```bash\nschema_size\n```\n\nThis will:\n\n1. Connect to each server using the specified connection strings\n1. Calculate size metrics for each database and schema\n1. Generate reports showing data and index sizes per schema\n1. Provide comparative summaries across all servers and databases\n\n### Data Compare\n\n```bash\ndata_compare\n```\n\n1. Connect to the configured database sources (supports both MSSQL and PostgreSQL)\n1. Execute the defined queries against both data sources\n1. Compare the results of both queries\n1. Generate a detailed report of matching and non-matching data\n1. Display performance comparison between execution time of each data source\n\n## Development\n\n### Linting and Formatting\n\n```bash\nmake lint    # Run ruff and mypy linters\nmake format  # Format code with ruff\n```\n\n### Clean Up\n\n```bash\nmake clean   # Remove temporary files and virtual environment\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnathanthorell%2Fsql-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnathanthorell%2Fsql-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnathanthorell%2Fsql-tools/lists"}