{"id":26522462,"url":"https://github.com/suniljeurkar/snowflakebd","last_synced_at":"2025-03-21T13:27:07.679Z","repository":{"id":283236819,"uuid":"951101011","full_name":"suniljeurkar/snowflakeBD","owner":"suniljeurkar","description":"SnowflakeBD is a Python-based test automation framework designed to perform data integrity, business validation, and performance testing on Snowflake databases. It uses Pytest, Snowflake Python Connector, and RSA key authentication to ensure secure and efficient testing.","archived":false,"fork":false,"pushed_at":"2025-03-19T07:45:22.000Z","size":7685,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-19T08:21:47.789Z","etag":null,"topics":["big-data","python3","query","snowflake","sql"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/suniljeurkar.png","metadata":{"files":{"readme":"ReadMe.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-19T06:51:56.000Z","updated_at":"2025-03-19T07:48:40.000Z","dependencies_parsed_at":"2025-03-19T08:32:09.341Z","dependency_job_id":null,"html_url":"https://github.com/suniljeurkar/snowflakeBD","commit_stats":null,"previous_names":["suniljeurkar/snowflakebd"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suniljeurkar%2FsnowflakeBD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suniljeurkar%2FsnowflakeBD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suniljeurkar%2FsnowflakeBD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suniljeurkar%2FsnowflakeBD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/suniljeurkar","download_url":"https://codeload.github.com/suniljeurkar/snowflakeBD/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244805230,"owners_count":20513237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","python3","query","snowflake","sql"],"created_at":"2025-03-21T13:27:06.939Z","updated_at":"2025-03-21T13:27:07.673Z","avatar_url":"https://github.com/suniljeurkar.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SnowflakeBD\n\n**SnowflakeBD** is a Python-based project designed to perform data validation and integrity checks on Snowflake databases. The project leverages the Snowflake Python Connector and utilizes RSA key pair authentication for secure connections.\n\n## Table of Contents\n\n- [Features](#features)\n- [Prerequisites](#prerequisites)\n- [Setup Instructions](#setup-instructions)\n  - [1. Clone the Repository](#1-clone-the-repository)\n  - [2. Install Dependencies](#2-install-dependencies)\n  - [3. Generate RSA Key Pair using OpenSSL](#3-generate-rsa-key-pair-using-openssl)\n  - [4. Configure Snowflake User with Public Key](#4-configure-snowflake-user-with-public-key)\n  - [5. Set Environment Variables](#5-set-environment-variables)\n  - [6. Run Tests](#6-run-tests)\n- [Project Structure](#project-structure)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Features\n\n- **Data Integrity Tests**: Ensures the consistency and accuracy of data within Snowflake tables.\n- **Business Logic Validation**: Validates that business rules and constraints are correctly enforced in the database.\n- **Performance Testing**: Monitors and reports on query performance to identify potential bottlenecks.\n- **API vs Database Consistency**: Compares data between APIs and the Snowflake database to ensure synchronization.\n\n## Prerequisites\n\n- **Python 3.6 or higher**: Ensure Python is installed on your system.\n- **Snowflake Account**: Access to a Snowflake account with appropriate permissions.\n- **OpenSSL**: Required for generating RSA key pairs.\n\n## Setup Instructions\n\n### 1. Clone the Repository\n\n```bash\ngit clone https://github.com/suniljeurkar/snowflakeBD.git\ncd snowflakeBD\n```\n\n### 2. Install Dependencies\n\nIt's recommended to use a virtual environment:\n\n```bash\npython -m venv env\nsource env/bin/activate  # On Windows: env\\Scripts\\activate\n```\n\nInstall the required Python packages:\n\n```bash\npip install -r requirements.txt\n```\n\n### 3. Generate RSA Key Pair using OpenSSL\n\nFor secure authentication, generate an RSA key pair:\n\n1. **Generate a 2048-bit RSA Private Key**:\n\n   ```bash\n   openssl genrsa -out rsa_key.pem 2048\n   ```\n\n2. **Convert Private Key to PKCS#8 Format**:\n\n   ```bash\n   openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in rsa_key.pem -out rsa_key.p8\n   ```\n\n3. **Extract the Public Key**:\n\n   ```bash\n   openssl rsa -in rsa_key.pem -pubout -out rsa_key.pub\n   ```\n\n   Ensure the public key starts with `-----BEGIN PUBLIC KEY-----`.\n\n**Note**: Keep your private key (`rsa_key.p8`) secure and do not share it.\n\n### 4. Configure Snowflake User with Public Key\n\nAssign the public key to your Snowflake user:\n\n1. Open the `rsa_key.pub` file and copy its contents.\n\n2. In Snowflake, execute:\n\n   ```sql\n   ALTER USER your_username SET RSA_PUBLIC_KEY='paste_public_key_here';\n   ```\n\n   Replace `your_username` with your Snowflake username and `paste_public_key_here` with the content of your public key.\n\n### 5. Set Environment Variables\n\nTo avoid hardcoding sensitive information, set the following environment variables:\n\n```bash\nexport SNOWFLAKE_ACCOUNT='your_account_identifier'\nexport SNOWFLAKE_USER='your_username'\nexport SNOWFLAKE_PRIVATE_KEY_PATH='/path/to/rsa_key.p8'\n```\n\nOn Windows, use:\n\n```cmd\nset SNOWFLAKE_ACCOUNT=your_account_identifier\nset SNOWFLAKE_USER=your_username\nset SNOWFLAKE_PRIVATE_KEY_PATH=C:\\path\\to\\rsa_key.p8\n```\n\n### 6. Run Tests\n\nExecute the test suite using pytest:\n\n```bash\npytest -v\n```\n\n## Project Structure\n\n```\nsnowflakeBD/\n├── BusinessValidation/\n│   └── test_business_logic.py\n├── ConnectionTest/\n│   └── test_connection.py\n├── OrdersValidation/\n│   └── test_orders.py\n├── SupplierValidation/\n│   └── test_suppliers.py\n├── reports/\n│   └── test_reports.py\n├── conftest.py\n└── requirements.txt\n```\n\n- **BusinessValidation/**: Contains tests for business logic validation.\n- **ConnectionTest/**: Tests related to Snowflake connection.\n- **OrdersValidation/**: Validations specific to orders data.\n- **SupplierValidation/**: Validations specific to suppliers data.\n- **reports/**: Scripts for generating test reports.\n- **conftest.py**: Configuration file for pytest fixtures.\n- **requirements.txt**: Lists Python dependencies.\n\n## Contributing\n\nContributions are welcome! Please fork the repository and create a pull request with your changes.\n\n## License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for details.\n\n---\n\nFor a visual guide on setting up key-pair authentication with Snowflake, you might find this video helpful:\n\u003ca href=\"https://www.youtube.com/watch?v=WdCLossuS8U\" target=\"_blank\"\u003e\n  \u003cimg src=\"https://img.youtube.com/vi/WdCLossuS8U/0.jpg\" alt=\"Watch on YouTube\" width=\"600\"\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuniljeurkar%2Fsnowflakebd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuniljeurkar%2Fsnowflakebd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuniljeurkar%2Fsnowflakebd/lists"}