{"id":29534419,"url":"https://github.com/amr-yasser226/datagovernanceworkflow","last_synced_at":"2026-02-05T23:32:07.395Z","repository":{"id":286894336,"uuid":"962898470","full_name":"amr-yasser226/DataGovernanceWorkflow","owner":"amr-yasser226","description":"Comprehensive data governance pipeline for SSH honeypot logs—covering data profiling, cleansing, quality assurance, encryption, classification, and GDPR/CCPA/HIPAA compliance. Built with Pandas, Pandera, YData Profiling, and cryptography, with simulated Caesar cipher attacks to demonstrate practical data-security techniques.","archived":false,"fork":false,"pushed_at":"2025-06-23T21:48:45.000Z","size":36899,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-09T06:40:06.601Z","etag":null,"topics":["caesar-cipher","ccpa","cryptography","cybersecurity","data-cleaning","data-encryption","data-governance","data-profiling","data-quality","data-validation","data-visualization","gdpr","hipaa","honeypot-analysis","open-source","pandas","privacy-compliance","python","ssh-logs"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amr-yasser226.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-08T21:03:07.000Z","updated_at":"2025-08-09T17:21:57.000Z","dependencies_parsed_at":"2025-07-17T07:51:53.562Z","dependency_job_id":null,"html_url":"https://github.com/amr-yasser226/DataGovernanceWorkflow","commit_stats":null,"previous_names":["amr-yasser226/datagovernanceworkflow"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/amr-yasser226/DataGovernanceWorkflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amr-yasser226%2FDataGovernanceWorkflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amr-yasser226%2FDataGovernanceWorkflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amr-yasser226%2FDataGovernanceWorkflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amr-yasser226%2FDataGovernanceWorkflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amr-yasser226","download_url":"https://codeload.github.com/amr-yasser226/DataGovernanceWorkflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amr-yasser226%2FDataGovernanceWorkflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29138375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T23:14:48.546Z","status":"ssl_error","status_checked_at":"2026-02-05T23:14:35.724Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["caesar-cipher","ccpa","cryptography","cybersecurity","data-cleaning","data-encryption","data-governance","data-profiling","data-quality","data-validation","data-visualization","gdpr","hipaa","honeypot-analysis","open-source","pandas","privacy-compliance","python","ssh-logs"],"created_at":"2025-07-17T00:07:48.461Z","updated_at":"2026-02-05T23:32:07.364Z","avatar_url":"https://github.com/amr-yasser226.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DataGovernanceWorkflow\n\n## Project Overview\n\nThe DataGovernanceWorkflow repository provides a comprehensive pipeline for managing, profiling, encrypting, and auditing sensitive data. It includes encryption routines, data profiling and quality control notebooks, compliance report generation (GDPR, CCPA, HIPAA), and attack simulation scripts. The workflow is organized to separate raw data, analysis notebooks, scripts, and generated reports for clarity and reproducibility.\n\n## Repository Structure\n\n```\nDataGovernanceWorkflow/\n├── data/              # Raw and processed datasets (CSV, JSON)\n├── scripts/           # Standalone Python scripts for encryption, decryption, and attack simulations\n├── notebooks/         # Jupyter notebooks for interactive exploration and profiling\n├── reports/           # Generated HTML and PDF reports (profiling, compliance, quality control)\n├── requirements.txt   # Python package dependencies\n├── LICENSE            # Project license\n└── README.md          # Project overview and instructions\n```\n\n## Data Directory (`data/`)\n\nContains raw input files and outputs from processing steps:\n\n* `ccpa_compliant.csv`: Data annotated for CCPA compliance (DoNotSell flag and can\\_sell\\_data column).\n* `Cleaned_csv.csv`: Preprocessed dataset used for encryption and profiling.\n* `encrypted_data.csv`: Sensitive columns encrypted using Fernet, Caesar, and Playfair ciphers.\n* `gdpr_compliant.csv`: Data anonymized for GDPR fields (IP, Username, Password, City, Country).\n* `hipaa_report.json`: HIPAA compliance findings in JSON format.\n* `recovered_columns.csv`: Columns recovered after brute-force decryption of Caesar-encrypted fields.\n* `ssh_logs_processed.csv`: SSH log dataset cleaned and formatted for profiling and validation.\n\n## Scripts Directory (`scripts/`)\n\n* `frequency_attack.py`: Implements an improved brute-force attack on Caesar-ciphered columns, diagnoses mismatches, and applies custom fixes to maximize recovery accuracy.\n* `profilling_code.py`: Generates programmatic, text-based profiling of numeric, datetime, and categorical columns, and visualizes login attempt patterns by hour, country, and city.\n\n## Notebooks Directory (`notebooks/`)\n\n1. **Data\\_encryption.ipynb**\n\n   * Reads the cleaned CSV and drops index columns.\n   * Encrypts `Password` with Fernet.\n   * Applies Ceasar cipher (shift=3) to `Username`, `City`, and `Country`.\n   * Assigns usernames to random categories for role-permissions testing.\n   * Integrates GDPR, CCPA, and HIPAA pseudonymization or stub routines, exporting compliance artifacts.\n\n2. **data\\_profiling.ipynb**\n\n   * Uses `ydata_profiling` to generate an HTML profiling report of the SSH log dataset.\n\n3. **profilling\\_code.ipynb**\n\n   * Programmatic profiling: computes summary statistics for each column (numeric, datetime, categorical).\n   * Builds a pandas DataFrame of profiling information and displays it.\n   * Converts and analyzes combined datetime fields and plots login attempts by hour, country, and city.\n\n4. **Quality\\_Control.ipynb**\n\n   * Loads the SSH log data and inspects schema.\n   * Cleans duplicates and missing values (median for numeric, mode for categorical).\n   * Removes outliers based on 1.5 × IQR rule.\n   * Validates the cleaned dataset against a Pandera schema, reporting any failures.\n\n## Reports Directory (`reports/`)\n\n* `profiling_report.html`\n  Interactive HTML summary of data profiling.\n* `profiling_report.pdf`\n  PDF export of the profiling report.\n* `profiling_data_ssh_logs_process.html`\n  HTML rendering of the profiling steps for SSH logs.\n* `Phase 1.pdf`\n  Quality Control notebook report summarizing cleaning, outlier handling, and schema validation.\n\n## Compliance Workflows\n\n1. **GDPR Compliance**\n\n   * Anonymizes IP addresses and pseudonymizes other sensitive fields using `python_gdpr_utils` if available, else a stub based on MD5 hashing.\n   * Outputs `gdpr_compliant.csv`.\n\n2. **CCPA Compliance**\n\n   * Adds `DoNotSell` flag per user with consistent random assignment.\n   * Derives `can_sell_data` column.\n   * Outputs `ccpa_compliant.csv`.\n\n3. **HIPAA Compliance**\n\n   * Runs HIPAA scanners (`HippoScanner`, `TenableIO`, `SecurityMonkey`) if installed, else returns an empty stub.\n   * Outputs `hipaa_report.json`.\n\n## Setup and Usage\n\n1. **Environment Setup**\n\n   ```bash\n   python3 -m venv .venv\n   source .venv/bin/activate\n   pip install -r requirements.txt\n   ```\n\n2. **Run Encryption and Compliance Pipeline**\n\n   ```bash\n   python scripts/frequency_attack.py       # Attacks and recovers encrypted fields\n   # For notebooks, launch Jupyter Lab:\n   jupyter lab notebooks\n   ```\n\n3. **Generate Reports**\n\n   * Open `notebooks/data_profiling.ipynb` to regenerate profiling HTML.\n   * Run `Quality_Control.ipynb` to validate data schema and update the Phase 1 report.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famr-yasser226%2Fdatagovernanceworkflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famr-yasser226%2Fdatagovernanceworkflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famr-yasser226%2Fdatagovernanceworkflow/lists"}