{"id":17508147,"url":"https://github.com/d3m1s4m/sparse2json","last_synced_at":"2026-04-08T16:32:38.963Z","repository":{"id":258407292,"uuid":"871112391","full_name":"d3m1s4m/sparse2json","owner":"d3m1s4m","description":"A tool to identify and convert sparse columns in PostgreSQL databases into JSON format, optimizing data structure and improving query performance, particularly for datasets with high null value percentages like OpenStreetMap.","archived":false,"fork":false,"pushed_at":"2024-10-16T21:37:33.000Z","size":43,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-28T22:34:47.561Z","etag":null,"topics":["database","json","openstreetmap","postgresql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/d3m1s4m.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-11T09:44:38.000Z","updated_at":"2024-10-16T21:37:36.000Z","dependencies_parsed_at":"2024-10-18T21:47:36.954Z","dependency_job_id":null,"html_url":"https://github.com/d3m1s4m/sparse2json","commit_stats":null,"previous_names":["d3m1s4m/sparse2json"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/d3m1s4m/sparse2json","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d3m1s4m%2Fsparse2json","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d3m1s4m%2Fsparse2json/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d3m1s4m%2Fsparse2json/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d3m1s4m%2Fsparse2json/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/d3m1s4m","download_url":"https://codeload.github.com/d3m1s4m/sparse2json/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d3m1s4m%2Fsparse2json/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31564900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","json","openstreetmap","postgresql","python"],"created_at":"2024-10-20T04:12:29.658Z","updated_at":"2026-04-08T16:32:38.943Z","avatar_url":"https://github.com/d3m1s4m.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sparse2JSON\n\nSparse2JSON is a powerful tool designed to identify and convert sparse columns in PostgreSQL databases into a single JSONB column. This tool is particularly useful for datasets, such as those derived from OpenStreetMap (OSM), where many columns contain a high percentage of null values. By consolidating sparse columns into a JSON format, users can optimize their database structure and improve query performance.\n\n## Table of Contents\n\n- [Features](#features)\n- [Getting Started](#getting-started)\n- [Usage](#usage)\n- [Contributing](#contributing)\n\n## Features\n\n- **Identify Sparse Columns**: Automatically detects columns with a significant percentage of NULL values.\n- **Convert Sparse Columns**: Migrates sparse data into a new JSONB column, enhancing database efficiency.\n- **Transaction Management**: Ensures data integrity through proper transaction handling.\n- **Comprehensive Logging**: Tracks the process and errors with detailed logging.\n- **Flexible Configuration**: Adjust parameters for minimum row counts (default: 30 row) and sparsity percentages (default: 40% of NULL values).\n- **Multi-threading for Performance**: Utilizes multi-threading to parallelize the conversion process for multiple tables, significantly improving performance on larger datasets.\n- **Safe Password Handling**: Ensures secure password entry to protect sensitive user information.\n\n## Getting Started\n\n### Prerequisites\n\nTo use Sparse2JSON, ensure you have the following installed:\n\n- Python 3.x\n- PostgreSQL\n- Required Python libraries (install via requirements.txt)\n\n### Installation\n\n1. Clone the repository:\n\n   ```bash\n   git clone https://github.com/d3m1s4m/Sparse2JSON\n   cd sparse2json\n   ```\n   \n2. Install the required packages:\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n## Usage\n\nSparse2JSON can be executed from the command line to identify and convert sparse columns in PostgreSQL databases. Below are detailed instructions on how to use the tool effectively, including all available command-line options.\n\n### Command-Line Arguments\n\n- `-H` or `--host`: The hostname of the PostgreSQL server (Required).\n- `-P` or `--port`: The port of the PostgreSQL server. defaults to **5432** if not specified.\n- `-U` or `--user`: The username for database authentication (Required).\n- `-D` or `--dbname`: The name of the database to connect to (Required).\n- `-T` or `--table`: Specify a particular table to analyze for sparse columns. If omitted, the tool will analyze all tables in the database (Optional).\n- `-v` or `--verbose`: Enable verbose output for detailed logging during execution. This provides real-time feedback about the process and any errors encountered.\n- `-h` or `--help`: for help and more information.\n\n### Example Usage\n\n1. **Analyze a Specific Table**: To analyze a specific table and convert its sparse columns:\n   ```bash\n   python main.py -H=\u003cDB_HOST\u003e -D \u003cDB_NAME\u003e -U \u003cDB_USER\u003e -T=\u003cDB_TABLE\u003e -v\n   ```\n2. **Analyze the Entire Database**: To analyze all tables in the specified database, omit the -T argument:\n   ```bash\n   python main.py -H=\u003cDB_HOST\u003e -D \u003cDB_NAME\u003e -U \u003cDB_USER\u003e -v\n   ```\n\n## Contributing\n\nContributions are welcome! To contribute to Sparse2JSON, follow these steps:\n\n1. Fork the repository.\n2. Create a new feature branch:\n\n   ```bash\n   git checkout -b feature/YourFeatureName\n   ```\n\n3. Make your changes and commit them:\n\n   ```bash\n   git commit -m 'Add a new feature'\n   ```\n\n4. Push to the branch:\n\n   ```bash\n   git push origin feature/YourFeatureName\n   ```\n\n5. Open a pull request.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd3m1s4m%2Fsparse2json","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fd3m1s4m%2Fsparse2json","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd3m1s4m%2Fsparse2json/lists"}