{"id":19594495,"url":"https://github.com/salpreh/transpydata","last_synced_at":"2025-11-21T13:03:39.425Z","repository":{"id":54845267,"uuid":"319737534","full_name":"salpreh/transpydata","owner":"salpreh","description":"A minimalist framework for managing migrations","archived":false,"fork":false,"pushed_at":"2021-05-30T23:01:24.000Z","size":90,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-10-28T17:59:51.728Z","etag":null,"topics":["etl","framework","migrations","python","tool"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salpreh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-08T19:17:41.000Z","updated_at":"2023-02-15T10:04:06.000Z","dependencies_parsed_at":"2022-08-14T04:40:10.992Z","dependency_job_id":null,"html_url":"https://github.com/salpreh/transpydata","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/salpreh/transpydata","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salpreh%2Ftranspydata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salpreh%2Ftranspydata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salpreh%2Ftranspydata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salpreh%2Ftranspydata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salpreh","download_url":"https://codeload.github.com/salpreh/transpydata/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salpreh%2Ftranspydata/sbom","scorecard":{"id":796608,"data":{"date":"2025-08-11","repo":{"name":"github.com/salpreh/transpydata","commit":"88cecf53f93b316e8fd95983c28a918ca2f41cf4"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.5,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/pr_checks.yml:1","Warn: no topLevel permission defined: .github/workflows/pypublish.yml:1","Warn: no topLevel permission defined: .github/workflows/pypublish_test.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'develop'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pr_checks.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/salpreh/transpydata/pr_checks.yml/develop?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pypublish.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/salpreh/transpydata/pypublish.yml/develop?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pypublish.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/salpreh/transpydata/pypublish.yml/develop?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pypublish_test.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/salpreh/transpydata/pypublish_test.yml/develop?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pypublish_test.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/salpreh/transpydata/pypublish_test.yml/develop?enable=pin","Warn: containerImage not pinned by hash: docker/transpydata/Dockerfile:1: pin your Docker image by updating python:3.8-alpine to python:3.8-alpine@sha256:3d93b1f77efce339aa77db726656872517b0d67837989aa7c4b35bd5ae7e81ba","Warn: containerImage not pinned by hash: examples/mysql_to_http/docker/dummy_server/Dockerfile:1: pin your Docker image by updating php:7.0-apache to php:7.0-apache@sha256:1d34b2e491a02ba7a8d26478132015e197a5ffea37f0a93b42621d11cfe042cc","Warn: pipCommand not pinned by hash: docker/transpydata/Dockerfile:9","Warn: pipCommand not pinned by hash: .github/workflows/pypublish.yml:26","Warn: pipCommand not pinned by hash: .github/workflows/pypublish.yml:27","Warn: pipCommand not pinned by hash: .github/workflows/pypublish_test.yml:26","Warn: pipCommand not pinned by hash: .github/workflows/pypublish_test.yml:27","Info:   0 out of   5 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 containerImage dependencies pinned","Info:   0 out of   5 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 4 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"35 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2021-865 / GHSA-vv2x-vrpj-qqpq","Warn: Project is vulnerable to: PYSEC-2022-42986 / GHSA-43fp-rhv2-5gv8","Warn: Project is vulnerable to: PYSEC-2023-135 / GHSA-xqr8-7jwr-rhp7","Warn: Project is vulnerable to: GHSA-3ww4-gg4f-jr7f","Warn: Project is vulnerable to: GHSA-5cpq-8wj7-hf2v","Warn: Project is vulnerable to: GHSA-9v9h-cgj8-h64p","Warn: Project is vulnerable to: PYSEC-2023-254 / GHSA-jfhm-5ghh-2f97","Warn: Project is vulnerable to: GHSA-jm77-qphf-c4w8","Warn: Project is vulnerable to: PYSEC-2021-63 / GHSA-rhm9-p9w5-fwm7","Warn: Project is vulnerable to: GHSA-v8gr-m533-ghj9","Warn: Project is vulnerable to: GHSA-w7pp-m8wf-vj6r","Warn: Project is vulnerable to: GHSA-x4qr-2fvf-3mr5","Warn: Project is vulnerable to: PYSEC-2024-60 / GHSA-jjg7-2v4v-x38h","Warn: Project is vulnerable to: GHSA-cpwx-vrp4-4pq7","Warn: Project is vulnerable to: PYSEC-2021-66 / GHSA-g3rq-g295-4j3m","Warn: Project is vulnerable to: GHSA-h5c8-rqwp-cp95","Warn: Project is vulnerable to: GHSA-h75v-3vvj-5mfj","Warn: Project is vulnerable to: GHSA-q2x7-8rv6-6q7h","Warn: Project is vulnerable to: GHSA-hgjp-83m4-h4fj","Warn: Project is vulnerable to: PYSEC-2022-48 / GHSA-77rm-9x9h-xj3g","Warn: Project is vulnerable to: GHSA-8gq9-2x98-w8hf","Warn: Project is vulnerable to: GHSA-8qvm-5x2c-j2w7","Warn: Project is vulnerable to: PYSEC-2021-140 / GHSA-9w8r-397f-prfh","Warn: Project is vulnerable to: PYSEC-2023-117 / GHSA-mrwq-x4v8-fh7p","Warn: Project is vulnerable to: PYSEC-2021-141 / GHSA-pq64-v7f5-gqh8","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: GHSA-g7vv-2v7x-gj9p","Warn: Project is vulnerable to: GHSA-34jh-p97f-mpxf","Warn: Project is vulnerable to: PYSEC-2021-59 / GHSA-5phf-pp7p-vc2r","Warn: Project is vulnerable to: PYSEC-2023-212 / GHSA-g4mx-q9vg-27p4","Warn: Project is vulnerable to: GHSA-pq67-6m6q-mj2v","Warn: Project is vulnerable to: PYSEC-2021-108 / GHSA-q2q7-5pp4-w6pg","Warn: Project is vulnerable to: PYSEC-2023-192 / GHSA-v845-jxx5-vc9f"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-23T09:11:19.548Z","repository_id":54845267,"created_at":"2025-08-23T09:11:19.548Z","updated_at":"2025-08-23T09:11:19.548Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285620632,"owners_count":27203062,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-21T02:00:06.175Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","framework","migrations","python","tool"],"created_at":"2024-11-11T08:43:45.069Z","updated_at":"2025-11-21T13:03:39.409Z","avatar_url":"https://github.com/salpreh.png","language":"Python","readme":"# TransPyData\n\n[![PyPI version](https://badge.fury.io/py/transpydata.svg)](https://badge.fury.io/py/transpydata)\n[![PyPI version](https://img.shields.io/github/license/salpreh/transpydata.svg)](https://img.shields.io/github/license/salpreh/transpydata.svg)\n\n**A minimal framework for managing migrations**\n\n---\n\n## Overview\nTransPyData implements a generic pipeline to perform migrations. It has 2 main components. First one is `TransPy` class, which executes the migration pipeline according to a configuration. Second the _data services_ implementations (`IDataInput`, `IDataProcess` and `IDataOutput`), this services manages how data is gathered, processed and sent to the new destination.\n\n### TransPy\nThe `TransPy` class manages the migration pipeline. It needs to be provided with an instance of: \n- `IDataInput`: Manages the gathering of source data.\n- `IDataProcess`: Manages data transformation and filtering prior to pass it to the data output.\n- `IDataOutput`: Manages data sending to the new destination.\n\n_**NOTE**: Data services overview below_\n\nApart from the data services there are other optional configurations:\n```python\ntrans_py = TransPy()\n\nconfig = {\n  'datainput_source': [], # If working with single record pipeline this should be an iterable of data to feed IDataInput\n  'datainput_by_one': False, # Enable single record pipeline on input\n  'dataprocess_by_one': False, # Enable single record pipeline on processing\n  'dataoutput_by_one': False, # Enable single record pipeline on output\n}\ntrans_py.configure(config)\n```\n\nThe values in the snippet are the defaults, so by default the migration will move all migration data through the pipeline at once.\n\n#### All processing mode\nWhen all data services have the \"_by\\_one_\" flag to `False` the migration will move all data at once through the pipeline. So the `TransPy` instance will call the method `get_all` of `IDataInput` configured to get all input data, with the response will call `process_all` of `IDataProcess`, and with the response of `IDataProcess` will call `send_all` of `IDataOutput`. Finally a list with `IDataOutput` results is returned by `TransPy`.\n\n#### Single record mode\nIf \"_by\\_one_\" flags are `True` the data are \"_queried_\" by one and moved through all the pipeline. The `IDataOutput` return are accumulated and returned as list at the end of the processing, so the `TransPy` return type is the same.\n\nThere are some additional cases, what if ***datainput*** and ***dataprocess*** are in \"_by\\_one_\" mode and dataoutput not? In this case the data is gathered and processed one by one, at the end of processing (`IDataProcess`) the results are accumulated and the `IDataOutput` is called with all data. Similar case when ***dataprocess*** and ***dataoutput*** are in \"_by\\_one_\" mode, data is gathered all at once and then piped one by one through `IDataProcess` and `IDataOutput`.\n\n### Data services\n_under construction_\n\n## Getting started\nTo start a migration create an instance of `TransPy` and configure it. At least instances of `IDataInput`, `IDataProcess` and `IDataOutput` needs to be provided. Prior to starting the migration the data services might need to be configured too. Here is an code example:\n\n```python\nimport json\n\nfrom transpydata import TransPy\nfrom transpydata.config.datainput.MysqlDataInput import MysqlDataInput\nfrom transpydata.config.dataprocess.NoneDataProcess import NoneDataProcess\nfrom transpydata.config.dataoutput.RequestDataOutput import RequestDataOutput\n\n\ndef main():\n    # Configure imput\n    mysql_input = MysqlDataInput()\n\n    config = {\n        'db_config': {\n            'user': 'root',\n            'password': 'TryingTh1ngs',\n            'host': 'localhost',\n            'port': '3306',\n            'database': 'migration'\n        },\n        'get_one_query': None, # We'll go with all query\n        'get_all_query': \"\"\"\n            SELECT s.staff_Id, s.staff_name, s.staff_grade, m.module_Id, m.module_name\n            FROM staff s\n            LEFT JOIN teaches t ON s.staff_Id = t.staff_Id\n            LEFT JOIN module m ON t.module_Id = m.module_Id\n        \"\"\",\n        'all_query_params': {} # No where clause, no interpolation\n    }\n    mysql_input.configure(config)\n\n    # Configure process\n    none_process = NoneDataProcess()\n\n    # Configure output\n    request_output = RequestDataOutput()\n    request_output.configure({\n        'url': 'http://localhost:8008',\n        'req_verb': 'POST',\n        'headers': {\n            'content-type': 'application/json',\n            'accept-encoding': 'application/json',\n            'x-app-id': 'MT1'\n        },\n        'encode_json': True,\n        'json_response': True\n    })\n\n    # Configure TransPy\n    trans_py = TransPy()\n    trans_py.datainput = mysql_input\n    trans_py.dataprocess = none_process\n    trans_py.dataoutput = request_output\n\n    res = trans_py.run()\n    print(json.dumps(res))\n\nif __name__ == '__main__':\n    main()\n```\n\nFull working example could be found at `examples/mysql_to_http/`, there is a [docker-compose](https://docs.docker.com/compose/gettingstarted/#step-6-re-build-and-run-the-app-with-compose) to launch an instance of mysql and a webserver.\n\n## Custom data services\nFor now you can check the interfaces `IDataInput`, `IDataProcess` and `IDataOutput` to see what needs to be implemented in a custom data service.\n\n_(I'll improve this section in the future)_\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalpreh%2Ftranspydata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalpreh%2Ftranspydata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalpreh%2Ftranspydata/lists"}