{"id":13566022,"url":"https://github.com/okp4/data-join-tabular","last_synced_at":"2025-04-03T23:30:57.816Z","repository":{"id":103067816,"uuid":"510679241","full_name":"okp4/data-join-tabular","owner":"okp4","description":"Tabular data join service","archived":false,"fork":false,"pushed_at":"2024-02-13T09:35:05.000Z","size":23946,"stargazers_count":3,"open_issues_count":12,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-04T19:42:32.113Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/okp4.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-05T09:58:02.000Z","updated_at":"2023-01-11T09:20:32.000Z","dependencies_parsed_at":"2024-08-01T13:39:13.675Z","dependency_job_id":null,"html_url":"https://github.com/okp4/data-join-tabular","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":"okp4/template-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okp4%2Fdata-join-tabular","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okp4%2Fdata-join-tabular/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okp4%2Fdata-join-tabular/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okp4%2Fdata-join-tabular/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/okp4","download_url":"https://codeload.github.com/okp4/data-join-tabular/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247097644,"owners_count":20883122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:02:00.282Z","updated_at":"2025-04-03T23:30:52.809Z","avatar_url":"https://github.com/okp4.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Data-join-tabular\n\n[![version](https://img.shields.io/github/v/release/okp4/data-join-tabular?style=for-the-badge\u0026logo=github)](https://github.com/okp4/data-join-tabular/releases)\n[![lint](https://img.shields.io/github/actions/workflow/status/okp4/data-join-tabular/lint.yml?branch=main\u0026label=lint\u0026style=for-the-badge\u0026logo=github)](https://github.com/okp4/data-join-tabular/actions/workflows/lint.yml)\n[![build](https://img.shields.io/github/actions/workflow/status/okp4/data-join-tabular/build.yml?branch=main\u0026label=build\u0026style=for-the-badge\u0026logo=github)](https://github.com/okp4/data-join-tabular/actions/workflows/build.yml)\n[![test](https://img.shields.io/github/actions/workflow/status/okp4/data-join-tabular/test.yml?branch=main\u0026label=test\u0026style=for-the-badge\u0026logo=github)](https://github.com/okp4/data-join-tabular/actions/workflows/test.yml)\n[![codecov](https://img.shields.io/codecov/c/github/okp4/data-join-tabular?style=for-the-badge\u0026token=G5OBC2RQKX\u0026logo=codecov)](https://codecov.io/gh/okp4/data-join-tabular)\n[![conventional commits](https://img.shields.io/badge/Conventional%20Commits-1.0.0-yellow.svg?style=for-the-badge\u0026logo=conventionalcommits)](https://conventionalcommits.org)\n[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg?style=for-the-badge)](https://github.com/semantic-release/semantic-release)\n[![contributor covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg?style=for-the-badge)](https://github.com/okp4/.github/blob/main/CODE_OF_CONDUCT.md)\n[![License](https://img.shields.io/badge/License-BSD_3--Clause-blue.svg?style=for-the-badge)](https://opensource.org/licenses/BSD-3-Clause)\n\n## Purpose \u0026 Philosophy\n\nThis repository contains data tabular join service.\n\n### Description\n\n2 sets of input data, giving 1 output with associated data based on a common column.\n\n### Specification\n\n- Read different file format (csv, geojson, shp)\n- Optional argument depending on the type of input file (ex: separator for a csv)\n- The name of the new columns created (suffix, prefix...)\n- Type of join ('left', 'right', 'outer', 'inner', 'cross)\n- Validate the output data\n\n## Technologies\n\n__[pandas.merge](https://pandas.pydata.org/docs/reference/api/pandas.merge.html)__\n\nThe join is done on columns or indexes. If joining columns on columns, the DataFrame indexes will be ignored. Otherwise if joining indexes on indexes or indexes on a column or columns, the index will be passed on. When performing a cross merge, no column specifications to merge on are allowed.\n\n- the use of [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/), [semantic versioning](https://semver.org/) and [semantic releasing](https://github.com/cycjimmy/semantic-release-action) which automates the whole package release workflow including: determining the next version number, generating the release notes, and publishing the artifacts (project tarball, docker images, etc.)\n- a uniform way for managing the project lifecycle (dependencies management, building, testing)\n- KISS principles: simple for developers\n- a consistent coding style\n  \n## Usage\n\nThe usage is given as follows:\n\n```sh\nUsage: data-join-tabular join [OPTIONS]\n\n  Represents cli 'join' command\n\nOptions:\n  -i1, --input1 FILE              path to first file to join  [required]\n  -i2, --input2 FILE              path to second file to join  [required]\n  -s1, --sep1 TEXT                separtor for reading the first file\n  -s2, --sep2 TEXT                separtor for reading the second file\n  -sr, --sufrigh TEXT             the suffix to add to overlapping column\n                                  names in right\n  -sl, --sufleft TEXT             the suffix to add to overlapping column\n                                  names in left\n  -onm, --outname TEXT            output file name, if not provioded, output\n                                  name will be the same as file1\n  -o, --on TEXT                   Column or index level names to join on.\n                                  These must be found in both DataFrames.\n                                  If on is None and not merging on indexes\n                                  then this defaults to the intersection of\n                                  the columns in both DataFrames\n  -v, --validate [one_to_one|one_to_many|many_to_one|many_to_many]\n                                  If specified, checks if merge is of\n                                  specified type.        “one_to_one” or\n                                  “1:1”: check if merge keys are unique in\n                                  both left and right datasets.\n                                  “one_to_many” or “1:m”: check if merge keys\n                                  are unique in left dataset.\n                                  “many_to_one” or “m:1”: check if merge keys\n                                  are unique in right dataset.\n                                  “many_to_many” or “m:m”: allowed, but does\n                                  not result in checks.\n                                  It will raise a MergeError if the validation fails\n  -how, --how [left|right|outer|inner|cross]\n                                  Type of merge to be performed.    left: use\n                                  only keys from left frame, similar to a SQL\n                                  left outer join; preserve key order.\n                                  right: use only keys from right frame,\n                                  similar to a SQL right outer join; preserve\n                                  key order.    outer: use union of keys from\n                                  both frames, similar to a SQL full outer\n                                  join; sort keys lexicographically.    inner:\n                                  use intersection of keys from both frames,\n                                  similar to a SQL inner join; preserve the\n                                  order of the left keys.    cross: creates\n                                  the cartesian product from both frames,\n                                  preserves the order of the left keys.\n  -so, --sort TEXT                Sort the join keys lexicographically in the\n                                  result DataFrame. If False,        the order\n                                  of the join keys depends on the join type\n                                  (how keyword).\n  -or, --onrigh TEXT              Column name to join in the right DataFrame\n  -ol, --onleft TEXT              Column name to join in the left DataFrame, \n                                  it must be sorted to match the on_right columns\n  -out, --output DIRECTORY        output directory where output file will be\n                                  written  [default: .]\n  -f, --force                     overwrite existing file\n  -ft, --fix-types                fix types issues\n  --dry-run                       passthrough, will not write anything\n  --help                          Show this message and exit.\n```\n\n```shell\npoetry run data-join-tabular  join -i1 ./tests/data/inputs1/input_test1.csv -i2 ./tests/data/inputs2/input_test1.csv -o categorie -o statut -o effectif -o genre -s1 ';' -s2 ';' -out ./tests/data -f\n```\n\n## System requirements\n\n### Python\n\nThe repository targets python `3.9` and higher.\n\n### Poetry\n\nThe repository uses [Poetry](https://python-poetry.org) as python packaging and dependency management. Be sure to have it properly installed before.\n\n```sh\n  curl -sSL https://install.python-poetry.org | python3 \n```\n\n### Docker\n\nYou can follow the link below on how to install and configure __Docker__ on your local machine:\n\n- [Docker Install Documentation](https://docs.docker.com/install/)\n\n## What's included\n\nThis template provides the following:\n\n- [poetry](https://python-poetry.org) for dependency management.\n- [flake8](https://flake8.pycqa.org) for linting python code.\n- [mypy](http://mypy-lang.org/) for static type checks.\n- [pytest](https://docs.pytest.org) for unit testing.\n- [click](https://palletsprojects.com/p/click/) to easily setup your project commands\n\nThe project is also configured to enforce code quality by declaring some CI workflows:\n\n- conventional commits\n- lint\n- unit test\n- semantic release\n\n## Everyday activity\n\n### Build\n\nProject is built by [poetry](https://python-poetry.org).\n\n```sh\npoetry install\n```\n\n### Lint\n\n\u003e ⚠️ Be sure to write code compliant with linters or else you'll be rejected by the CI.\n\n__Code linting__ is performed by [flake8](https://flake8.pycqa.org).\n\n```sh\npoetry run flake8 --count --show-source --statistics\n```\n\n__Static type check__ is performed by [mypy](http://mypy-lang.org/).\n\n```sh\npoetry run mypy .\n```\n\nTo improve code quality, we use other linters in our workflows, if you don't want to be rejected by the CI,\nplease check these additional linters.\n\n__Markdown linting__ is performed by [markdownlint-cli](https://github.com/igorshubovych/markdownlint-cli).\n\n```sh\nmarkdownlint \"**/*.md\"  \n```\n\n__Docker linting__ is performed by [dockerfilelint](https://github.com/replicatedhq/dockerfilelint) and\n[hadolint](https://github.com/hadolint/hadolint).\n\n```sh\ndockerfilelint Dockerfile\n```\n\n```sh\nhadolint Dockerfile\n```\n\n### Unit Test\n\n\u003e ⚠️ Be sure to write tests that succeed or else you'll be rejected by the CI.\n\nUnit tests are performed by the [pytest](https://docs.pytest.org) testing framework.\n\n```sh\npoetry run pytest -v\n```\n\n### Build \u0026 run docker image (locally)\n\nBuild a local docker image using the following command line:\n\n```sh\ndocker build -t data-join-tabular .\n```\n\nOnce built, you can run the container locally with the following command line:\n\n```sh\ndocker run -ti --rm data-join-tabular\n```\n\n## You want to get involved? 😍\n\nPlease check out OKP4 health files :\n\n- [Contributing](https://github.com/okp4/.github/blob/main/CONTRIBUTING.md)\n- [Code of conduct](https://github.com/okp4/.github/blob/main/CODE_OF_CONDUCT.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokp4%2Fdata-join-tabular","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fokp4%2Fdata-join-tabular","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokp4%2Fdata-join-tabular/lists"}