{"id":26406343,"url":"https://github.com/planet-a-ventures/dlt-source-personio","last_synced_at":"2025-03-17T17:19:18.203Z","repository":{"id":282328038,"uuid":"946599267","full_name":"planet-a-ventures/dlt-source-personio","owner":"planet-a-ventures","description":"DLT (www.github.com/dlt-hub/dlt) source for Personio (www.personio.com)","archived":false,"fork":false,"pushed_at":"2025-03-14T00:53:09.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-14T01:43:53.204Z","etag":null,"topics":["data-engineering","data-load-tool","data-loading","dlt","dlthub","personio","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/planet-a-ventures.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-11T11:47:00.000Z","updated_at":"2025-03-14T00:53:12.000Z","dependencies_parsed_at":"2025-03-14T01:44:17.750Z","dependency_job_id":"2949cf2c-5f72-4a10-a4ce-35627588e199","html_url":"https://github.com/planet-a-ventures/dlt-source-personio","commit_stats":null,"previous_names":["planet-a-ventures/dlt-source-personio"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/planet-a-ventures%2Fdlt-source-personio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/planet-a-ventures%2Fdlt-source-personio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/planet-a-ventures%2Fdlt-source-personio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/planet-a-ventures%2Fdlt-source-personio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/planet-a-ventures","download_url":"https://codeload.github.com/planet-a-ventures/dlt-source-personio/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244075644,"owners_count":20393980,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-load-tool","data-loading","dlt","dlthub","personio","python"],"created_at":"2025-03-17T17:19:17.717Z","updated_at":"2025-03-17T17:19:18.183Z","avatar_url":"https://github.com/planet-a-ventures.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\ndescription: dlt source for personio.com\nkeywords: [personio API, personio.com]\n---\n\n# dlt-source-personio\n\n[![PyPI version](https://img.shields.io/pypi/v/dlt-source-personio)](https://pypi.org/project/dlt-source-personio/)\n\n[DLT](https://dlthub.com/) source for [personio](https://www.personio.com/).\n\nCurrently loads the following data:\n\n| Table | Contains | Spec version |\n| -- | -- | -- |\n| `persons` | Items of the `Person` model with all properties | `V2` |\n| `persons_custom_attributes` | All defined custom attributes. The table is pivoted, so each custom attribute becomes a column in the table. | `V2` |\n| `employments` | Items of the `Employment` model with all properties | `V2` |\n\n## Why are you not using the `dlt-hub/verified-sources` personio source / Differences\n\nThe [official verified source](https://github.com/dlt-hub/verified-sources/tree/master/sources/personio)\nhas a few drawbacks:\n\n- it is based on the Personio API V1, not the current V2\n- on usage of the verified source, a copy of the current state of\n  the `dlt-hub/verified-sources` repository is copied into your project;\n  Once you make changes to it, it effectively becomes a fork,\n  making it hard to update after the fact.\n- The verified source does not use any data validation other than\n  ensuring dates are correct; This means that data shape is not guaranteed,\n  resulting in potential schema changes.\n  This data source uses an (unofficial) OpenAPI spec, which is transformed\n  into Pydantic 2 models.\n\nCurrently this data source does not support delta updates\n(the verified source does) and it also does not contain some of the data\nsources (absences, etc.). Contributions are welcome!\n\n## Usage\n\nCreate a `.dlt/secrets.toml` with your API key and email:\n\n```toml\npersonio_client_id = \"papi-...\"\npersonio_client_secret = \"papi-...\"\n```\n\nand then run the default source with optional list references:\n\n```py\nfrom dlt_source_personio import source as personio_source\n\npipeline = dlt.pipeline(\n   pipeline_name=\"personio_pipeline\",\n   destination=\"duckdb\",\n   dev_mode=True,\n)\npersonio_data = personio_source()\npipeline.run(personio_data)\n```\n\n## Development\n\nThis project is using [devenv](https://devenv.sh/).\n\nCommands:\n\n| Command | What does it do? |\n| -- | -- |\n| `generate-model` | generates the personio Pydantic model from the current spec file, applies patches, etc. |\n| `update-spec` | Pulls in the latest `master#HEAD` of [personio/api-docs](https://github.com/personio/api-docs) |\n| `validate-spec` | Validates the local (unofficial) Personio V2 spec |\n| `refresh-model` | Both commands above plus adds it to git and commits the changes. |\n| `format` | Formats \u0026 lints all code |\n| `sample-pipeline-run` | Runs the sample pipeline. By default `dev_mode=True` which fetches resources with a limit of 1 (page) |\n| `sample-pipeline-show` | Starts the streamlit-based dlt hub |\n\n### Run the sample\n\n```sh\nPERSONIO_CLIENT_ID=[...] \\\n   PERSONIO_CLIENT_SECRET=[...] \\\n      sample-pipeline-run\n```\n\nalternatively you can also create a `.dlt/secrets.toml`\n(excluded from git) with the following content:\n\n```toml\npersonio_client_id = \"papi-...\"\npersonio_client_secret = \"papi-...\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplanet-a-ventures%2Fdlt-source-personio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fplanet-a-ventures%2Fdlt-source-personio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplanet-a-ventures%2Fdlt-source-personio/lists"}