{"id":13737769,"url":"https://github.com/vmware/versatile-data-kit","last_synced_at":"2025-05-15T20:07:49.121Z","repository":{"id":37099697,"uuid":"387724608","full_name":"vmware/versatile-data-kit","owner":"vmware","description":"One framework to develop, deploy and operate data workflows with Python and SQL.","archived":false,"fork":false,"pushed_at":"2025-04-07T17:40:49.000Z","size":115499,"stargazers_count":449,"open_issues_count":248,"forks_count":59,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-05-08T04:37:16.058Z","etag":null,"topics":["analytics","data","data-engineer","data-engineering","data-engineering-pipeline","data-lineage","data-pipelines","data-science","data-structures","data-warehouse","database","dataops","elt","etl","pipeline","python","snowflake","sql","trino","warehouse"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vmware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":"support/git-commit-template.txt","governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-07-20T08:28:09.000Z","updated_at":"2025-05-06T08:55:26.000Z","dependencies_parsed_at":"2024-01-05T10:31:01.124Z","dependency_job_id":"fe4d9d2f-f2c5-4e20-bb74-6303c718e2ae","html_url":"https://github.com/vmware/versatile-data-kit","commit_stats":{"total_commits":1151,"total_committers":37,"mean_commits":31.10810810810811,"dds":0.7158992180712425,"last_synced_commit":"71e2de7a0d650d1319670a016d71c1d45a496fdd"},"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fversatile-data-kit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fversatile-data-kit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fversatile-data-kit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmware%2Fversatile-data-kit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vmware","download_url":"https://codeload.github.com/vmware/versatile-data-kit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254414501,"owners_count":22067272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","data","data-engineer","data-engineering","data-engineering-pipeline","data-lineage","data-pipelines","data-science","data-structures","data-warehouse","database","dataops","elt","etl","pipeline","python","snowflake","sql","trino","warehouse"],"created_at":"2024-08-03T03:02:00.338Z","updated_at":"2025-05-15T20:07:44.057Z","avatar_url":"https://github.com/vmware.png","language":"Python","readme":"![Versatile Data Kit](./support/images/versatile-data-kit.svg#gh-light-mode-only)\n![Versatile Data Kit](./support/images/versatile-data-kit.svg#gh-dark-mode-only)\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/vmware/versatile-data-kit/pulse\" alt=\"Activity\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/commit-activity/m/vmware/versatile-data-kit\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/vmware/versatile-data-kit/contributors\" alt=\"Last Activity\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/last-commit/vmware/versatile-data-kit\" alt=\"Last Activity\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://pypistats.org/packages/vdk-core\" alt=\"Monthly Downloads\"\u003e\n        \u003cimg src=\"https://img.shields.io/pypi/dm/vdk-core.svg\" alt=\"monthly download count for vdk-core\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/vmware/versatile-data-kit/blob/main/LICENSE\" alt=\"License\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/license/vmware/versatile-data-kit\" alt=\"license\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/pre-commit/pre-commit\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white\" alt=\"pre-commit\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/vmware/versatile-data-kit\"\u003e\n        \u003cimg src=\"https://gitlab.com/vmware-analytics/versatile-data-kit/badges/main/pipeline.svg\" alt=\"build status\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://twitter.com/intent/tweet?text=Wow: @VDKProject\"\u003e\n        \u003cimg src=\"https://img.shields.io/twitter/url?style=social\u0026url=https%3A%2F%2Ftwitter.com%2FVDKProject\" alt=\"twitter\"/\u003e\u003c/a\u003e\n     \u003ca href=\"https://www.youtube.com/channel/UCasf2Q7X8nF7S4VEmcTHJ0Q\"\u003e\n        \u003cimg alt=\"YouTube Channel Subscribers\" src=\"https://img.shields.io/youtube/channel/subscribers/UCasf2Q7X8nF7S4VEmcTHJ0Q?style=social\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003c!-- TODO: code coverage --\u003e\n\n---\n\n\u003c!-- Added the line here to have some mental space between the icons, as no one in the interviews saw the first sentence --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cspan\u003e One framework to🧑‍💻 Develop \u003c/span\u003e \u003cspan\u003e ▶️ Deploy \u003c/span\u003e \u003cspan\u003eand 📊 Operate \u003c/span\u003e \u003cbr\u003e\n   \u003cspan\u003edata workflows with Python and SQL\u003cspan\u003e\n\u003c/p\u003e\n\n---\n\n\u003cdiv align=\"left\"\u003e\n    \u003cspan\u003e🎯 Write shorter, more readable code. \u003c/span\u003e\u003cbr\u003e\n\u003c/div\u003e\n\u003cdiv align=\"left\"\u003e\n    \u003cspan\u003e🔄 Ready-to-use data ETL/ELT patterns. \u003c/span\u003e\u003cbr\u003e\n\u003c/div\u003e\n\u003cdiv align=\"left\"\u003e\n    \u003cspan\u003e🧩 Lego-like extensibility. \u003c/span\u003e\u003cbr\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\u003cdiv align=\"left\"\u003e\n    \u003cspan\u003e🚀 Single click deployment. \u003c/span\u003e\u003cbr\u003e\n\u003c/div\u003e\n\u003cdiv align=\"left\"\u003e\n    \u003cspan\u003e🛠 Operate and monitor. ️\u003c/span\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n---\n\n\u003cp id=\"intro\" align=\"center\"\u003e\n    \u003ca href=\"#intro\"\u003e\u003cimg src=\"https://img.shields.io/badge/VDK%20SDK-blue\" alt=\"Intro to VDK SDK\"\u003e\u003c/a\u003e\n    \u003ca href=\"#ingestion\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Ingestion-lightblue\" alt=\"Ingestion\"\u003e\u003c/a\u003e\n    \u003ca href=\"#transformation\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Transformation-lightblue\" alt=\"Transformation\"\u003e\u003c/a\u003e\n    \u003ca href=\"#deployment\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Deployment-lightblue\" alt=\"Job Deployment\"\u003e\u003c/a\u003e\n    \u003ca href=\"#operations\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Operations-lightblue\" alt=\"Job Operations\"\u003e\u003c/a\u003e\n    \u003ca href=\"#plugins\"\u003e\u003cimg src=\"https://img.shields.io/badge/Lego%20like%20Extensibility-lightblue\" alt=\"Extensibility\"\u003e\u003c/a\u003e\n    \u003ca href=\"#contributing\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support%20and%20Contributing-lightblue\" alt=\"Support and Contributing\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003ch1\u003eIntroduction to the VDK SDK\u003c/h1\u003e\n\n\u003ctable width=\"100%\" \u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cul\u003e\n          \u003cli\u003eFramework to simplify data ingestion and data processing.\u003c/li\u003e\n          \u003cli\u003eWrite any code using Python or SQL.\u003c/li\u003e\n          \u003cli\u003eA toolset enabling you to run data jobs.\u003c/li\u003e\n        \u003c/ul\u003e\n        \u003chr\u003e\n        \u003cp\u003e\u003cb\u003eGet started with VDK SDK:\u003c/b\u003e\u003c/p\u003e\n        \u003cspan\u003e\u0026#10145; \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Quickstart-VDK\"\u003eInstall Quickstart VDK\u003c/a\u003e. Only requirement is Python 3.7+.\u003c/span\u003e\u003cbr\u003e\n        \u003cpre\u003e\u003ccode class=\"bash\"\u003epip install quickstart-vdk\nvdk --help\u003c/code\u003e\u003c/pre\u003e\n        \u003cspan\u003e\u0026#10145; Develop your \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/First-Data-Job\"\u003eFirst Data Job\u003c/a\u003e if you are impatient to start quickly.\u003c/span\u003e\u003cbr\u003e\n    \u003c/td\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cvideo controls loop autoplay muted preload=\"metadata\" src=\"https://github.com/vmware/versatile-data-kit/assets/2536458/1e768ddb-c622-45e2-bfd9-20dd2ae9c1dd\" aria-label=\"30 seconds video tutorial of getting started and quickstart-vdk installation\"\u003e\u003c/video\u003e\n        \u003c!-- link to the video above https://www.canva.com/design/DAFqSRgP3xs/Tp5rVRywkwUQdUEsmWCcfw/edit?utm_content=DAFqSRgP3xs --\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n\n\n\n\n\u003cbr/\u003e\u003cbr/\u003e\n\u003cp id=\"ingestion\" align=\"center\"\u003e\n    \u003ca href=\"#intro\"\u003e\u003cimg src=\"https://img.shields.io/badge/VDK%20SDK-lightblue\" alt=\"Intro to VDK SDK\"\u003e\u003c/a\u003e\n    \u003ca href=\"#ingestion\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Ingestion-blue\" alt=\"Ingestion\"\u003e\u003c/a\u003e\n    \u003ca href=\"#transformation\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Transformation-lightblue\" alt=\"Transformation\"\u003e\u003c/a\u003e\n    \u003ca href=\"#deployment\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Deployment-lightblue\" alt=\"Job Deployment\"\u003e\u003c/a\u003e\n    \u003ca href=\"#operations\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Operations-lightblue\" alt=\"Job Operations\"\u003e\u003c/a\u003e\n    \u003ca href=\"#plugins\"\u003e\u003cimg src=\"https://img.shields.io/badge/Lego%20like%20Extensibility-lightblue\" alt=\"Extensibility\"\u003e\u003c/a\u003e\n    \u003ca href=\"#contributing\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support%20and%20Contributing-lightblue\" alt=\"Support and Contributing\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003ch1\u003eData Ingestion\u003c/h1\u003e\n\n\u003ctable width=\"100%\" \u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cul\u003e\n          \u003cli\u003eExtract data from various sources (HTTP APIs, Databases, CSV, etc.).\u003c/li\u003e\n          \u003cli\u003eEnsure data fidelity with minimal transformations.\u003c/li\u003e\n          \u003cli\u003eLoad data to your preferred destination (database, cloud storage).\u003c/li\u003e\n        \u003c/ul\u003e\n        \u003chr\u003e\n        \u003cp\u003e\u003cb\u003eIngestion examples:\u003c/b\u003e\u003c/p\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Ingesting-data-from-REST-API-into-Database\"\u003eIngesting data from REST API into Database\u003c/a\u003e \u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Ingesting-data-from-DB-into-Database\"\u003eIngesting data from DB into Database\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Ingesting-local-CSV-file-into-Database\"\u003eIngesting local CSV file into Database\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/blob/main/examples/incremental-ingest-from-db-example/README.md\"\u003eIncremental ingestion using Job Properties\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n    \u003c/td\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cvideo controls loop muted preload=\"metadata\" src=\"https://github.com/vmware/versatile-data-kit/assets/2536458/a46a5ecb-63f7-4b4f-9e2d-099e80522dac\" aria-label=\"30 seconds video tutorial for Data Ingestion\"\u003e\u003c/video\u003e\n        \u003c!-- link to the video above https://www.canva.com/design/DAFq7Tc2vNk/3Qfi7ge5nOkEAbAEohEzgw/edit?utm_content=DAFq7Tc2vNk --\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n\n\n\n\n\n\u003cbr/\u003e\u003cbr/\u003e\n\u003cp id=\"transformation\" align=\"center\"\u003e\n    \u003ca href=\"#intro\"\u003e\u003cimg src=\"https://img.shields.io/badge/VDK%20SDK-lightblue\" alt=\"Intro to VDK SDK\"\u003e\u003c/a\u003e\n    \u003ca href=\"#ingestion\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Ingestion-lightblue\" alt=\"Ingestion\"\u003e\u003c/a\u003e\n    \u003ca href=\"#transformation\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Transformation-blue\" alt=\"Transformation\"\u003e\u003c/a\u003e\n    \u003ca href=\"#deployment\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Deployment-lightblue\" alt=\"Job Deployment\"\u003e\u003c/a\u003e\n    \u003ca href=\"#operations\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Operations-lightblue\" alt=\"Job Operations\"\u003e\u003c/a\u003e\n    \u003ca href=\"#plugins\"\u003e\u003cimg src=\"https://img.shields.io/badge/Lego%20like%20Extensibility-lightblue\" alt=\"Extensibility\"\u003e\u003c/a\u003e\n    \u003ca href=\"#contributing\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support%20and%20Contributing-lightblue\" alt=\"Support and Contributing\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch1\u003eData Transformation\u003c/h1\u003e\n\n\u003ctable width=\"100%\" \u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cul\u003e\n          \u003cli\u003eSQL and Python parameterized transformations.\u003c/li\u003e\n          \u003cli\u003eExtensible templates for data modeling.\u003c/li\u003e\n          \u003cli\u003eCreates a dataset or table as a product.\u003c/li\u003e\n        \u003c/ul\u003e\n        \u003chr\u003e\n        \u003cp\u003e\u003cb\u003eGet started with transforming data:\u003c/b\u003e\u003c/p\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Data-Modeling-Guide:-Treating-Data-as-a-Product\"\u003eData Modeling: Treating Data as a Product\u003c/a\u003e \u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Processing-data-using-SQL-and-local-database\"\u003eProcessing data using SQL and local database\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/SQL-Data-Processing-templates-examples\"\u003eProcessing data using Kimball warehousing templates\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n    \u003c/td\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cvideo controls loop muted preload=\"metadata\" src=\"https://github.com/vmware/versatile-data-kit/assets/2536458/56be0a37-257e-4f9d-a558-afd012715dcb\" aria-label=\"30 seconds video tutorial for Data Transformation\"\u003e\u003c/video\u003e\n        \u003c!-- link to the video above https://www.canva.com/design/DAFqxoSAXdM/oOG1pB0ekgrCCnKUyaB60Q/edit?utm_content=DAFqxoSAXdM --\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n\n\n\n\n\n\u003cbr/\u003e\u003cbr/\u003e\n\u003cp id=\"deployment\" align=\"center\"\u003e\n    \u003ca href=\"#intro\"\u003e\u003cimg src=\"https://img.shields.io/badge/VDK%20SDK-lightblue\" alt=\"Intro to VDK SDK\"\u003e\u003c/a\u003e\n    \u003ca href=\"#ingestion\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Ingestion-lightblue\" alt=\"Ingestion\"\u003e\u003c/a\u003e\n    \u003ca href=\"#transformation\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Transformation-lightblue\" alt=\"Transformation\"\u003e\u003c/a\u003e\n    \u003ca href=\"#deployment\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Deployment-blue\" alt=\"Job Deployment\"\u003e\u003c/a\u003e\n    \u003ca href=\"#operations\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Operations-lightblue\" alt=\"Job Operations\"\u003e\u003c/a\u003e\n    \u003ca href=\"#plugins\"\u003e\u003cimg src=\"https://img.shields.io/badge/Lego%20like%20Extensibility-lightblue\" alt=\"Extensibility\"\u003e\u003c/a\u003e\n    \u003ca href=\"#contributing\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support%20and%20Contributing-lightblue\" alt=\"Support and Contributing\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch1\u003eData Job Deployment (build, deploy, release)\u003c/h1\u003e\n\n\u003ctable width=\"100%\" \u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cspan\u003eVDK Control Service provides REST API for users to create, deploy, manage, and execute data jobs in a Kubernetes runtime environment.\u003c/span\u003e\n        \u003cul\u003e\n          \u003cli\u003eScheduling, packaging, dependencies management, deployment.\u003c/li\u003e\n          \u003cli\u003eExecution management and monitoring.\u003c/li\u003e\n          \u003cli\u003eSource code versioning and tracking. Fast rollback.\u003c/li\u003e\n          \u003cli\u003eManage state and credentials using Properties and Secrets.\u003c/li\u003e\n        \u003c/ul\u003e\n        \u003chr\u003e\n        \u003cp\u003e\u003cb\u003eGet started with deploying jobs in control service:\u003c/b\u003e\u003c/p\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Versatile-Data-Kit-Control-Service#install-locally\"\u003eInstall Local Control Service with vdk server --install\u003c/a\u003e \u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Scheduling-a-Data-Job-for-automatic-execution\"\u003eScheduling a Data Job for automatic execution\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/wiki/Using-VDK-DAGs-to-orchestrate-data-jobs\"\u003eUsing VDK DAGs to orchestrate Data Jobs\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n    \u003c/td\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cvideo controls loop muted preload=\"metadata\" src=\"https://github.com/vmware/versatile-data-kit/assets/2536458/5bf96910-ff27-4ecf-a631-c0e64e1db249\" aria-label=\"30 seconds video tutorial for VDK Control Service\"\u003e\u003c/video\u003e\n        \u003c!-- link to the video above https://www.canva.com/design/DAFsIj6BtVY/QMLKEYNGCJZRHSIJmYW99A/edit?utm_content=DAFsIj6BtVY --\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n\n\n\n\n\n\n\n\n\u003cbr/\u003e\u003cbr/\u003e\n\u003cp id=\"operations\" align=\"center\"\u003e\n    \u003ca href=\"#intro\"\u003e\u003cimg src=\"https://img.shields.io/badge/VDK%20SDK-lightblue\" alt=\"Intro to VDK SDK\"\u003e\u003c/a\u003e\n    \u003ca href=\"#ingestion\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Ingestion-lightblue\" alt=\"Ingestion\"\u003e\u003c/a\u003e\n    \u003ca href=\"#transformation\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Transformation-lightblue\" alt=\"Transformation\"\u003e\u003c/a\u003e\n    \u003ca href=\"#deployment\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Deployment-lightblue\" alt=\"Job Deployment\"\u003e\u003c/a\u003e\n    \u003ca href=\"#operations\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Operations-blue\" alt=\"Job Operations\"\u003e\u003c/a\u003e\n    \u003ca href=\"#plugins\"\u003e\u003cimg src=\"https://img.shields.io/badge/Lego%20like%20Extensibility-lightblue\" alt=\"Extensibility\"\u003e\u003c/a\u003e\n    \u003ca href=\"#contributing\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support%20and%20Contributing-lightblue\" alt=\"Support and Contributing\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch1\u003eOperations and Monitoring\u003c/h1\u003e\n\n\u003ctable width=\"100%\" \u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cul\u003e\n          \u003cli\u003eUse Operations UI to monitor, troubleshoot data workloads in production.\u003c/li\u003e\n          \u003cli\u003eNotifications for errors during Data Job deployment or execution.\u003c/li\u003e\n          \u003cli\u003eRoute errors to correct people by classifying them into User or Platform errors.\u003c/li\u003e\n        \u003c/ul\u003e\n        \u003chr\u003e\n        \u003cp\u003e\u003cb\u003eGet started with operating and monitoring data jobs:\u003c/b\u003e\u003c/p\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://youtu.be/DLRGCCGUp0U?t=111\"\u003eVersatile Data Kit UI - Installation and Getting Started\u003c/a\u003e \u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://www.youtube.com/watch?v=9BkAOSvXuUg\"\u003eVDK Operations User Interface - Versatile Data Kit\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n    \u003c/td\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cvideo controls loop muted preload=\"metadata\" src=\"https://github.com/vmware/versatile-data-kit/assets/2536458/4e4b5677-267d-4b9b-9b61-65ae7623f161\" aria-label=\"30 seconds video tutorial for VDK Operations UI\"\u003e\u003c/video\u003e\n        \u003c!-- link to the video above https://www.canva.com/design/DAFsPZyGThU/nx5DNB7Ybzjx6OEkyjRVBQ/edit?utm_content=DAFsPZyGThU --\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n\n\n\n\u003cbr/\u003e\u003cbr/\u003e\n\u003cp id=\"plugins\" align=\"center\"\u003e\n    \u003ca href=\"#intro\"\u003e\u003cimg src=\"https://img.shields.io/badge/VDK%20SDK-lightblue\" alt=\"Intro to VDK SDK\"\u003e\u003c/a\u003e\n    \u003ca href=\"#ingestion\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Ingestion-lightblue\" alt=\"Ingestion\"\u003e\u003c/a\u003e\n    \u003ca href=\"#transformation\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Transformation-lightblue\" alt=\"Transformation\"\u003e\u003c/a\u003e\n    \u003ca href=\"#deployment\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Deployment-lightblue\" alt=\"Job Deployment\"\u003e\u003c/a\u003e\n    \u003ca href=\"#operations\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Operations-lightblue\" alt=\"Job Operations\"\u003e\u003c/a\u003e\n    \u003ca href=\"#plugins\"\u003e\u003cimg src=\"https://img.shields.io/badge/Lego%20like%20Extensibility-blue\" alt=\"Extensibility\"\u003e\u003c/a\u003e\n    \u003ca href=\"#contributing\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support%20and%20Contributing-lightblue\" alt=\"Support and Contributing\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003ch1\u003eLego like extensibility\u003c/h1\u003e\n\n\u003ctable width=\"100%\" \u003e\n  \u003ctr\u003e\n    \u003ctd width=\"50%\" \u003e\n        \u003cul\u003e\n          \u003cli\u003eModular: use only what you need. Extensible: build what you miss.\u003c/li\u003e\n          \u003cli\u003eEasy to install any plugins as python packages using \u003ccode\u003epip\u003c/code\u003e.\u003c/li\u003e\n          \u003cli\u003eUsed in enhancing data processing, ingestion, job execution, command-line lifecycle\u003c/li\u003e\n        \u003c/ul\u003e\n        \u003chr\u003e\n        \u003cp\u003e\u003cb\u003eGet started with using some VDK plugins:\u003c/b\u003e\u003c/p\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins\"\u003eBrowse available plugins\u003c/a\u003e \u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  Interesting plugins to check out:\u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; \u003ca href=\"https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/vdk-lineage#vdk-lineage\"\u003eTrack Lineage of your jobs using vdk-lineage\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; \u003ca href=\"https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/vdk-csv#versatile-data-kit-csv-plugin\"\u003eImport/Ingest or Export CSV files using vdk-csv\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n        \u003cspan\u003e\u0026#10145;  \u003ca href=\"https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins#write-your-own-plugin\"\u003eWrite your own plugin\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\n    \u003c/td\u003e\n    \u003ctd width=\"50%\" \u003e\n      \u003cvideo controls loop muted preload=\"metadata\" src=\"https://github.com/vmware/versatile-data-kit/assets/2536458/2fea5d6e-3fe4-475f-91ec-89838f18ebf3\" aria-label=\"30 seconds video tutorial for VDK Plugins\"\u003e\u003c/video\u003e\n        \u003c!-- link to the video above https://www.canva.com/design/DAFsQ3IeJy4/1b2x-HgStJAbztUwGANX5w/edit?utm_content=DAFsQ3IeJy4 --\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n\n\n\n\n\n\n\n\n\u003cbr/\u003e\u003cbr/\u003e\n\u003cp id=\"contributing\" align=\"center\"\u003e\n    \u003ca href=\"#intro\"\u003e\u003cimg src=\"https://img.shields.io/badge/VDK%20SDK-lightblue\" alt=\"Intro to VDK SDK\"\u003e\u003c/a\u003e\n    \u003ca href=\"#ingestion\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Ingestion-lightblue\" alt=\"Ingestion\"\u003e\u003c/a\u003e\n    \u003ca href=\"#transformation\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Transformation-lightblue\" alt=\"Transformation\"\u003e\u003c/a\u003e\n    \u003ca href=\"#deployment\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Deployment-lightblue\" alt=\"Job Deployment\"\u003e\u003c/a\u003e\n    \u003ca href=\"#operations\"\u003e\u003cimg src=\"https://img.shields.io/badge/Data%20Job%20Operations-lightblue\" alt=\"Job Operations\"\u003e\u003c/a\u003e\n    \u003ca href=\"#plugins\"\u003e\u003cimg src=\"https://img.shields.io/badge/Lego%20like%20Extensibility-lightblue\" alt=\"Extensibility\"\u003e\u003c/a\u003e\n    \u003ca href=\"#contributing\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support%20and%20Contributing-blue\" alt=\"Support and Contributing\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n# Support and Contributing\nFor Support, you can join our Slack channel, create an [issue](https://github.com/vmware/versatile-data-kit/issues) or [pull request](https://github.com/vmware/versatile-data-kit/pulls) on GitHub to submit suggestions or changes. \u003cbr\u003e\nIf you are interested in contributing as a developer, visit the [contributing](https://github.com/vmware/versatile-data-kit/blob/main/CONTRIBUTING.md) page.\n\n# Contacts\n- Message us on Slack:\u003cbr\u003e\n    ☝️ Join the [CNCF Slack workspace](https://communityinviter.com/apps/cloud-native/cncf).\u003cbr\u003e\n    ✌️ Join the [#versatile-data-kit](https://cloud-native.slack.com/archives/C033PSLKCPR) channel.\n- Join the [next Community Meeting](https://github.com/vmware/versatile-data-kit/wiki/Community-Meetings)\n- Follow us on [Twitter](https://twitter.com/VDKProject).\n- Subscribe to the [Versatile Data Kit YouTube Channel](https://www.youtube.com/channel/UCasf2Q7X8nF7S4VEmcTHJ0Q).\n- Join our [development mailing list](mailto:join-versatiledatakit@groups.vmware.com), used by developers and maintainers of VDK.\n\n# Code of Conduct\nEveryone involved in working on the project's source code, or engaging in any issue trackers, Slack channels,\nand mailing lists is expected to be familiar with and follow the [Code of Conduct](https://github.com/vmware/versatile-data-kit/blob/main/CODE_OF_CONDUCT.md).\n","funding_links":[],"categories":["Python","Slowly Changing Dimensions Data Modeling"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmware%2Fversatile-data-kit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvmware%2Fversatile-data-kit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmware%2Fversatile-data-kit/lists"}