{"id":20535720,"url":"https://github.com/e2fyi/kfx","last_synced_at":"2025-04-14T07:12:56.250Z","repository":{"id":49880622,"uuid":"243732050","full_name":"e2fyi/kfx","owner":"e2fyi","description":"Extensions to kubeflow pipeline sdk.","archived":false,"fork":false,"pushed_at":"2021-06-09T05:50:55.000Z","size":138,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-27T20:51:20.735Z","etag":null,"topics":["kfx","kubeflow","kubeflow-pipeline-task","kubeflow-pipeline-ui","kubeflow-pipelines","python","visualizations"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/e2fyi.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-28T10:04:54.000Z","updated_at":"2021-06-09T05:50:57.000Z","dependencies_parsed_at":"2022-09-10T08:40:35.645Z","dependency_job_id":null,"html_url":"https://github.com/e2fyi/kfx","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2fyi%2Fkfx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2fyi%2Fkfx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2fyi%2Fkfx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2fyi%2Fkfx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/e2fyi","download_url":"https://codeload.github.com/e2fyi/kfx/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248441433,"owners_count":21103990,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kfx","kubeflow","kubeflow-pipeline-task","kubeflow-pipeline-ui","kubeflow-pipelines","python","visualizations"],"created_at":"2024-11-16T00:33:40.091Z","updated_at":"2025-04-14T07:12:56.218Z","avatar_url":"https://github.com/e2fyi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kfx\n\n[![PyPI version](https://badge.fury.io/py/kfx.svg)](https://badge.fury.io/py/kfx)\n[![Build Status](https://travis-ci.com/e2fyi/kfx.svg?branch=master)](https://travis-ci.com/e2fyi/kfx)\n[![Coverage Status](https://coveralls.io/repos/github/e2fyi/kfx/badge.svg?branch=master)](https://coveralls.io/github/e2fyi/kfx?branch=master)\n[![Documentation Status](https://readthedocs.org/projects/kfx/badge/?version=latest)](https://kfx.readthedocs.io/en/latest/?badge=latest)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Downloads](https://pepy.tech/badge/kfx/month)](https://pepy.tech/project/kfx/month)\n\n`kfx` is a python package with the namespace `kfx`. Currently, it provides the\nfollowing sub-packages\n\n- `kfx.lib.dsl` - Extensions to the kubeflow pipeline dsl.\n\n- `kfx.lib.vis` - Data models and helpers to help generate the  `mlpipeline-metrics.json` and `mlpipeline-ui-metadata.json` required to render visualization in the kubeflow pipeline UI. See also https://www.kubeflow.org/docs/pipelines/sdk/pipelines-metrics/ and https://www.kubeflow.org/docs/pipelines/sdk/output-viewer/\n\n\u003e - Documentation: [https://kfx.readthedocs.io](https://kfx.readthedocs.io).\n\u003e - Repo: [https://github.com/e2fyi/kfx](https://github.com/e2fyi/kfx)\n\n\u003e ### NOTE this is currently alpha\n\u003e\n\u003e There will likely to have breaking changes, and feel free to do a feature request\n\u003e\n\u003e ### Known issues\n\u003e\n\u003e - `kfx.vis.vega.vega_web_app` and `KfpArtifact` does not work well together (see example) because of CORs - the web app is hosted inside an iFrame which prevents it from accessing the `ml-pipeline-ui` API server.\n\u003e - `kfx.vis.vega.vega_web_app` is only supported in the latest kubeflow pipeline UI (as inline is only supported after `0.2.5`)\n\n### Changelog\n\nRefer to [CHANGELOG.md](./CHANGELOG.md).\n\n## Quick start\n\nInstallation\n\n```bash\npip install kfx\n```\n\n## Usage\n\nExample: Using `ContainerOpTransform` to configure the internal k8s properties\nof kubeflow pipelines tasks.\n\n\u003e `kfx.dsl.ContainerOpTransform` is a helper to modify the interal k8s properties\n\u003e (e.g. resources, environment variables, etc) of kubeflow pipeline tasks.\n\n```python\nimport kfp.components\nimport kfp.dsl\nimport kfx.dsl\n\ntransforms = (\n    kfx.dsl.ContainerOpTransform()\n    .set_resources(cpu=\"500m\", memory=(\"1G\", \"4G\"))\n    .set_image_pull_policy(\"Always\")\n    .set_env_vars({\"ENV\": \"production\"})\n    .set_env_var_from_secret(\"AWS_ACCESS_KEY\", secret_name=\"aws\", secret_key=\"access_key\")\n    .set_annotations({\"iam.amazonaws.com/role\": \"some-arn\"})\n)\n\n\n@kfp.dsl.components.func_to_container_op\ndef echo(text: str) -\u003e str:\n    print(text)\n    return text\n\n\n@kfp.dsl.pipeline(name=\"demo\")\ndef pipeline(text: str):\n    op1 = echo(text)\n    op2 = echo(\"%s-%s\" % text)\n\n    # u can apply the transform on op1 only\n    # op1.apply(transforms)\n\n    # or apply on all ops in the pipeline\n    kfp.dsl.get_pipeline_conf().add_op_transformer(transforms)\n\n```\n\nExample: Using `ArtifactLocationHelper` and `KfpArtifact` to determine the\nuri of your data artifact generated by the kubeflow pipeline task.\n\n\u003e `kfx.dsl.ArtifactLocationHelper` is a helper to modify the kubeflow pipeline task\n\u003e so that you can use `kfx.dsl.KfpArtifact` to represent the artifact generated\n\u003e inside the task.\n\n```python\nimport kfp.components\nimport kfp.dsl\nimport kfx.dsl\n\n\n# creates the helper that has the argo configs (tells you how artifacts will be stored)\n# see https://github.com/argoproj/argo/blob/master/docs/workflow-controller-configmap.yaml\nhelper = kfx.dsl.ArtifactLocationHelper(\n    scheme=\"minio\", bucket=\"mlpipeline\", key_prefix=\"artifacts/\"\n)\n\n@kfp.components.func_to_container_op\ndef test_op(\n    mlpipeline_ui_metadata: OutputTextFile(str), markdown_data_file: OutputTextFile(str)\n):\n    \"A test kubeflow pipeline task.\"\n\n    import json\n\n    import kfx.dsl\n    import kfx.vis\n    import kfx.vis.vega\n\n    # `KfpArtifact` provides the reference to data artifact created\n    # inside this task\n    spec = {\n        \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.json\",\n        \"description\": \"A simple bar chart\",\n        \"data\": {\n            \"values\": [\n                {\"a\": \"A\", \"b\": 28},\n                {\"a\": \"B\", \"b\": 55},\n                {\"a\": \"C\", \"b\": 43},\n                {\"a\": \"D\", \"b\": 91},\n                {\"a\": \"E\", \"b\": 81},\n                {\"a\": \"F\", \"b\": 53},\n                {\"a\": \"G\", \"b\": 19},\n                {\"a\": \"H\", \"b\": 87},\n                {\"a\": \"I\", \"b\": 52},\n            ]\n        },\n        \"mark\": \"bar\",\n        \"encoding\": {\n            \"x\": {\"field\": \"a\", \"type\": \"ordinal\"},\n            \"y\": {\"field\": \"b\", \"type\": \"quantitative\"},\n        },\n    }\n\n    # write the markdown to the `markdown-data` artifact\n    markdown_data_file.write(\"### hello world\")\n\n    # creates an ui metadata object\n    ui_metadata = kfx.vis.kfp_ui_metadata(\n        # Describes the vis to generate in the kubeflow pipeline UI.\n        [\n            # markdown vis from a markdown artifact.\n            # `KfpArtifact` provides the reference to data artifact created\n            # inside this task\n            kfx.vis.markdown(kfx.dsl.KfpArtifact(\"markdown_data_file\")),\n            # a vega web app from the vega data artifact.\n            kfx.vis.vega.vega_web_app(spec),\n        ]\n    )\n\n    # writes the ui metadata object as the `mlpipeline-ui-metadata` artifact\n    mlpipeline_ui_metadata.write(kfx.vis.asjson(ui_metadata))\n\n    # prints the uri to the markdown artifact\n    print(ui_metadata.outputs[0].source)\n\n\n@kfp.dsl.pipeline()\ndef test_pipeline():\n    \"A test kubeflow pipeline\"\n\n    op: kfp.dsl.ContainerOp = test_op()\n\n    # modify kfp operator with artifact location metadata through env vars\n    op.apply(helper.set_envs())\n\n```\n\nExample: Using `pydantic` data models to generate `mlpipeline-metrics.json` and\n`mlpipeline-ui-metadata.json`.\n\n(See also https://www.kubeflow.org/docs/pipelines/sdk/output-viewer/ and\nhttps://www.kubeflow.org/docs/pipelines/sdk/pipelines-metrics/).\n\n\u003e `kfx.vis` has helper functions (with corresponding hints) to describe and\n\u003e create `mlpipeline-metrics.json` and `mlpipeline-ui-metadata.json` files\n\u003e (required by kubeflow pipeline UI to render any metrics or visualizations).\n\n```python\nimport functools\n\nimport kfp.components\n\n\n# install kfx\nkfx_component = functools.partial(kfp.components.func_to_container_op, packages_to_install=[\"kfx\"])\n\n\n@kfx_component\ndef some_op(\n    # mlpipeline_metrics is a path - i.e. open(mlpipeline_metrics, \"w\")\n    mlpipeline_metrics: kfp.components.OutputPath(str),\n    # mlpipeline_ui_metadata is a FileLike obj - i.e. mlpipeline_ui_metadata.write(\"something\")\n    mlpipeline_ui_metadata: kfp.components.OutputTextFile(str),\n):\n    \"kfp operator that provides metrics and metadata for visualizations.\"\n\n    # import inside kfp task\n    import kfx.vis\n\n    # output metrics to mlpipeline_metrics path\n    kfx.vis.kfp_metrics([\n        # render as percent\n        kfx.vis.kfp_metric(\"recall-score\", 0.9, percent=true),\n        # override metric format with custom value\n        kfx.vis.kfp_metric(name=\"percision-score\", value=0.8, metric_format=\"PERCENTAGE\"),\n        # render raw score\n        kfx.vis.kfp_metric(\"raw-score\", 123.45),\n    ]).write_to(mlpipeline_metrics)\n\n    # output visualization metadata to mlpipeline_ui_metadata obj\n    kfx.vis.kfp_ui_metadata(\n        [\n            # creates a confusion matrix vis\n            kfx.vis.confusion_matrix(\n                source=\"gs://your_project/your_bucket/your_cm_file\",\n                labels=[\"True\", \"False\"],\n            ),\n            # creates a markdown with inline source\n            kfx.vis.markdown(\n                \"# Inline Markdown: [A link](https://www.kubeflow.org/)\",\n                storage=\"inline\",\n            ),\n            # creates a markdown with a remote source\n            kfx.vis.markdown(\n                \"gs://your_project/your_bucket/your_markdown_file\",\n            ),\n            # creates a ROC curve with a remote source\n            kfx.vis.roc(\n                \"gs://your_project/your_bucket/your_roc_file\",\n            ),\n            # creates a Table with a remote source\n            kfx.vis.table(\n                \"gs://your_project/your_bucket/your_csv_file\",\n                header=[\"col1\", \"col2\"],\n            ),\n            # creates a tensorboard viewer\n            kfx.vis.tensorboard(\n                \"gs://your_project/your_bucket/logs/*\",\n            ),\n            # creates a custom web app from a remote html file\n            kfx.vis.web_app(\n                \"gs://your_project/your_bucket/your_html_file\",\n            ),\n            # creates a Vega-Lite vis as a web app\n            kfx.vis.vega.vega_web_app(spec={\n                \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.json\",\n                \"description\": \"A simple bar chart with embedded data.\",\n                \"data\": {\n                    \"values\": [\n                        {\"a\": \"A\", \"b\": 28}, {\"a\": \"B\", \"b\": 55}, {\"a\": \"C\", \"b\": 43},\n                        {\"a\": \"D\", \"b\": 91}, {\"a\": \"E\", \"b\": 81}, {\"a\": \"F\", \"b\": 53},\n                        {\"a\": \"G\", \"b\": 19}, {\"a\": \"H\", \"b\": 87}, {\"a\": \"I\", \"b\": 52}\n                    ]\n                },\n                \"mark\": \"bar\",\n                \"encoding\": {\n                    \"x\": {\"field\": \"a\", \"type\": \"ordinal\"},\n                    \"y\": {\"field\": \"b\", \"type\": \"quantitative\"}\n                }\n            })\n        ]\n    ).write_to(mlpipeline_ui_metadata)\n```\n\n## Developer guide\n\nThis project used:\n\n- isort: to manage import order\n- pylint: to manage general coding best practices\n- flake8: to manage code complexity and coding best practices\n- black: to manage formats and styles\n- pydocstyle: to manage docstr style/format\n- pytest/coverage: to manage unit tests and code coverage\n- bandit: to find common security issues\n- pyenv: to manage dev env: python version (3.6)\n- pipenv: to manage dev env: python packages\n\nConvention for unit tests are to suffix with `_test` and colocate with the actual\npython module - i.e. `\u003cmodule_name\u003e_test.py`.\n\nThe version of the package is read from `version.txt` - i.e. please update the\nappropriate semantic version (major -\u003e breaking changes, minor -\u003e new features, patch -\u003e bug fix, postfix -\u003e pre-release/post-release).\n\n### `Makefile`:\n\n```bash\n# autoformat codes with docformatter, isort, and black\nmake format\n\n# check style, formats, and code complexity\nmake check\n\n# check style, formats, code complexity, and run unit tests\nmake test\n\n# test everything including building the package and check the sdist\nmake test-all\n\n# run unit test only\nmake test-only\n\n# generate and update the requirements.txt and requirements-dev.txt\nmake requirements\n\n# generate the docs with sphinx and autoapi extension\nmake docs\n\n# generate distributions\nmake dists\n\n# publish to pypi with twine (twine must be configured)\nmake publish\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fe2fyi%2Fkfx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fe2fyi%2Fkfx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fe2fyi%2Fkfx/lists"}