{"id":24175848,"url":"https://github.com/iht/vertex-tfx-pipeline","last_synced_at":"2025-09-20T20:31:22.678Z","repository":{"id":65251273,"uuid":"556190359","full_name":"iht/vertex-tfx-pipeline","owner":"iht","description":"An example of TFX intended to work with Vertex AI in Google Cloud","archived":false,"fork":false,"pushed_at":"2023-08-07T09:21:43.000Z","size":67020,"stargazers_count":7,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-18T00:13:56.859Z","etag":null,"topics":["google-cloud","mlops","tfx","vertex-ai"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iht.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-23T09:15:46.000Z","updated_at":"2024-04-18T00:13:56.860Z","dependencies_parsed_at":"2023-02-12T03:31:13.935Z","dependency_job_id":null,"html_url":"https://github.com/iht/vertex-tfx-pipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iht%2Fvertex-tfx-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iht%2Fvertex-tfx-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iht%2Fvertex-tfx-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iht%2Fvertex-tfx-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iht","download_url":"https://codeload.github.com/iht/vertex-tfx-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233684413,"owners_count":18713888,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-cloud","mlops","tfx","vertex-ai"],"created_at":"2025-01-13T02:33:22.051Z","updated_at":"2025-09-20T20:31:14.990Z","avatar_url":"https://github.com/iht.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TFX pipelines with Vertex AI\n\n## Setup\n\nThis project can be run from the Cloud Shell of your Google Cloud project.\n\nYou will need a Google Cloud project with owner permissions, and you also need \nto have the [Google Cloud SDK configured to use that project](https://cloud.google.com/sdk/docs/install-sdk).\nFor instance, you could use the [Cloud Shell in your Google Cloud project](https://cloud.google.com/shell/docs), \nwhich is configured by default with the Google Cloud SDK.\n\n### Setup Google Cloud project\n\nThis repository contains some Terraform code in the `terraform` directory to setup \nVertex AI and all the required APIs and permissions in the Google Cloud project.\n\nPlease check the README.md in the [terraform/](terraform/) directory for more details. \nYou only need to run the Terraform code once.\n\n### Prepare the data\nPROJECT_ID=\u003cPROJECT_ID\u003e\ngcloud storage cp data/creditcard.csv.gz gs://$PROJECT_ID/data/\n\nbq load --project_id $PROJECT_ID --autodetect --source_format=CSV --replace=true data_playground.transactions gs://$PROJECT_ID/data/creditcard.csv.gz\n\n## Running the pipeline\n\n### Python version\n\nPlease don't use Python \u003c 3.7 (e.g. 3.6) or Python \u003e 3.9 (e.g. 3.10), they will \nnot work with TFX. For more details, please check:\n\n* https://www.tensorflow.org/tfx\n* https://github.com/tensorflow/tfx\n\nAt the moment of writing this, the Cloud Shell has Python 3.9. You can check your \nPython version by running the following command:\n\n```shell\npython --version\n```\n\nOnce you have made sure you have the correct Python version, create a virtualenv: \n\n```shell\npython -m venv tfxenv\n```\n\nActivate it:\n\n```shell\nsource ./tfxenv/bin/activate\n```\n\nAnd install the dependencies in the file `requirements.txt`, by running:\n\n```shell\npip install -r requirements.txt\n```\n\n### Run the pipeline\n\nEdit the scripts in the directory `scripts` to point to your project id and region \nof choice.\n\nThe `playground` branch of this repository contains incomplete code that you need to\nfinish, as an exercise to learn the ropes of TFX pipelines.\n\nTo run the pipeline in Google Cloud, you need to run the provided scripts from the \ntop level directory of the repository:\n\n```shell\n./scripts/launch_google_cloud.sh\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiht%2Fvertex-tfx-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiht%2Fvertex-tfx-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiht%2Fvertex-tfx-pipeline/lists"}