{"id":15978125,"url":"https://github.com/yu-iskw/tensorflow-hub-with-ml-engine","last_synced_at":"2025-03-17T22:31:10.594Z","repository":{"id":145587361,"uuid":"133695891","full_name":"yu-iskw/tensorflow-hub-with-ml-engine","owner":"yu-iskw","description":"Example of transfer learning with tensorflow-hub and Google ML Engine","archived":false,"fork":false,"pushed_at":"2018-05-22T17:05:21.000Z","size":23,"stargazers_count":3,"open_issues_count":2,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T05:47:48.571Z","etag":null,"topics":["google-cloud","machine-learning","ml-engine","tensorflow","tensorflow-hub"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yu-iskw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-16T16:39:17.000Z","updated_at":"2019-09-03T10:55:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"3c4170d0-d3c4-41c4-8f0a-d8aa722f3043","html_url":"https://github.com/yu-iskw/tensorflow-hub-with-ml-engine","commit_stats":{"total_commits":2,"total_committers":1,"mean_commits":2.0,"dds":0.0,"last_synced_commit":"2352c338aa62420bcd3cf893efa0aef09bc48aba"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yu-iskw%2Ftensorflow-hub-with-ml-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yu-iskw%2Ftensorflow-hub-with-ml-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yu-iskw%2Ftensorflow-hub-with-ml-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yu-iskw%2Ftensorflow-hub-with-ml-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yu-iskw","download_url":"https://codeload.github.com/yu-iskw/tensorflow-hub-with-ml-engine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243891353,"owners_count":20364576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-cloud","machine-learning","ml-engine","tensorflow","tensorflow-hub"],"created_at":"2024-10-07T23:05:50.599Z","updated_at":"2025-03-17T22:31:10.589Z","avatar_url":"https://github.com/yu-iskw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Retrain flower data with tensorflow-hub\n\nThis is an example code to reproduce an issue of `tensorflow-hub` with Google ML Engine.\nThe reason why I made the repository is to reproduce the issue that `tensorflow-hub` doesn't work on Google ML Engine at the time I am making the repository.\n\n- [How to use TF Hub on a distributed setting? · Issue \\#48 · tensorflow/hub](https://github.com/tensorflow/hub/issues/48)\n- [Tensorflow\\-Hub doesn't work on the runtime version 1\\.7\\. \\[78898344\\] \\- Visible to Public \\- Issue Tracker](https://issuetracker.google.com/issues/78898344)\n\n```\nOP_REQUIRES failed at save_restore_tensor.cc:170 : Invalid argument: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /tmp/tfhub_modules/11d9faf945d073033780fd924b2b09ff42155763/variables/variables: Not found: /tmp/tfhub_modules/11d9faf945d073033780fd924b2b09ff42155763/variables; No such file or directory\n```\n\n## Requirements\n- Anaconda\n- Google Cloud SDK\n\n## Prepare for the python environment\n```\n# Create conda environment\nmake create-conda\n\n# Remove conda environment\nmake remove-conda\n```\n\n## Hot wo run\n\n1. Download flower data.\nThe script downloads the flower data in `data/flower_photos`\n```\nbash ./dev/prepare-dataset.sh\n```\n\n2. Create TFRecord data.\n```\npython crete_tfrecord.py \\\n  --input ./data/flower_photos \\\n  --train_output ./train.tfrecord \\\n  --eval_output ./eval.tfrecord\n```\n\n3. Upload TFRecord data to Google Cloud Storage.\n```\nPROJECT_ID=...\nGCS_BUCKET=...\n\n# Create a bucket if necessary.\ngsutil mb -p \"$PROJECT_ID\" \"gs://${GCS_BUCKET}\"\n\n# Copy TFRecord files.\ngsutil cp -p train.tfrecord \"gs://${GCS_BUCKET}/train.tfrecord\"\ngsutil cp -p eval.tfrecord \"gs://${GCS_BUCKET}/eval.tfrecord\"\n\n# Check the uploaded files.\ngsutil ls \"gs://${GCS_BUCKET}/train.tfrecord\"\ngsutil ls \"gs://${GCS_BUCKET}/eval.tfrecord\"\n```\n\n4. Train a model.\n```\n# Run on your local machine to test.\nGCS_BUCKET=...\nTRAIN_DATA=\"gs://${GCS_BUCKET}/train.tfrecord\"\nEVAL_DATA=\"gs://${GCS_BUCKET}/eval.tfrecord\"\nbash run_local.sh \"${TRAIN_DATA}\" \"${EVAL_DATA}\"\n\n# Run on ML Engine\nPROJECT_ID=...\nGCS_BUCKET=...\nMODEL_DIR=\"gs://${GCS_BUCKET}/model/\"\nTRAIN_DATA=\"gs://${GCS_BUCKET}/train.tfrecord\"\nEVAL_DATA=\"gs://${GCS_BUCKET}/eval.tfrecord\"\nbash run_cloud.sh \\\n  \"$PROJECT_ID\" \\\n  \"$GCS_BUCKET\" \\\n  \"$MODEL_DIR\" \\\n  \"$TRAIN_DATA\" \\\n  \"$EVAL_DATA\"\n\n# Monitor the training\ntensorboard --logdir=\"$MODEL_DIR\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyu-iskw%2Ftensorflow-hub-with-ml-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyu-iskw%2Ftensorflow-hub-with-ml-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyu-iskw%2Ftensorflow-hub-with-ml-engine/lists"}