{"id":15908700,"url":"https://github.com/graykode/mnist-flow","last_synced_at":"2025-03-22T03:31:36.891Z","repository":{"id":39736316,"uuid":"204306786","full_name":"graykode/mnist-flow","owner":"graykode","description":"This Project is only repository for solving AI Engineer Party","archived":false,"fork":false,"pushed_at":"2022-11-21T21:37:40.000Z","size":28569,"stargazers_count":3,"open_issues_count":3,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T07:43:13.440Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/graykode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-25T14:38:09.000Z","updated_at":"2020-11-14T05:52:02.000Z","dependencies_parsed_at":"2023-01-22T10:00:29.943Z","dependency_job_id":null,"html_url":"https://github.com/graykode/mnist-flow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fmnist-flow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fmnist-flow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fmnist-flow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graykode%2Fmnist-flow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/graykode","download_url":"https://codeload.github.com/graykode/mnist-flow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244902929,"owners_count":20529114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-06T14:23:30.012Z","updated_at":"2025-03-22T03:31:31.883Z","avatar_url":"https://github.com/graykode.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mnist-flow\n\nThis Project is only repository for solving [AI Engineer Party](https://www.facebook.com/groups/TensorFlowKR/permalink/971390023202056/). DataSet is MNIST to reduce GCE cost.\n\n\n\n## Architecture\n\n![](architecture.png)\n\n\n\n## Quick Start using GCE Configuration\n\n1. Set your GCE region, Please use this command on Python2.7, beacase of gcloud cli. To use gcloud, it had been connected your GCE Account, see `gcloud info`.\n\n   ```shell\n    $ gcloud config set compute/zone asia-northeast1\n   ```\n\n2. Install Cloud Dataflow Python SDK\n\n   ```shell\n   $ pip install --upgrade google-cloud-dataflow --user\n   ```\n   \n4. Create IAM which have roles in BigQuery and Google Storage View.\n\n   ```shell\n   $ export GOOGLE_APPLICATION_CREDENTIALS=\"key.json\"\n   ```\n   \n4. Preprocess `tf.keras.datasets.mnist` to be flatten in [script/make_data.py](script/make_data.py) and upload to Google Bigquery as splitting train/test dataset.\n\n   ```shell\n   $ python script/make_data.py\n   $ gzip data/*.txt\n   $ bq load --source_format=CSV -F\":\" mnist.train data/train.txt.gz \\\n       \"key:integer,image:string,label:integer\"\n   $ bq load --source_format=CSV -F\":\" mnist.test data/test.txt.gz \\\n       \"key:integer,image:string,label:integer\"\n   ```\n\n5. Install BigQuery Client Libraries if you want to run on real time. **Be careful call one query making below traffic.  Google BigQuery price is $0.035 per GB**. So we should be cached dataset.\n\n   - 60000 MNIST train Dataset : 151.4MB\n   - 10000 MNIST test Dataset : 25.3MB\n\n   ```python\n   from google.cloud import bigquery\n   client = bigquery.Client()\n   query = (\"SELECT image, label FROM mnist.train\")\n   query_job = client.query(\n       query,\n   )  # API request - starts the query\n   for row in query_job:  # API request - fetches results\n       print(row)\n   ```\n\n6. To Training Adanet, run [train_adanet.py](train_adanet.py])\n\n   ```shell\n   $ python train_adanet.py \\\n   \t\t\t--RANDOM_SEED 42 \\\n   \t\t\t--NUM_CLASSES 10 \\\n   \t\t\t--TRAIN_STEPS 5000 \\\n   \t\t\t--BATCH_SIZE 64 \\\n   \t\t\t--LEARNING_RATE 0.001 \\\n   \t\t\t--FEATURES_KEY \"images\" \\\n   \t\t\t--ADANET_ITERATIONS 2\\\n   \t\t\t--MODEL_DIR \"./models\" \\\n   \t\t\t--SAVED_DIR \"./saved_model\" \\\n   \t\t\t--logfile \"log.txt\"\n   ```\n\n   See saved model directory structure below:\n\n   ```\n   models\n   └── 1566790420\n       ├── saved_model.pb\n       └── variables\n           ├── variables.data-00000-of-00001\n           └── variables.index\n   ```\n\n7. Create Google Storage Bucket to upload saved model folder\n\n   ```shell\n   $ PROJECT_ID=$(gcloud config list project --format \"value(core.project)\")\n   $ BUCKET=\"${PROJECT_ID}-ml\"\n   \n   # create bucket\n   $ gsutil mb -c regional -l asia-northeast1 gs://${BUCKET}\n   \n   # upload saved model\n   $ gsutil -m cp -R ./saved_model gs://${BUCKET}\n   ```\n\n8. Deploy google function\n\n   ```shell\n   $ cd gfunction\n   $ gcloud functions deploy handler --runtime python37 \\\n   \t\t\t\t--trigger-http --memory 2048 --region asia-northeast1\n   ```\n\n9. Test API using `curl`\n\n   ```shell\n   $ curl -F 'file=@testdata/0.png' 'your api handler server'\n   output : 0\n   ```\n\n#### pylint\n\n```shell\n$ pip install pylint\n$ pylint **/*.py\n```\n\nBut I use pep8 supported on Pycharm.\n\n\n\n## Author\n\n- Tae Hwan Jung(Jeff Jung) @graykode, Kyung Hee Univ CE(Undergraduate).\n- Author Email : [nlkey2022@gmail.com](mailto:nlkey2022@gmail.com)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraykode%2Fmnist-flow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgraykode%2Fmnist-flow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraykode%2Fmnist-flow/lists"}