{"id":15399273,"url":"https://github.com/raviqqe/tensorflow-qnd","last_synced_at":"2025-04-15T11:52:46.356Z","repository":{"id":57474157,"uuid":"76908814","full_name":"raviqqe/tensorflow-qnd","owner":"raviqqe","description":"Quick and Dirty TensorFlow command framework to train and evaluate models and make inference","archived":false,"fork":false,"pushed_at":"2019-04-19T01:40:03.000Z","size":268,"stargazers_count":56,"open_issues_count":5,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-28T19:44:58.593Z","etag":null,"topics":["machine-learning","python","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raviqqe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-12-20T00:36:39.000Z","updated_at":"2024-01-04T16:09:54.000Z","dependencies_parsed_at":"2022-09-12T21:01:31.998Z","dependency_job_id":null,"html_url":"https://github.com/raviqqe/tensorflow-qnd","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raviqqe%2Ftensorflow-qnd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raviqqe%2Ftensorflow-qnd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raviqqe%2Ftensorflow-qnd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raviqqe%2Ftensorflow-qnd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raviqqe","download_url":"https://codeload.github.com/raviqqe/tensorflow-qnd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249067756,"owners_count":21207395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","python","tensorflow"],"created_at":"2024-10-01T15:47:57.189Z","updated_at":"2025-04-15T11:52:46.338Z","avatar_url":"https://github.com/raviqqe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"img/logo.png\"\u003e\n\u003c/div\u003e\n\n# tensorflow-qnd\n\n[![PyPI version](https://badge.fury.io/py/tensorflow-qnd.svg)](https://badge.fury.io/py/tensorflow-qnd)\n[![Python versions](https://img.shields.io/pypi/pyversions/tensorflow-qnd.svg)](setup.py)\n[![Build Status](https://travis-ci.org/raviqqe/tensorflow-qnd.svg?branch=master)](https://travis-ci.org/raviqqe/tensorflow-qnd)\n[![License](https://img.shields.io/badge/license-unlicense-lightgray.svg)](https://unlicense.org)\n\nQuick and Dirty TensorFlow command framework\n\ntensorflow-qnd is a TensorFlow framework to create commands to train and\nevaluate models and make inference with them.\nThe framework is built on top of\n[tf.contrib.learn module](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/learn/python/learn).\nEspecially if you are working on research projects using TensorFlow, you can\nremove most of boilerplate code with the framework.\nAll you need to do is to define a model constructor `model_fn` and input\nproducer(s) `input_fn` to feed a dataset to the model.\n\n## Features\n\n-   Command creation for:\n    -   Training and evaluation of models\n    -   Inference of labels or regression values with trained models\n-   Configuration of command line options to set hyperparameters of models etc.\n-   [Distributed TensorFlow](https://www.tensorflow.org/how_tos/distributed/)\n    -   Just set an optional argument `distributed` of `def_train_and_evaluate()`\n        as `True` (i.e. `def_train_and_evaluate(distributed=True)`) to enable it.\n    -   Supports only data parallel training\n    -   Only for training but not for inference\n\n## Installation\n\nPython 3.5+ and TensorFlow 1.1+ are required.\n\n```\npip3 install --user --upgrade tensorflow-qnd\n```\n\n## Usage\n\n1.  Add command line arguments with `add_flag` and `add_required_flag` functions.\n2.  Define a `train_and_evaluate` or `infer` function with\n    `def_train_and_evaluate` or `def_infer` function\n3.  Pass `model_fn` (model constructor) and `input_fn` (input producer) functions\n    to the defined function.\n4.  Run the script with appropriate command line arguments.\n\nFor more information, see [documentation](https://raviqqe.github.io/tensorflow-qnd/qnd).\n\n## Examples\n\n`train.py` (command script):\n\n```python\nimport logging\nimport os\n\nimport qnd\n\nimport mnist\n\n\ntrain_and_evaluate = qnd.def_train_and_evaluate(\n    distributed=(\"distributed\" in os.environ))\n\n\nmodel = mnist.def_model()\n\n\ndef main():\n    logging.getLogger().setLevel(logging.INFO)\n    train_and_evaluate(model, mnist.read_file)\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\n`mnist.py` (module):\n\n```python\nimport qnd\nimport tensorflow as tf\n\n\ndef _preprocess_image(image):\n    return tf.to_float(image) / 255 - 0.5\n\n\ndef read_file(filename_queue):\n    _, serialized = tf.TFRecordReader().read(filename_queue)\n\n    def scalar_feature(dtype): return tf.FixedLenFeature([], dtype)\n\n    features = tf.parse_single_example(serialized, {\n        \"image_raw\": scalar_feature(tf.string),\n        \"label\": scalar_feature(tf.int64),\n    })\n\n    image = tf.decode_raw(features[\"image_raw\"], tf.uint8)\n    image.set_shape([28**2])\n\n    return _preprocess_image(image), features[\"label\"]\n\n\ndef serving_input_fn():\n    features = {\n        'image': _preprocess_image(tf.placeholder(tf.uint8, [None, 28**2])),\n    }\n\n    return tf.contrib.learn.InputFnOps(features, None, features)\n\n\ndef minimize(loss):\n    return tf.train.AdamOptimizer().minimize(\n        loss,\n        tf.contrib.framework.get_global_step())\n\n\ndef def_model():\n    qnd.add_flag(\"hidden_layer_size\", type=int, default=64,\n                 help=\"Hidden layer size\")\n\n    def model(image, number=None, mode=None):\n        h = tf.contrib.layers.fully_connected(image,\n                                              qnd.FLAGS.hidden_layer_size)\n        h = tf.contrib.layers.fully_connected(h, 10, activation_fn=None)\n\n        predictions = tf.argmax(h, axis=1)\n\n        if mode == tf.contrib.learn.ModeKeys.INFER:\n            return predictions\n\n        loss = tf.reduce_mean(\n            tf.nn.sparse_softmax_cross_entropy_with_logits(labels=number,\n                                                           logits=h))\n\n        return predictions, loss, minimize(loss), {\n            \"accuracy\": tf.contrib.metrics.streaming_accuracy(predictions,\n                                                              number)[1],\n        }\n\n    return model\n```\n\nWith the code above, you can create a command with the following interface.\n\n```\nusage: train.py [-h] [--output_dir OUTPUT_DIR] [--train_steps TRAIN_STEPS]\n                [--eval_steps EVAL_STEPS]\n                [--min_eval_frequency MIN_EVAL_FREQUENCY]\n                [--num_cores NUM_CORES] [--log_device_placement]\n                [--save_summary_steps SAVE_SUMMARY_STEPS]\n                [--save_checkpoints_steps SAVE_CHECKPOINTS_STEPS]\n                [--keep_checkpoint_max KEEP_CHECKPOINT_MAX]\n                [--batch_size BATCH_SIZE]\n                [--batch_queue_capacity BATCH_QUEUE_CAPACITY]\n                [--num_batch_threads NUM_BATCH_THREADS] --train_file\n                TRAIN_FILE [--filename_queue_capacity FILENAME_QUEUE_CAPACITY]\n                --eval_file EVAL_FILE [--hidden_layer_size HIDDEN_LAYER_SIZE]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --output_dir OUTPUT_DIR\n                        Directory where checkpoint and event files are stored\n                        (default: output)\n  --train_steps TRAIN_STEPS\n                        Maximum number of train steps (default: None)\n  --eval_steps EVAL_STEPS\n                        Maximum number of eval steps (default: 100)\n  --min_eval_frequency MIN_EVAL_FREQUENCY\n                        Minimum evaluation frequency in number of train steps\n                        (default: 1)\n  --num_cores NUM_CORES\n                        Number of CPU cores used. 0 means use of a default\n                        value. (default: 0)\n  --log_device_placement\n                        If specified, log device placement information\n                        (default: False)\n  --save_summary_steps SAVE_SUMMARY_STEPS\n                        Number of steps every time of which summary is saved\n                        (default: 100)\n  --save_checkpoints_steps SAVE_CHECKPOINTS_STEPS\n                        Number of steps every time of which a model is saved\n                        (default: None)\n  --keep_checkpoint_max KEEP_CHECKPOINT_MAX\n                        Max number of kept checkpoint files (default: 86058)\n  --batch_size BATCH_SIZE\n                        Mini-batch size (default: 64)\n  --batch_queue_capacity BATCH_QUEUE_CAPACITY\n                        Batch queue capacity (default: 1024)\n  --num_batch_threads NUM_BATCH_THREADS\n                        Number of threads used to create batches (default: 2)\n  --train_file TRAIN_FILE\n                        File path of train data file(s). A glob is available.\n                        (e.g. train/*.tfrecords) (default: None)\n  --filename_queue_capacity FILENAME_QUEUE_CAPACITY\n                        Capacity of filename queues of train, eval and infer\n                        data (default: 32)\n  --eval_file EVAL_FILE\n                        File path of eval data file(s). A glob is available.\n                        (e.g. eval/*.tfrecords) (default: None)\n  --hidden_layer_size HIDDEN_LAYER_SIZE\n                        Hidden layer size (default: 64)\n```\n\nExplore [examples](examples) directory for more information and see how to run\nthem.\n\n\n## Caveats\n\n### Necessary update of a global step variable\n\nAs done in [examples](examples), you must get a global step variable with\n`tf.contrib.framework.get_global_step()` and update (increment) it in each\ntraining step.\n\n### Use streaming metrics for `eval_metric_ops`\n\nWhen non-streaming metrics such as `tf.contrib.metrics.accuracy` are used in a\nreturn value `eval_metric_ops` of your `model_fn` or as arguments of\n`ModelFnOps`, their values will be ones of the last batch in every evaluation\nstep.\n\n## Contributing\n\nPlease send issues about any bugs, feature requests or questions, or pull\nrequests.\n\n## License\n\n[The Unlicense](https://unlicense.org)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraviqqe%2Ftensorflow-qnd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraviqqe%2Ftensorflow-qnd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraviqqe%2Ftensorflow-qnd/lists"}