{"id":26451076,"url":"https://github.com/malcolmlett/training-instrumentation-toolkit","last_synced_at":"2025-07-03T14:05:31.753Z","repository":{"id":279937636,"uuid":"940472691","full_name":"malcolmlett/training-instrumentation-toolkit","owner":"malcolmlett","description":"TensorFlow addons for instrumenting NN model training loops in order to collect, analyse, and visualize detailed metrics on gradients, weights, and layer activations","archived":false,"fork":false,"pushed_at":"2025-03-17T03:16:12.000Z","size":107441,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-17T04:26:14.614Z","etag":null,"topics":["instrumentation","neural-network","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/malcolmlett.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-28T08:35:05.000Z","updated_at":"2025-03-17T03:16:15.000Z","dependencies_parsed_at":"2025-02-28T16:39:44.569Z","dependency_job_id":"2dc9a55f-c431-439b-846d-a81fbf16a445","html_url":"https://github.com/malcolmlett/training-instrumentation-toolkit","commit_stats":null,"previous_names":["malcolmlett/training-instrumentation-toolkit"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malcolmlett%2Ftraining-instrumentation-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malcolmlett%2Ftraining-instrumentation-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malcolmlett%2Ftraining-instrumentation-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malcolmlett%2Ftraining-instrumentation-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/malcolmlett","download_url":"https://codeload.github.com/malcolmlett/training-instrumentation-toolkit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244259956,"owners_count":20424648,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["instrumentation","neural-network","tensorflow"],"created_at":"2025-03-18T16:31:35.670Z","updated_at":"2025-07-03T14:05:31.739Z","avatar_url":"https://github.com/malcolmlett.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Training Instrumentation Toolkit\nTensorFlow addons for instrumenting NN model training loops in order to collect, analyse, and visualize detailed\nmetrics on gradients, weights, and layer activations.\n\nIntended to improve development turnaround time by making it easier to access subtle indicators of training progress and\npotential training problems. Aids in troubleshooting by providing insight into many aspects of model training in one go,\nquickly highlighting if any of several common issues are occurring.\n\nExample visualisation of metrics gathered during training:\n\n![training overview plot](doc/training-overview-example.png)\n\nSome of the kinds of problems that this tooling is targeted at include:\n* vanishing and exploding gradients\n* oscillating gradients\n* neuron death\n* and more generally for identifying the cause of slow training progress\n\nThis project came out of experiments looking into options for improving the \"training observability\" - ie: improving our\nability to observe how the model training algorithm is behaving. There's a gap between the techniques used in academic\nliterature to investigate these kinds of problems versus what's easy to do out of the box with the likes of TensorFlow\nand PyTorch. I wanted to see what it would take to bridge that gap. I'm not seeking to replace existing tooling,\nrather to suggest some data collection and visualisation approaches that I think would be good to add to existing\ntooling for TensorFlow and PyTorch. In the meantime, this toolkit is easy to incorporate into your existing\ntraining pipeline.\n\nI have intentionally focused on basic plotting via matplotlib so that I could get away from any limitations inherent\nwithin the existing plotting tools such as that provided by TensorBoard, Weights \u0026 Biases, etc.\n\nAnyone is free to take what's here and use as a basis for their own work, though attribution would be greatly\nappreciated.\n\n## Importing\n\nThe following code can be used to import the package into a Jupiter notebook:\n\n```python\nimport os\nimport sys\nif not os.path.isdir('training-instrumentation-toolkit'):\n  !git clone https://github.com/malcolmlett/training-instrumentation-toolkit.git\nsys.path.append('training-instrumentation-toolkit')\nimport training_instrumentation as tinstr\nimport training_explainer as texpl\n```\n\nThe toolkit is not currently published as a python package.\n\n## Quickstart\n\nThe following runs a model training while capturing the metrics needed to generate the image above.\n\n```python\nimport tensorflow as tf\nimport training_instrumentation as tinstr\n\ndef my_model():\n    ....\n    \ndef my_dataset():\n    ...\n\nvariables = tinstr.VariableHistoryCallback(per_step=True)\ngradients = tinstr.GradientHistoryCallback(per_step=True)\noutputs = tinstr.LayerOutputHistoryCallback(per_step=True)\noutput_gradients = tinstr.LayerOutputGradientHistoryCallback(per_step=True)\n\nmodel = my_model()\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy', 'mse', 'binary_crossentropy'])\ndataset = my_dataset()\nhistory = tinstr.fit(model, dataset.batch(32), epochs=10, callbacks=[\n    variables, gradients, outputs, output_gradients, tinstr.HistoryStats(per_step=True)])\n\ntinstr.plot_history_overview([history, variables, gradients, outputs, output_gradients])\n```\n\n## Docs\n\nThe following Jupiter notebooks demonstrate the core functionality of the _training-instrumentation-toolkit_,\nincluding how you can easily use it as a basis for collecting your metrics. They include links to open the\nnotebooks directly within Google Colab so you can run them yourself.\n\n1. [Instrumentation](doc/instrumenting.ipynb)\n2. [Visualisations](doc/visualisations.ipynb)\n3. [Train Explainer](doc/train-explainer.ipynb)\n\nDetailed pydocs are available within the source code.\n\nThe above documentation notebooks are all found in the `doc` folder. If you navigate there you will also find other\nexperiments, including a series of notebooks that feed into blog posts.\n\n**Blog Posts**\n\nThis toolkit is extensively used in a series of blog posts on improving instrumentation and visualisation of model dynamics during training:\n* [Better ways to monitor NNs while training](https://medium.com/ai-advances/better-ways-to-monitor-nns-while-training-7c246867ca4f).\n* [Neuron Death in ANNs](https://medium.com/ai-advances/neuron-death-in-anns-detecting-and-troubleshooting-4a7b5cc2f099)\n* [Vanishing and Exploding Gradients](https://medium.com/ai-advances/vanishing-and-exploding-gradients-5dcdf905c550)\n\n## License\nSee the [LICENSE](./LICENSE) file for details about the license under which this code is made available.\n\n## Citation\nIf you find this repository useful in your research, please consider giving a star :star: and a citation.\nFor citation information, please see the \"Cite this repository\" button on the right.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmalcolmlett%2Ftraining-instrumentation-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmalcolmlett%2Ftraining-instrumentation-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmalcolmlett%2Ftraining-instrumentation-toolkit/lists"}