{"id":37065046,"url":"https://github.com/levnikmyskin/that_metric_timeline","last_synced_at":"2026-01-14T07:36:35.956Z","repository":{"id":39976687,"uuid":"507277225","full_name":"levnikmyskin/that_metric_timeline","owner":"levnikmyskin","description":"That Metric Timeline (TMT) is a Python library aimed at the machine/deep learning practitioner/researcher. It helps tracking experiments, saving code, metrics and results","archived":false,"fork":false,"pushed_at":"2023-02-13T10:07:09.000Z","size":569,"stargazers_count":21,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-05T19:56:52.964Z","etag":null,"topics":["deep-learning","machine-learning","python","utility-library"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/levnikmyskin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-25T10:13:13.000Z","updated_at":"2024-10-16T11:02:31.000Z","dependencies_parsed_at":"2022-09-23T12:42:31.014Z","dependency_job_id":null,"html_url":"https://github.com/levnikmyskin/that_metric_timeline","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/levnikmyskin/that_metric_timeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/levnikmyskin%2Fthat_metric_timeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/levnikmyskin%2Fthat_metric_timeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/levnikmyskin%2Fthat_metric_timeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/levnikmyskin%2Fthat_metric_timeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/levnikmyskin","download_url":"https://codeload.github.com/levnikmyskin/that_metric_timeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/levnikmyskin%2Fthat_metric_timeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413441,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","machine-learning","python","utility-library"],"created_at":"2026-01-14T07:36:35.302Z","updated_at":"2026-01-14T07:36:35.942Z","avatar_url":"https://github.com/levnikmyskin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# That Metric Timeline (TMT) ⌚\n[![Documentation Status](https://readthedocs.org/projects/thatmetrictimeline/badge/?version=latest)](https://thatmetrictimeline.readthedocs.io/en/latest/?badge=latest) [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) ![GitHub release (latest by date)](https://img.shields.io/github/v/release/levnikmyskin/that_metric_timeline) [![DOI](https://zenodo.org/badge/507277225.svg)](https://zenodo.org/badge/latestdoi/507277225)\n\nThat Metric Timeline (TMT) is a Python library aimed at the machine/deep learning practitioner/researcher. This library aims to:  \n  * Help the user to keep track of experiments, their metrics, their results and the code used to produce them;  \n  * Provide an easy-to-use and [KISS](https://en.wikipedia.org/wiki/KISS_principle) based library to the user;  \n  * Do the bare-minimum (i.e., no fancy web interfaces, plotting and so on), but do it (hopefully) right;  \n  * Give an open-source experiment tracking library to the community which is free both as in \"free beer\" and \"free speech\".  \n  \nSo many times when working on a research project I've seen the number of experiments skyrocketing in the blink of an eye...and then when we come back to our project after \"some time off\", we may have lost track of where the results are, what was the code that produced them and so on.  \nThis library can hopefully be a solution to all of this :)\n\nRead the full documentation on [ReadTheDocs](https://thatmetrictimeline.readthedocs.io/en/latest/).\n\n## Installation\nThe library is available on PyPI, and can be installed with\n```\npip install ThatMetricTimeline\n```\nAfter a successful installation, a binary `tmt_tui` will be available in your path. This is the library terminal user interface (TUI). More on this [later](#tui).\n\n## Usage\n`tmt` can keep track of your experiments. Every experiment will be saved as an entry in a json database, with results and code snapshot backups[^1] saved in different folders. By default, `tmt` will:  \n * Create a `.tmt` directory in your current working directory;  \n * Create a `.tmt/tmt_db.json` file which will be used as a database;  \n * Create a `.tmt/snapshots` directory, where code snapshot backups will be saved. A symlink `.tmt/snapshots/last` will also be created and will always point to the last snapshot taken. See the [Snapshots](#snapshots) section for more details.\n\nShould you want to change where all of this is saved, check the [Custom configuration](#custom-configuration) section.  \nCode examples are provided in the `examples` folder in this repository. More examples will be added in the future.\n\n[^1]: Code backup is taken as a \"snapshot\". It means that the first time it will be a standard copy of all your files (you can provide a `.gitignore` file to ignore files). The following backups will only copy new and changed files, while everything else is copied as a hard-link (therefore not occupying space on your disk).\n\n### Tracking experiments\nWhile more optional features will probably come in the future, the library goal is to be simple, both for the user and for the library developer.  \nThe main function exposed by `tmt` is actually the `tmt_recorder` decorator. This is what we use to actually store and keep track of experiments.  \n\nThe decorator takes a `name` parameter (and a few optional more). The experiment will be saved and later searched with this name.  \nThe decorated function might return a dictionary with the metrics the user wishes to save for later retrieval. If you don't want to save any metric, the function must return `None` or an empty object (e.g. `{}`).\n```python\nfrom tmt import tmt_recorder\n\n@tmt_recorder(name=\"some_experiment\")\ndef train_and_predict(x_tr, y_tr, x_te, y_te):\n    lr = LogisticRegression()\n    lr.fit(x_tr, y_tr)\n    preds = lr.predict(x_te)\n    return {'f1': f1_score(y_te, preds), 'accuracy': accuracy_score(y_te, preds)}\n```\nThe other key function `tmt` exposes is `tmt_save`. This function should be called by the user to save any kind of pickable object, at any time.  \nIf we wanted to save the predictions in the example above, we would do:\n```python\nfrom tmt import tmt_recorder, tmt_save\n\n@tmt_recorder(name=\"some_experiment_with_data\")\ndef train_and_predict(...):\n    ...\n    preds = lr.predict(x_te)\n    tmt_save(preds, name='lr_predictions')\n    return {'f1': f1_score(y_te, preds), 'accuracy': accuracy_score(y_te, preds)}\n```\nAs you can see, we give a name to the saved object as well. This should make it easier to recognize what this pickled object refers to.\n\n## TUI\n\n### Searching and looking at experiments\n`tmt` offers a terminal user interface (TUI) which should be installed in your path when you `pip install` the library.\nYou can access the TUI by typing:\n\n```\ntmt_tui\n```\n\nIf you're using a custom configuration (see [Custom configuration](#custom-configuration)), specify it like this:\n\n```\ntmt_tui -c /path/to/your/config.json\n```\n\nYou will be presented with the following old-fashioned interface (who doesn't love the 90s?):\n![main_tui](.github/assets/main_tui.png)\n\nYou can move around with the arrow (or the tab and shift+tab) keys. You can then search by name (supports regex!) in this interface:\n![search_tui](.github/assets/search_tui.png)\n\nOnce you select an experiment you can see some details about it:\n![experiment_tui](.github/assets/experiment_tui.png)\n\nThe __Search experiment by date__ functionality is not implemented yet and will come in a future release. You can however use the `TmtManager` (see the next section).\n\n## Loading and using tracked experiments in your code\n`tmt` offers a minimalistic `TmtManager` helper class, which can help you load an experiment in your code, load pickled results, see metrics etc.  \nOnce you have the ID (or a unique name for your experiment) you can:\n```python\nfrom tmt import TmtManager\n\n# Let's say we know there is an experiment with id \"example\"\n\n\n# An Entry is a row in the database, i.e. an experiment that was tracked.\nmanager = TmtManager()\nmanager.set_entry_by_id('example') \n\n# load the results and unpickle them\nfor name, path in manager.results_paths():\n    with open(path, 'rb') as f:\n        # do stuff with your results. If it's a pickle it's \n        # more convenient to use the code block below this one\n        res = pickle.load(f)\n\n# load the unpickled results\nfor name, res in manager.load_results():\n    # do something with your results.\n    # if res is a numpy array...\n    print(res.mean())\n\n\nfor name, val in manager.get_metrics():\n    print(f\"{name}: {val}\")\n```\n\nShould you need it, you can access the \"low level\" database manager from the `manager.db` member.\n```python\n# If you need to do other stuff, like searching for \n# experiments between two datetimes and so on\n# you can access the `db` member like\nmanager.db.get_entries_greater_than_date(date_or_timestamp)\n\n# You can also search names with a regex\nmanager.db.get_entries_by_name_regex(r'experiment\\d+')\n```\n\n## Snapshots\nEvery time you track an experiment with `tmt_recorder`, a code snapshot backup will be saved (by default in `.tmt/snapshots`). This means that:  \n * the first time you use the library in your project, a simple copy of your project is made (by default, this is the current working directory (_cwd_) from which you launch the experiment);  \n * subsequent backups will only copy new and different files, while hard-linking all other files. This limits the space taken on your disk;  \n * by default, the library will look for a `.gitignore` file in your _cwd_ and ignore (i.e., not copy) all files listed in there (the [PathSpec](https://python-path-specification.readthedocs.io/en/latest/readme.html) library is used for gitignore parsing;\n * a symlink pointing to the last snapshot taken is created (and updated everytime) in `.tmt/snapshots/last`.  \n\nYou can change the default paths by using a [Custom configuration](#custom-configuration) file.\n\n## Custom configuration\n`tmt` can be used as-is and does not require any configuration file. By default, everything the library needs or save is stored in a `.tmt` hidden directory, in the current working directory(_cwd_). If your _cwd_ changes often for different experiments, or if you want to specify which folder is backed up and more, you may want to create and specify a custom configuration file.  \n  \nTo do so, create a `config.json` file: if you're fine with using the `cwd/.tmt` directory, place this file in `cwd/.tmt/config.json`. This way, you won't have to specify the path to this configuration file to library related functions.  \nThe configuration file has the following structure\n```json\n{\n    // tmt_dir specifies the path where code snapshots and \n    // results will be saved. You may use an absolute \n    // path as well\n    \"tmt_dir\": \".example\",\n\n    // this is the folder we will take a snapshot of \n    // for every experiment  \n    \"snapshot_source\": \".\", \n\n    // snapshot_target is where code snapshots will be \n    // saved. It will be joined with tmt_dir. So in this \n    // case the target will be .examples/snapshot_example\n    \"snapshot_target\": \"snapshot_example\",\n\n    // this path will be a symlink to the last snapshot\n    // taken. Same rules as for snapshot_target apply\n    \"last_snapshot_link\": \"snapshot_example/last\",\n\n    // this might actually be any file with a .gitignore \n    // syntax. These files will be ignored and not backupped\n    \"gitignore_path\": \"path/to/.gitignore\",\n\n    // the two paths below are for the db and the results\n    // directory, respectively. Same rules apply as for \n    // snapshot_target, so path will be .example/tmt_db.json\n    \"json_db_path\": \"tmt_db.json\",\n    \"results_path\": \"results\"\n}\n```\nAs mentioned, if you save this file in `.tmt/config.json`, no other action is necessary and `tmt` will pick it up and use it for its configuration.  \nIf instead you save it somewhere else, say `/config/path/config.json`, you will have to specify this path in the code. When recording experiments:\n```python\n@tmt_recorder('custom_config', config_path='/config/path/config.json')\ndef with_custom_config():\n    x, y = make_classification()\n    lr = LogisticRegression()\n    lr.fit(x, y)\n    preds = lr.predict(x)\n    return {'f1': f1_score(y, preds), 'accuracy': accuracy_score(y, preds)}\n```\nAnd when managing experiments:\n```python\nmanager = TmtManager(config='/config/path/config.json')\n# do your stuff\n```\n\n## Final remarks\nThis project was mainly developed because I wanted to have a simple library to keep track of the machine learning experiments I run for my papers in my PhD (and also, because I had fun :D).  \nI'm well aware that there are other options at the moment such as [Weights\u0026Biases](https://wandb.ai/site) and [ModelChimp](https://github.com/ModelChimp/modelchimp), but I wanted something that was as straightforward and as simple as possible.   \nThe aim of this project should be to keep a [KISS](https://en.wikipedia.org/wiki/KISS_principle) approach, making it \neasier for other researchers to tinker with the library code or to make adjustments.  \nFor this reason, there is not much automation provided, and, for instance, navigating snapshots is left to the user: when you want to quickly check the code for many experiments this \nmight be a pain and I indeed plan to add a better and more automated way to do that.  \nThat said, most of the other library functionalities will remain as they are now.\n\nIf you have any suggestions, feedbacks or problems, feel free to open an issue or a pull request here on Github.  \nCheers!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flevnikmyskin%2Fthat_metric_timeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flevnikmyskin%2Fthat_metric_timeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flevnikmyskin%2Fthat_metric_timeline/lists"}