{"id":26702357,"url":"https://github.com/alexandresajus/experiment-tracking-benchmark","last_synced_at":"2025-10-18T01:03:35.313Z","repository":{"id":62828141,"uuid":"559027040","full_name":"AlexandreSajus/Experiment-Tracking-Benchmark","owner":"AlexandreSajus","description":"A benchmark of multiple experiment tracking tools (Tensorboard, Weights and Biases, Neptune and Comet)","archived":false,"fork":false,"pushed_at":"2022-11-11T22:23:13.000Z","size":4635,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2023-02-28T09:13:19.586Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AlexandreSajus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-28T21:23:11.000Z","updated_at":"2022-11-15T15:30:34.000Z","dependencies_parsed_at":"2023-01-22T00:16:45.902Z","dependency_job_id":null,"html_url":"https://github.com/AlexandreSajus/Experiment-Tracking-Benchmark","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlexandreSajus%2FExperiment-Tracking-Benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlexandreSajus%2FExperiment-Tracking-Benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlexandreSajus%2FExperiment-Tracking-Benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlexandreSajus%2FExperiment-Tracking-Benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AlexandreSajus","download_url":"https://codeload.github.com/AlexandreSajus/Experiment-Tracking-Benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245770531,"owners_count":20669391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-27T02:37:33.550Z","updated_at":"2025-10-18T01:03:30.274Z","avatar_url":"https://github.com/AlexandreSajus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **Experiment Tracking Benchmark**\n\n![Unit Coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/AlexandreSajus/8971164299ef4415d92fbb6b9fc81744/raw/trackbench_unit_coverage_badge.json)\n![Pylint](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/AlexandreSajus/e3303bee97d9b5ec257184aa34125e43/raw/trackbench_pylint_badge.json)\n\nThis is a package to test the performance of different experiment tracking tools: `Tensorboard`, `Weights and Biases`, `Neptune`, `Comet` and `ClearML`.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/tools.png\" alt=\"Tools\" width=\"60%\"/\u003e\n\u003c/p\u003e\n\n**Table of Contents**\n- [**Experiment Tracking Benchmark**](#experiment-tracking-benchmark)\n  - [**Usage**](#usage)\n  - [**Comparison**](#comparison)\n    - [**Tensorboard**](#tensorboard)\n    - [**Weights and Biases**](#weights-and-biases)\n\n## **Usage**\n\nMake sure you have Python installed:\n\n**1. Install the package with pip in your terminal:**\n\n```\npip install git+https://github.com/AlexandreSajus/Experiment-Tracking-Benchmark.git\n```\n\n**2. Run the benchmark with the following command:**\n\n```\npython -m trackbench \u003ctracking library\u003e \u003ctraining steps\u003e\n```\n\nCurrently the supported experiments are: `tensorboard`\n\nThis will train DQN and PPO on CartPole and output the experiment tracking website in the terminal.\n\nFor example, for tensorboard, the output will be:\n\n```\nTraining... (this might take some time)\nTraining DQN on CartPole...\nTraining PPO on CartPole...\nTensorBoard launched at http://localhost:6006/\nPress Ctrl+C to stop TensorBoard\n```\n\nYou will then be able to access the experiment tracking website at http://localhost:6006/ which will show result curves.\n\n## **Comparison**\n\n### **Tensorboard**\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"assets/tensorboard_logo.png\" alt=\"Tensorboard Logo\" width=\"6%\"/\u003e\n\u003c/p\u003e\n\nTensorboard creates a local webpage with curves of the training process. It is supported on many platforms and is really easy to start with but it is not very customizable.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/tensorboard_rew.png\" alt=\"Tensorboard Curves\" width=\"80%\"/\u003e\n\u003c/p\u003e\n\n:white_check_mark: Advantages:\n- Easy to start\n- Supported on many platforms\n\n:x: Disadvantages:\n- Difficult to customize\n- Limited\n- Basic interface\n\n### **Weights and Biases**\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"assets/wandb_logo.png\" alt=\"Wandb Logo\" width=\"6%\"/\u003e\n\u003c/p\u003e\n\n\nWeights and Biases records a lot of data about the training process and creates a webpage to visualize it\n\nThis allows the creation of customizable dashboards to visualize and analyze the training process.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/wandb_dash.png\" alt=\"Wandb Dashboard\" width=\"100%\"/\u003e\n\u003c/p\u003e\n\nBut also hosted webpages with reports that you can share with colleagues\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/wandb_report.gif\" alt=\"Wandb Report\" width=\"100%\"/\u003e\n\u003c/p\u003e\n\n:white_check_mark: Advantages:\n- Many features (dashboards, reports, video, audio, ...)\n- Very customizable\n- Very ergonomic interface\n- Supported everywhere with detailed documentation\n\n:x: Disadvantages:\n- Requires an account and an internet connection\n- Paid when working as a team\n- Setup is more complicated, takes time to learn","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexandresajus%2Fexperiment-tracking-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexandresajus%2Fexperiment-tracking-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexandresajus%2Fexperiment-tracking-benchmark/lists"}