{"id":16837778,"url":"https://github.com/zhreshold/dl_benchmark","last_synced_at":"2025-03-18T03:42:33.895Z","repository":{"id":67409265,"uuid":"106876708","full_name":"zhreshold/dl_benchmark","owner":"zhreshold","description":null,"archived":false,"fork":false,"pushed_at":"2017-10-11T21:07:42.000Z","size":24,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"develop","last_synced_at":"2025-01-24T10:47:57.603Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhreshold.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-13T21:54:08.000Z","updated_at":"2018-03-02T12:03:30.000Z","dependencies_parsed_at":"2023-02-25T00:16:45.112Z","dependency_job_id":null,"html_url":"https://github.com/zhreshold/dl_benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhreshold%2Fdl_benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhreshold%2Fdl_benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhreshold%2Fdl_benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhreshold%2Fdl_benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhreshold","download_url":"https://codeload.github.com/zhreshold/dl_benchmark/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244153308,"owners_count":20406995,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T12:18:48.100Z","updated_at":"2025-03-18T03:42:33.875Z","avatar_url":"https://github.com/zhreshold.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep Learning Benchmark Script Driver\n\nThis repo holds the benchmark tasks that are executed regularly and also the driver script.\nIt is designed to be simple and flexible. Basically it is a thin wrapper on top of your benchmark script which provides\nbasic utilities for managing tasks, profiling gpu and cpu memories, and extract metrics from your logging.\n\nTo use the benchmark driver, you just need to look at the task_config_template.cfg to see what benchmark tasks are supported\nand use the command `python benchmark_driver.py --task-name [TASK_NAME] --num-gpus [NUM_GPUS]` then the driver will trigger a\ntask and write the benchmark result to a JSON file called \"dlbenchmark_result.json\". e.g.\n\n```\n{\n    'resnet50_cifar10_hybrid.total_training_time': 797.3191379999998,\n    'resnet50_cifar10_hybrid.gpu_memory_usage_max': 749.0,\n    'resnet50_cifar10_hybrid.cpu_memory_usage': 788,\n    'resnet50_cifar10_hybrid.validation_acc': 0.560998,\n    'resnet50_cifar10_hybrid.speed': 1268.8090892,\n    'resnet50_cifar10_hybrid.gpu_memory_usage_std': 66.27262632335068,\n    'resnet50_cifar10_hybrid.training_acc': 0.751256,\n    'resnet50_cifar10_hybrid.gpu_memory_usage_mean': 602.6772235225278\n}\n```\n\nThere are also optional flags such as `--framework`, and `--metrics-suffix`, which are used for decorating the metrics names.\n\nTo add a new task you just need to put your benchmark script or your benchmark script directory under this repo and\nadd a section in the `task_config_template.cfg` like the following example:\n\n```\n [resnet50_cifar10_imperative]\n patterns = ['Speed: (\\d+\\.\\d+|\\d+) samples/sec', 'training: accuracy=(\\d+\\.\\d+|\\d+)', 'validation: accuracy=(\\d+\\.\\d+|\\d+)', 'time cost: (\\d+\\.\\d+|\\d+)']\n metrics = ['speed', 'training_acc', 'validation_acc', 'total_training_time']\n compute_method = ['average', 'last', 'last', 'total']\n command_to_execute = python image_classification/image_classification.py --model resnet50_v1 --dataset cifar10 --gpus 8 --epochs 20 --log-interval 50\n num_gpus = 8\n```\n\n`patterns`, `metrics`, `compute_method` need to be placed in corresponding order so that the metrics map knows the key pair relationship.\nUsers need to defined the logging extraction rule using python's regular expression.\n\nTypical logging looks like the following example:\n\n```\nINFO:root:Epoch[0] Batch [49]\tSpeed: 829.012392 samples/sec\taccuracy=0.172578\nINFO:root:Epoch[0] Batch [99]\tSpeed: 844.157227 samples/sec\taccuracy=0.206563\nINFO:root:Epoch[0] Batch [149]\tSpeed: 835.582445 samples/sec\taccuracy=0.230781\nINFO:root:[Epoch 0] training: accuracy=0.248027\nINFO:root:[Epoch 0] time cost: 69.030488\nINFO:root:[Epoch 0] validation: accuracy=0.296484\nINFO:root:Epoch[1] Batch [49]\tSpeed: 797.687061 samples/sec\taccuracy=0.322344\nINFO:root:Epoch[1] Batch [99]\tSpeed: 821.444257 samples/sec\taccuracy=0.329883\nINFO:root:Epoch[1] Batch [149]\tSpeed: 810.339386 samples/sec\taccuracy=0.342969\nINFO:root:[Epoch 1] training: accuracy=0.351983\nINFO:root:[Epoch 1] time cost: 61.266612\nINFO:root:[Epoch 1] validation: accuracy=0.393930\n```\n\nHere you can see the pattern `Speed: (\\d+\\.\\d+|\\d+) samples/sec` correspond to `Speed: 829.012392 samples/sec`,\n`Speed: 844.157227 samples/sec`, `Speed: 835.582445 samples/sec`, ... So the driver will extract these parts\ninto a list and map a number extractor to this list. So in the end we will get `metric=[829.012392, 844.157227,\n835.582445,...]`. The `compute_method` is `average`, suppose `average([829.012392, 844.157227,\n835.582445,...])=825.25` ,so it will put the pair `speed: 825.25` in the final result file.\n\nThe driver will redirect the logging into a logfile and will remove it after the metrics have been successfully\nextracted.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhreshold%2Fdl_benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhreshold%2Fdl_benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhreshold%2Fdl_benchmark/lists"}