{"id":18470487,"url":"https://github.com/ds2-lab/docker-registry-trace-replayer","last_synced_at":"2025-05-12T03:11:48.513Z","repository":{"id":132149664,"uuid":"212207334","full_name":"ds2-lab/docker-registry-trace-replayer","owner":"ds2-lab","description":null,"archived":false,"fork":false,"pushed_at":"2019-10-01T21:58:50.000Z","size":23,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-16T18:42:58.435Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ds2-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-01T21:58:31.000Z","updated_at":"2021-01-15T19:05:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"d2d9d32e-d648-43de-8303-235621350a04","html_url":"https://github.com/ds2-lab/docker-registry-trace-replayer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Fdocker-registry-trace-replayer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Fdocker-registry-trace-replayer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Fdocker-registry-trace-replayer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2Fdocker-registry-trace-replayer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ds2-lab","download_url":"https://codeload.github.com/ds2-lab/docker-registry-trace-replayer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253176418,"owners_count":21866141,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T10:14:07.457Z","updated_at":"2025-05-09T01:39:46.857Z","avatar_url":"https://github.com/ds2-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Docker Registry Trace Player\n\nThe Docker registry trace player is used to replay anonimized production level traces for a registry, available in the traces directory.\nThe traces are from the IBM docker registry, separated by availability zones.\nThe trace player can send traces to an actual registry or be used to simulate different caching or prefetching. The registry must be confiugred as a test registry in the current version, however.\n\nThe traces are available at http://research.cs.vt.edu/dssl/drtp-head.html. The size of the traces is 22 GB when unzipped.\n\nThe trace player has two applications: the master and the client. The master is reponsible for reading a trace, generating layers, and distributing the layer requests amoung client applications to forward to a registry. The client application is only needed for the run mode.\n\nThe trace player has 3 modes:\n\n* warmup\n* run\n* simulate\n\nThe warmup mode connects to a registry and pushes all the layers in the traces specified by the configuration file. Layers are generated as a string of 0's equal to the size of the request reported in the trace. Because the traces are anonimized, manifests are are treated as layers for the warmup and run modes. During warmup, layers will be pushed to the first registry in the configuration file.\n\nThe Run mode replays the anonymized traces to the registry or registries specified in the configuration file. One or more clients need to be up before the master node is started.\n\nThe simulate mode takes a python file from the configuration file and attempts to call an init function from it. If arguments are specified in the configuration file, then init will be called with two arguments: a sorted list of the requests, and a dictionary of the arguments. If arguments are not specified, then init will be called with just the list of requests.\n\n# Usage\n\nTo Run the master:\n\npython master.py -c \u003ccommand\u003e -i \u003cconfig.yaml\u003e\n\nPossible commands: warmup, run, and simulate\n\nTo run client:\n\npython client.py -i 0.0.0.0 -p \u003cport number\u003e\n\n## Configuration file Options\n\nThe configuration should be a yaml file, refer to config-example.yaml as an example. The following are all the supported options for the configuration file. Options not needed for a given mode will be ignored.\n\n### Options:\n\n#### Cient_info\n\n* required for run mode\n\nclient_info options:\n\n* client_list: list of hostname:port for all client nodes. Required option\n* port: int, the master port the clients will connect to to forward their results.\n* threads: int, Specifies how many processes a single client should spawn to forward requests. The number of possible concurrent connections is limited by the number of threads per client times the number of clients\n* wait: boolean, if true, the client threads will wait the relative time between requests as recorded in the traces. If false or absent, the clients will send requests as fast as possible.\n* route: optional boolean which instructs the master to route requests to clients based on the remote address of the traces rather than round robin routing.\n\n#### verbose\n* Optional Booblean, if true prints more information to standard out\n\n#### trace\n* Required for all modes\n\ntrace options:\n\n* location, optional path to the anonymized traces, current directory assumed if absent. Absolute path should be used.\n* traces, mandatory list of trace files to be read as input\n* limit, optional structure to specify how to limit the number of requests\n..* type: string that can be either \"seconds\" or \"requests\" \n..* amount: specifies the limit of the specified type. For example, if type is seconds and amount is 100, only the first 100 seconds of requests will be used\n* output: optional filename of where to store output trace of run mode. Default is output.json\n\n### registry\n\n* list of registries to send requests to in warmup and run mode. Only first entry is used in warmup mode, however\n\n### warmup\n\n* required for run and warmup modes\n\nOptions:\n\n* output: required filename, is populated by warmup and needed by run for mapping the trace uri to a blob\n* threads: optional int, specifies how many threads should be used to send requests to registry in warmup mode\n* random: optional boolean, instructs the master to create random files for layers rather than creating files of 0's\n\n### simulate\n\n* Required for simulate mode\n\nOptions:\n\n* name: required string, should be the name of the python file containing the simulation. File should contain an init(requests) or init(requests, args) function to start the simulation\n* args: optional dictionary that can contain any arguments the simulation needs\n\n### Example\n\n\nclient_info:\n\n    client_list:\n        - localhost:8081\n        - localhost:8082\n    port: 8080\n    threads: 100\n    wait: true\n\nverbose: true\n\ntrace:\n\n    location: /data/data_centers/dal09\n    traces:\n        - prod-dal09-logstash-2017.08.01-0.json\n    limit:\n        type: requests\n        amount: 20\n\n    output: results.json\n\nregistry:\n\n    - localhost:5000\n\nwarmup:\n\n    output: interm.json\n\n    threads: 100\n\nsimulate:\n\n    name: cache.py\n    args:\n        cache_size: 8\n\n\n# Instalation\n\nsudo apt-get install python-pip\n\nsudo pip install python-dxf\n\nsudo pip install requests\n\nsudo pip install bottle\n\nsudo pip install pyyaml\n\nsudo pip install hash_ring\n\n# Data format for simulation\n* sorted by delay (timestamp of request)\n\n[\n\n    {\n        'delay': timestamp (int),\n        'uri': 'v2/user/repo/\u003cblobs or manifests\u003e/layer',\n        'size', size (int),\n        'method': \u003c'GET' or 'PUT'\u003e,\n        'duration': duration (int),\n        'client': remote address (string)\n    },\n]\n\n# Support Channel\n\nMichael Littley: milit93@vt.edu\nAli Anwar: ali@vt.edu\n\nSupport is also available through the github repository\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds2-lab%2Fdocker-registry-trace-replayer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fds2-lab%2Fdocker-registry-trace-replayer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds2-lab%2Fdocker-registry-trace-replayer/lists"}