{"id":21216536,"url":"https://github.com/nodegame/nodegame-benchmark","last_synced_at":"2025-03-15T00:41:20.978Z","repository":{"id":145309794,"uuid":"12759569","full_name":"nodeGame/nodegame-benchmark","owner":"nodeGame","description":"Benchmark and Testing for nodeGame","archived":false,"fork":false,"pushed_at":"2016-08-09T22:02:11.000Z","size":15215,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-21T16:44:20.547Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nodeGame.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-09-11T15:12:16.000Z","updated_at":"2016-08-09T21:56:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"9c459bd8-4579-4705-8987-ec7f7cb0f514","html_url":"https://github.com/nodeGame/nodegame-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodeGame%2Fnodegame-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodeGame%2Fnodegame-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodeGame%2Fnodegame-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodeGame%2Fnodegame-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nodeGame","download_url":"https://codeload.github.com/nodeGame/nodegame-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243667964,"owners_count":20328036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T21:52:54.737Z","updated_at":"2025-03-15T00:41:20.962Z","avatar_url":"https://github.com/nodeGame.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nodegame-benchmark (beta)\n\nBenchmark and Testing for nodeGame.\n\n## Overview\n\nThe benchmark script reads settings from a config file, starts the\nnodeGame server, connects a number of PhantomJS clients to the specified\nchannel and lets them play.\n\nWhile playing, the script collects statistics about the number and the\ntiming of all exchanged messages, and also about CPU and memory usage\n(if the dependency `psutil` is installed).\n\nWhen all games are finished, all statistics are saved to csv files, as\nspecified in the settings. If an automated test script is defined by the\n`package.json` file inside the game directory, such a test-script will\nexecuted and the results displayed on screen.\n\n\n## Usage\n\n1. Create a copy of the file `config-sample.ini` with name `config.ini`\n2. Edit `config.ini` with the correct settings for your environment (see below)\n3. Run the benchmark either with `python run_benchmark.py` or simply with\n   `./run_benchamrk.py`\n\n### config.ini\n\nThis file needs to define the following variables. Note that it\nis possible to refer to variables defined in other sections via\n`${SECTION_NAME:VAR_NAME}` syntax.\n\n```ini\n[General Settings]\n    game: ultimatum\n\n; This is the directories section.\n; All of these directories need to exist.\n[Directories]\n    nodegame_dir: ~/nodegame\n    benchmark_dir: ${nodegame_dir}/node_modules/nodegame-benchmark\n    client_dir: ${nodegame_dir}/node_modules/nodegame-client\n    server_dir: ${nodegame_dir}/node_modules/nodegame-server\n    game_dir: ${nodegame_dir}/games/${General Settings:game}\n\n    ; This is the directory where stdout and stderr logs will be written.\n    log_dir: ${benchmark_dir}/log\n\n    ; This is the directory where data in csv format will be written.\n    csv_dir: ${benchmark_dir}/csv\n\n    ; This folder contains the launcher of the benchmark.\n    launcher_dir: ${nodegame_dir}/test\n\n    ; This folder determines the \"current working directory\" of the launcher.\n    launcher_cwd: ${nodegame_dir}\n\n    ; This folder contains the logs of messages.\n    msg_log_dir: ${nodegame_dir}/log\n\n    ; This folder is the current working directory of the test that sanitizes\n    ; benchmark results.\n    test_cwd: ${game_dir}\n\n; This is the files section.\n; All of these files need to exist.\n[Files]\n    ; This is where the values of the [Client Variables] section are written.\n    client_var_file: ${Directories:client_dir}/lib/modules/variables.js\n\n    ; This is where the values of the [Server Variables] section are written.\n    server_var_file: ${Directories:server_dir}/conf/servernode.js\n\n    ; This file contains the messages received and sent by the server.\n    server_msg_file: ${Directories:msg_log_dir}/messages.log\n\n    ; These files specify the launcher and its settings file.\n    launcher_file: ${Directories:launcher_dir}/launcher-autoplay.js\n    launcher_settings_file: ${Directories:game_dir}/test/settings.js\n\n[Client Variables]\n    rel_msg_var: k.reliableMessaging\n    rel_retry_var: k.reliableRetryInterval\n\n[Server Variables]\n    rel_msg_var: servernode.reliableMessaging\n    rel_retry_var: servernode.reliableRetryInterval\n\n; The settings specified here will be written to the launcher_settings_file\n[Launcher Settings]\n    ; sioTransports are not working in Express 4 at the moment.\n    ; sioTransports: [\"websocket\", \"flashsocket\", \"htmlfile\", \"xhr-polling\",\n    ;                 \"jsonp-polling\"]\n```\n\n## Options for run_benchmark\n\n```\n$ ./run_benchmark.py --help\nusage: run_benchmark.py [-h] -c CONFIG [-n NUM_CONNS [NUM_CONNS ...]] [-r]\n                        [-nr] [-t TIMEOUTS [TIMEOUTS ...]]\n\nExecute nodegame benchmark and write benchmark data to csv file.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -c CONFIG, --config CONFIG\n                        Benchmark configuration file in INI format containing\n                        variables that likely do not change between\n                        benchmarks.\n  -n NUM_CONNS [NUM_CONNS ...], --num_conns NUM_CONNS [NUM_CONNS ...]\n                        Number of simultaneous connections to consider for the\n                        benchmark, can be a list.\n  -r, --reliable        Boolean flag to turn on reliable messaging.\n  -nr, --no_run         Boolean flag to disable launching the game. Will just\n                        process existing log files.\n  -t TIMEOUTS [TIMEOUTS ...], --timeouts TIMEOUTS [TIMEOUTS ...]\n                        Timeouts to consider for the benchmark when reliable\n                        messaging is used, can be a list.\n```\n\n## File format of metrics.csv\n\n`metrics.csv` defines the following data headers:\n\n- `id`: Unix timestamp of the experiment in milliseconds. It is very unlikely\n  that two different runs have the same timestamp, hence the use as an\n  identifier is justified.\n- `machine`: Name of the current machine as reported by Python's\n  [`platform.platform()`](https://docs.python.org/3.4/library/platform.html#platform.platform).\n- `num_conns`: Number of simultaneous connections during this run of the\n  experiment.  This is the number specified by the `-n` flag.\n- `is_reliable`: Reports whether reliable messaging was turned on or off. This\n  is specified by the presence or absence of the `-r` flag.\n- `timeout`: Reports the timeout used during the current run. Only meaningful\n  if reliable messaging was turned on. Controlled via the `-t` flag.\n- `benchmark_ret_code`: Return code of the benchmark process. A number\n  different from 0 indicates that there was a problem.\n- `test_ret_code`: Return code of the test process. A number different from 0\n  indicates that there was a problem.\n- `cpu_time_user`: User CPU time as reported by\n  [`psutil.Process.cpu_times()`](https://pythonhosted.org/psutil/#psutil.Process.cpu_times).\n  This time also includes CPU time spent by children processes of the current\n  process.\n- `cpu_time_system`: System CPU time as reported by\n  [`psutil.Process.cpu_times()`](https://pythonhosted.org/psutil/#psutil.Process.cpu_times).\n  This time also includes CPU time spent by children processes of the current\n  process.\n- `mem_info_rss`: Resident Set Size (RSS) usage as reported by\n  [`psutil.Process.memory_info()`](https://pythonhosted.org/psutil/#psutil.Process.memory_info).\n  This usage also includes memory usage by children processes of the current\n  process.\n- `mem_info_vms`: Virtual Memory Size (VMS) usage as reported by\n  [`psutil.Process.memory_info()`](https://pythonhosted.org/psutil/#psutil.Process.memory_info).\n  This usage also includes memory usage by children processes of the current\n  process.\n- `avg_client_time`: When reliable messaging is enabled, this is the average\n  time to process a client message. Time is measured as the duration between\n  receiving a client message and sending the corresponding ACK message.\n  Currently this is only measured on the server and hence is not meaningful.\n- `avg_server_time`: When reliable messaging is enabled, this is the average\n  time for a round trip for messages from the server. The duration reported is\n  the difference between sending a message from a server and receiving the ACK\n  for it.\n\n\n## Example Runs\n\nReads the config file and run 1 benchmark where there is 1 game:\n\n    ./run_benchmark.py -c config.json -n 1\n\n\nHere we consider 1, 2, 4 and 8 simultaneous connections, reliable\nmessaging to be activated, and timeouts of 1000, 2000 and 4000\nmilliseconds.\n\n    ./run_benchmark.py -c config.json -n 1 2 4 8 -r -t 1000 2000 4000\n\n\n## Requirements and Dependencies\n\nTo be able to report CPU and memory usage, `run_benchmark.py` relies\non the third party module `psutil`. You should be able to install it\nvia:\n\n     pip install psutil\n\nSee\n[this manual](https://github.com/giampaolo/psutil/blob/master/INSTALL.rst)\nfor further help.\n\nFurthermore the script relies on features on included with Python 2.7,\nso please make sure to have a recent Python installation available.\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnodegame%2Fnodegame-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnodegame%2Fnodegame-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnodegame%2Fnodegame-benchmark/lists"}