{"id":19211341,"url":"https://github.com/truebrain/openttd-performance","last_synced_at":"2025-09-09T14:40:12.002Z","repository":{"id":188052480,"uuid":"678030241","full_name":"TrueBrain/OpenTTD-performance","owner":"TrueBrain","description":"Performance measurement of OpenTTD","archived":false,"fork":false,"pushed_at":"2024-04-11T07:13:30.000Z","size":16500,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-04T17:14:11.096Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TrueBrain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-13T13:04:06.000Z","updated_at":"2023-08-13T13:11:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"93a05502-f360-482f-ae07-e8f226571014","html_url":"https://github.com/TrueBrain/OpenTTD-performance","commit_stats":null,"previous_names":["truebrain/openttd-performance"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBrain%2FOpenTTD-performance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBrain%2FOpenTTD-performance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBrain%2FOpenTTD-performance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBrain%2FOpenTTD-performance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TrueBrain","download_url":"https://codeload.github.com/TrueBrain/OpenTTD-performance/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240286517,"owners_count":19777354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T13:42:12.795Z","updated_at":"2025-02-23T07:43:45.418Z","avatar_url":"https://github.com/TrueBrain.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenTTD performance\n\nThis repository monitors OpenTTD's performance day-to-day, to ensure that the developers are made aware of any regression in terms of CPU or memory.\n\nThese performance tests are automatically executed every night, with the last commit since midnight.\n\n## Running the performance tests\n\n### Method of measuring\n\nMeasuring performance is not trivial, as it strongly depends on the CPU and other hardware.\nBut even with the exact same hardware, performance can differ (CPU branch-prediction etc).\n\nFor memory, GNU time is used to find the peak memory usage, in KiB.\nThis value is accurate within +/- 1% between runs.\n\nFor CPU, \"perf\" is used to measure user-time.\nFor each performance test, OpenTTD is ran 5 times, and it is validated the deviation between the runs isn't more than 0.5%.\nIf it is, the test is restarted, until it is within 0.5%.\n\nAdditionally, as we use GitHub Runners to execute the performance tests, we also have to account for different hardware.\nLastly, dependencies might be different from day-to-day.\nTo compensate as much as possible for these differences, each savegame is run in multiple different setups:\n\n1) a pre-compiled 12.0 binary that is identical between every run.\n   This is used to normalize the results between different days.\n2) a freshly compiled 12.0 binary, to validate if the dependencies / compiler have any influence on the result.\n3) yesterday's version, to compare today's version against.\n4) today's version.\n\nWith gathering these four results for every savegame in this performance suite every night, we should be able to discover trends, and which night CPU / memory changed.\n\n### OpenTTD's configuration\n\nEvery performance test is run based on the last commit of that day.\nIt is compiled as `Release` build with `assert()` turned off.\nWe also forcefully disable the use of threads, as they are a known source of CPU timing difference (most commonly when linkgraph is enabled).\n\nAdditionally, a configuration is loaded to disable autosaves.\nLastly, patches are loaded to make OpenTTD compile with modern compilers / dependencies.\n\nOpenGFX 7.1 is used; not that it really matters, but a BaseGraphics is required for OpenTTD to run.\n\nOpenTTD is run with `-X -c performance-config/openttd.cfg -x -snull -mnull -vnull:ticks=${ticks} -g ${map}`, where `${ticks}` and `${map}` is depending on the savegame.\nThe amount of ticks are given such that each run takes ~10 seconds.\n\nAs each savegame is ran 5 times to get an accurate CPU time, this means it takes ~1 minutes for a single benchmark to finish.\n\n### CPU / Memory profiling\n\nAdditional to measuring performance difference, we also create a profile of the CPU usage and memory usage, to quickly spot what changed.\n\nFor memory, this requires us to link against `tcmalloc` library.\nThis should have no impact on performance normally, but does allow us to turn on memory profiling.\n\nIn both cases, we make use of gperftools, to capture a CPU and memory profile.\nThey are stored as pprof blobs, and can be analyzed with pprof.\nFor this you need golang installed.\n\n```bash\ngo tool prof -http :8080 \u003chttps-link-to-raw-file-on-github\u003e\n```\n\nYou can also compare two versions with each other, by using `-base \u003clink\u003e` first.\n\nNote: memory profiling is done with fewer ticks than CPU profiling.\nWe do a lot of heap activity, slowing down the game by a lot, making it impractical to run memory profiling for longer.\n\n## Savegames\n\nThis performance suite contains a wide variation of savegames, ranging from very large maps, to very busy networks, etc.\nWe are constantly looking for good maps to add to the performance suite.\nBut, as every map takes ~10 minutes to run, we do need to be realistic in how many savegames we want to benchmark against.\nSome savegames simply do not give new information, as they are very similar to other savegames, and show identical behaviour.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftruebrain%2Fopenttd-performance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftruebrain%2Fopenttd-performance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftruebrain%2Fopenttd-performance/lists"}