{"id":17349831,"url":"https://github.com/stasm/raptor-compare","last_synced_at":"2025-04-14T21:12:43.295Z","repository":{"id":18897743,"uuid":"22115891","full_name":"stasm/raptor-compare","owner":"stasm","description":"Compare sets of Raptor results and test for statistical significance of the observed difference.","archived":false,"fork":false,"pushed_at":"2015-11-19T10:54:18.000Z","size":71,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-14T21:12:38.055Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/raptor-compare","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stasm.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-07-22T18:31:15.000Z","updated_at":"2015-10-14T13:52:57.000Z","dependencies_parsed_at":"2022-09-10T21:04:08.378Z","dependency_job_id":null,"html_url":"https://github.com/stasm/raptor-compare","commit_stats":null,"previous_names":["stasm/test-perf-summary"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stasm%2Fraptor-compare","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stasm%2Fraptor-compare/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stasm%2Fraptor-compare/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stasm%2Fraptor-compare/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stasm","download_url":"https://codeload.github.com/stasm/raptor-compare/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248961236,"owners_count":21189993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T16:57:15.748Z","updated_at":"2025-04-14T21:12:43.269Z","avatar_url":"https://github.com/stasm.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"raptor-compare [![Build Status][travisimage]][travislink]\n============================================================\n\n[travisimage]: https://travis-ci.org/stasm/raptor-compare.png?branch=master\n[travislink]: https://travis-ci.org/stasm/raptor-compare\n\nCompare sets of Raptor results and test for their statistical significance \n([t-test][] with 0.05 alpha).\n\n[t-test]: https://en.wikipedia.org/wiki/Student%27s_t-test\n\n    $ raptor-compare my_test.ldjson\n\n    music.gaiamobile.org   base: mean  1: mean  1: delta  1: p-value\n    ---------------------  ----------  -------  --------  ----------\n    navigationLoaded              711      726        14        0.06\n    navigationInteractive         737      748        12        0.10\n    visuallyLoaded               1322     1217      -105      * 0.00\n    contentInteractive           1323     1217      -105      * 0.00\n    fullyLoaded                  1462     1442       -20        0.14\n    uss                        19.881   20.370     0.489      * 0.00\n    pss                        23.468   23.981     0.513      * 0.00\n    rss                        39.640   40.152     0.512      * 0.00\n\nIn the example above, Raptor measurements for the Music app were stable for the \n`visuallyLoaded` and `contentInteractive` events, as indicated by the asterisks \nnext to the p-values.  At the same time, we can see that the memory footprint \nhas regressed: the mean `uss` usage is higher than the base measurement and the \ndifference is statistically significant as well.\n\nFor all measurements marked with the asterisk (`*`) it is valid to assume that \nthe means are indeed significantly different between the base and the try runs.\n\nThe remaining results, e.g. the 20 ms `fullyLoaded` speed-up, are not \nsignificant and might be caused by a random instability of the data.  Try \nincreasing the sample size (via Raptor's `--runs` option; see below) and run \nRaptor again.\n\n\nWhat is p-value?\n----------------\n\nThe p-value is a concept used in statistical testing which represents our \nwillingness to make mistakes about the data.  A low p-value means that there's \nonly a small risk of making a mistake by concluding that the test data \nindicates that the means are truly different and that the observed differences \nare not due to poor sampling and randomness.\n\nFor the data above, a p-value of 0.14 for `fullyLoaded` means that the risk of \nbeing wrong is 14% when we conclude that the 20 ms difference between the means \nis due to an actual code change and not to randomness.\n\nGood p-values are below 0.05.\n\n\nInstallation\n------------\n\n    npm install -g raptor-compare\n\n\nRunning Raptor tests\n--------------------\n\n(For best results, follow the [Raptor][] guide on MDN.)\n\nInstall Raptor with:\n\n    $ sudo npm install -g @mozilla/raptor\n\nConnect your device to the computer, go into you Gaia directory and build Gaia:\n\n    $ make raptor\n\nThen, run the desired perf test:\n\n    $ raptor test coldlaunch --runs 30 --app music --metrics my_test.ldjson\n\nRaptor will print the output to `stdout`.  The raw data will be saved in the \n`ldjson` file specified in the `--metrics` option.  The data is appended so you \ncan runmultiple tests for different revisions and apps and `raptor-compare` \nwill figure out how to handle it.  All testing is conducted relative to the \nfirst result set for the given app.\n\n[Raptor]: https://developer.mozilla.org/en-US/Firefox_OS/Automated_testing/Raptor\n\n\nAPI\n---\n\nYou can also use _raptor-compare_ programmatically.  It exposes three functions \nfor working with Raptor data: `read` reads in a LDJSON stream with the raw \nmetrics data, `parse` aggregates the data into a Map and `build` creates the \ncomparison tables with p-values for significance testing.\n\n```javascript\n// Needed for Node.js 0.10 and 0.12.\nrequire('babel/polyfill');\n\nconst fs = require('fs');\nconst compare = require('raptor-compare');\n\ncompare.read(fs.createReadStream(filename))\n  .then(compare.parse)\n  .then(compare.build)\n  .then(tables =\u003e tables.forEach(\n    table =\u003e console.log(table.toString())))\n  .catch(console.error);\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstasm%2Fraptor-compare","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstasm%2Fraptor-compare","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstasm%2Fraptor-compare/lists"}