{"id":13851998,"url":"https://github.com/trulia/webpagetest-charts-api","last_synced_at":"2025-07-13T03:32:53.065Z","repository":{"id":29293449,"uuid":"32826338","full_name":"trulia/webpagetest-charts-api","owner":"trulia","description":"An api of endpoints to build charts from WebPagetest results","archived":false,"fork":false,"pushed_at":"2024-03-25T20:59:52.000Z","size":71,"stargazers_count":67,"open_issues_count":4,"forks_count":11,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-08-05T22:37:39.655Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trulia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-24T21:30:58.000Z","updated_at":"2023-08-19T00:35:53.000Z","dependencies_parsed_at":"2022-08-17T20:00:20.160Z","dependency_job_id":null,"html_url":"https://github.com/trulia/webpagetest-charts-api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trulia%2Fwebpagetest-charts-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trulia%2Fwebpagetest-charts-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trulia%2Fwebpagetest-charts-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trulia%2Fwebpagetest-charts-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trulia","download_url":"https://codeload.github.com/trulia/webpagetest-charts-api/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225855381,"owners_count":17534962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T22:00:52.893Z","updated_at":"2024-11-22T06:30:49.119Z","avatar_url":"https://github.com/trulia.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# WebPagetest Charts API\n[WebPagetest](http://www.webpagetest.org/) Rules. There are tools that are easier to use, but nothing that lets you\nreally see deeply into the browser side of things. But there's no easy way to compare results over time.\nSo this is a small express application that runs tests, stores them, and offers endpoints to access the\ndata. It assumes that you'll want to look at a variety of charts for your data, so the\nfollowing datapoints are available:\n\n- SpeedIndex: Google's special score.  It's an excellent\nsummtion of how Google will see your site, and a great\nnumerical indicator for how fast your site feels to\nvisitors. Read the  [SpeedIndex docs](https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/speed-index)  for more info\n- loadTime: How long (ms) to load the critical page content\n- fullyLoaded: How long (ms) to fully load all the page content\n- ~~requests: how many requests are made when loading the page~~ (needs update for new API response)\n- TTFB: time to the first byte recieved, rumored to be the most important for SEO\n- visualComplete: Time (ms) until the page is done changing visually\n- Lighthouse Suite: [Lighthouse](https://developers.google.com/web/tools/lighthouse/) tracks a lot of metrics, the ones WepageTest expsoses are:\n  - Performance\n  - Performance.speed-index-metric\n  - Performance.first-meaningful-paint\n  - Performance.estimated-input-latency\n  - Performance.first-interactive\n  - Performance.consistently-interactive\n  - BestPractices\n  - Accessibility\n  - SEO\n  - ProgressiveWebApp\n\n\nIt also keeps links to\nthe full WebPagetest results for deeper introspection. You can build a UI on\nthis API. A working example is https://github.com/trulia/webpagetest-charts-ui. Visit that repo to see screenshots of what it can display.\n\nAnd none of this would have happened without [marcelduran](https://github.com/marcelduran) and his\n[webpagetest-api](https://github.com/marcelduran/webpagetest-api)\nmodule which made the data side of this very easy to prototype quickly.\n\n## How It Works\nIn this repo, there's no database: just the file system. (The data storage logic is its own\nmodule, so it could be replaced with something else, mongo, etc. PRs welcome!) The app saves results into\n`public/results/\u003ctest-suite-name\u003e/\u003ctest-name\u003e/\u003cyyyy-mm-dd-hh-mm-ss/` directories\ncontaining the json results from the test.\n\nThe endpoints then serve this data up in chartable summaries, as well as specific\ndatapoints. The available endpoints are served from the `/` url of the api.\n\n\n## Installation\n\n1. clone this repo\n\n1. Create a json config file with your test suites. See 'Test Suite Config' below.\n\n1. Decide if you want to use the filesystem to store your data (default),\nor a database. The Filesystem works well, but can get slow if there are\nlots of data points.  If you want to use a database:\n  1. Create a postgres/mysql/sqlite table with the schema (postgres shown, datapoint_id needs to be an auto increment column.):\n\n    ```sql\n    CREATE TABLE webpagetestcharts (\n  \t  test_results text  NOT NULL,\n  \t  date timestamp without time zone  NOT NULL,\n  \t  suite_id text  NOT NULL,\n  \t  test_id text  NOT NULL,\n  \t  datapoint_id integer DEFAULT nextval('id_seq'::regclass) NOT NULL\n    );\n    ```\n\n  1. Edit the file `data_store/index.js` to require the correct interface (`db`).\n  1. Edit the connection string in `data_store/db.js` to connect to the database.\n  1. In `package.json`, update the `any-db-*` database module to reference\n  the correct db type (postgres is the default).\n\n1. do an `npm install`\n\n1. Run the server with: `SUITE_CONFIG=/your/wpt-api/config-file.json npm start`. You MUST specify a `SUITE_CONFIG` otherwise you'll get an error when the app starts.\n\n1. Visit the host/port the application is running on.  By default this will be http://localhost:3001 to get a list of available endpoints.\n\nNote: The tests take a\nfew minutes to run, so there might not be much to see initially. And please don't\nspam WPT with requests, if you are constantly\nrestarting the server, you may be sending\nlots of tests to your API key.\n\nIf things get weird You can get into debug mode by adding `DEBUG='wpt-api*'` to your startup command.\n\nOr specify just the module you're debugging for less verbosity.\nBuilding a service on top of a module that talks to a service can get weird sometimes,\nso there's lots of debugging.\n\nOnce everything is working well wrap it up in [forever](https://www.npmjs.com/package/forever) or [pm2](https://www.npmjs.com/package/pm2) or whatever you want to keep it up and running.  For example, the forever command\nshould look like:\n\n```\nSUITE_CONFIG=config.json forever start bin/www\n```\n\n## Test Suite Config\nThe json file is the only file needed for this app to run the way you want it to. The trickiest\nidea here is the 'parentPage' concept. Often you want to test a page that's in an ever changing\nlist (eg: users with the most photos), or isn't persistent (eg: search results on temporal objects). For those\ncases you define the page that has your list, and give a selector to get the link that actually\nshould be tested.\n\nSometimes WebPagetest struggles, sometimes your site struggles. Sometimes for the sake\nof the chart, you need to ignore outliers. You can specify a data range and date range\nin your config and those results will be ignored in the chart\n(though you can choose to show them)\n\n\nHere is a annotated sample. Note that these comments are not valid json, so it's not\ncut and pasteable.  A usable test version is in the repo under example_config.json\n\n```JavaScript\n{\n  // you'll need one of these\n  \"wptApiKey\": \"get one from http://www.webpagetest.org/getkey.php\",\n\n  // A Suite is a collection of urls under a theme.\n  // e.g.: User Profiles.\n  \"testSuites\":[\n    {\n      // The title of the suite, used in links in the UI\n      \"suiteDisplayName\": \"Mobile User Profiles\",\n\n      // used in urls and in the file system (`[\\w\\-]` only please)\n      \"suiteId\": \"user_profiles\",\n\n      // nice to tell people what's happening\n      \"desc\": \"This suite runs on a Motorola G phone using Chrome in Dulles, VA over 3G data\",\n\n      // minutes\n      \"runEvery\": 120,\n\n      // the host for the suite's urls\n      \"testHost\": \"http://example.com/\",\n\n      // this magic string comes from WPT via the locations command:\n      // https://github.com/marcelduran/webpagetest-api\n      \"location\": \"Dulles_MotoG:Motorola G - Chrome\",\n\n      // if you want to specify a user agent when making the request\n      // for a parentPage (eg: a mobile device).\n      \"parentRequestUserAgent\": \"\",\n\n      // if you want to pass data along in the query string for each test\n      // handy for turning features on and off to compare\n      \"queryStringData\": {},\n\n      // config for the chart data basedon the chart type\n      // helps keep the default chart view sane sometimes if\n      // there are occasional outliers (eg: speedindex 9999999)\n      // that can skew the chart display. I recommend skipping this\n      // bit initially until you better know your data.\n      \"chartConfig\" : [\n        {\n          // the type of chart these settings apply to\n          \"type\" : \"SpeedIndex\",\n          // the lo/hi values to give to the chart\n          \"dataRange\": [0, 8000],\n          // how many days of datapoints to return\n          \"dateCutoff\": 30\n        }\n      ],\n\n      // the array of urls to test.  you can give it normal urls as\n      // well as urls to get urls from (eg: to get the first result from a search)\n      \"testPages\": [\n        {\n          // nice to read\n          \"testDisplayName\": \"Test User Profile\",\n\n          // used in urls and in the file system (`[\\w\\-]` only please)\n          \"testId\": \"publicProfile\",\n\n          // url to test using the host above.  \n          \"path\": \"/profile/testUser\"\n\n          // tests can have their own host if they need to override the suite\n          // eg, our pages vs. our competitors pages.\n          \"testHost\": \"http://example.org/\"\n        },\n        {\n\n          // Nice to read\n          \"testDisplayName\": \"Most Popular User Profile\",\n\n          // used in urls and in the file system (`[\\w\\-]` only please)\n          \"testId\": \"mostPopularUser\",\n\n          // The path that holds the url that will ultimately be tested\n          \"parentPath\": \"/search/profiles/popular\",\n\n          // the selector to get the item that has the href to be tested.\n          // The first match will be used. What goes here is wrapped in `$()`\n          \"parentHrefSelector\": \".userResult\"\n        }\n      ]\n    }\n  ]\n}\n  ```\n\n## Contributing\nPRs are Happily Accepted! The preferred PR method is:\n\n1. Fork this repo\n2. Create a feature branch on your fork\n3. code things\n4. PR your feature branch to this master\n5. We'll check out your PR, test, code review and when it's ready merge it in.\n\nIf you have a larger idea, feel free to bring it up in an issue first.  Please\nnote that this project is released with a Contributor Code of Conduct. By\nparticipating in this project you agree to abide by its terms.\n\n## Todo\n1. Package into something npm installable\n1. Allow a custom directory for data, as opposed to `public/results`\n1. Let the config do more around configuring the tests.\n1. Refactor config to be less confusing (each test a file in the config directory?)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrulia%2Fwebpagetest-charts-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrulia%2Fwebpagetest-charts-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrulia%2Fwebpagetest-charts-api/lists"}