{"id":13821179,"url":"https://github.com/edmunds/shadowreader","last_synced_at":"2026-01-16T06:44:51.956Z","repository":{"id":141420159,"uuid":"126246025","full_name":"edmunds/shadowreader","owner":"edmunds","description":"Serverless load testing for replaying website traffic. Powered by AWS Lambda.","archived":false,"fork":false,"pushed_at":"2019-08-15T17:45:00.000Z","size":4623,"stargazers_count":158,"open_issues_count":8,"forks_count":13,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-11-19T21:35:58.991Z","etag":null,"topics":["aws","aws-lambda","aws-s3","benchmarking","load-testing","performance-testing","python","serverless"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edmunds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-03-21T22:04:40.000Z","updated_at":"2024-11-05T18:26:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"5c181cef-0170-4259-849d-74d8e73faa9f","html_url":"https://github.com/edmunds/shadowreader","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmunds%2Fshadowreader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmunds%2Fshadowreader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmunds%2Fshadowreader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmunds%2Fshadowreader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edmunds","download_url":"https://codeload.github.com/edmunds/shadowreader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254530601,"owners_count":22086644,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","aws-s3","benchmarking","load-testing","performance-testing","python","serverless"],"created_at":"2024-08-04T08:01:16.874Z","updated_at":"2026-01-16T06:44:51.945Z","avatar_url":"https://github.com/edmunds.png","language":"Python","readme":"# ShadowReader\n\n\u003ca href=\"https://travis-ci.com/edmunds/shadowreader\"\u003e\u003cimg alt=\"Build Status\" src=\"https://travis-ci.com/edmunds/shadowreader.svg?branch=master\"\u003e\u003c/a\u003e\n\u003ca href=\"http://www.serverless.com\"\u003e\u003cimg alt=\"Serverless\" src=\"http://public.serverless.com/badges/v3.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/ambv/black\"\u003e\u003cimg alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"\u003e\u003c/a\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/edmunds/shadowreader/master/docs/imgs/logo.png\" alt=\"ShadowReader Logo\" width=\"50%\" height=\"50%\"/\u003e\n\u003c/p\u003e\n\nShadowReader has the ability to replay production traffic to a destination of your choice by collecting traffic patterns from access logs. It is built on AWS Lambda, S3 and Elastic Load Balancers.\n\n\u003c!-- \u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/edmunds/shadowreader/blob/master/docs/imgs/example1.png?raw=true\" alt=\"shadow-reader-example1\" width=\"75%\" height=\"75%\"/\u003e\n\u003c/p\u003e --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/edmunds/shadowreader/blob/master/docs/imgs/load-test-animation.gif?raw=true\" alt=\"shadow-reader-animation\" width=\"90%\" height=\"90%\"/\u003e\n\u003c/p\u003e\nIn the chart above, the blue line is the request rate of ShadowReader while in orange is the load on the production website.\n\nShadowReader mimics real user traffic by replaying URLs from production at the same rate as the live website. Being serverless, it is more efficient cost and performance wise than traditional distributed load tests and in practice has scaled beyond 50,000 requests / minute.\n\nSupport for replaying logs from these load balancers:\n\n- [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html)\n- [Classic Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/introduction.html)\n- [Any logs stored locally, just provide a RegEx for the log format](docs/local_parser.md)\n- (Support for other types of load balancers planned)\n\n### Read about it\n### [How we fixed a Node.js memory leak by using ShadowReader to replay production traffic into QA](http://technology.edmunds.com/2018/08/25/Investigating-a-Memory-Leak-and-Introducing-ShadowReader/)\n\n\n## Documentation\n\nTo get started, see:\n\n- [Batteries included demo for trying out live replay](#live-replay)\n- [More in-depth guide for setting up past replay](docs/setup.md)\n- [How to send load test metrics to CloudWatch](docs/cloudwatch-guide.md)\n- [Parse locally stored logs](docs/local_parser.md)\n- [Filter requests from replay](docs/filters.md)\n\n## Trying Live Replay \u003ca name=\"live-replay\"/\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/edmunds/shadowreader/blob/master/docs/imgs/live-replay2.png?raw=true\" alt=\"shadow-reader-live-demo-results\" width=\"80%\" height=\"80%\"/\u003e\n\u003c/p\u003e\n\nThis guide will deploy all the necessary AWS resources to try out ShadowReader's live replay feature (replay requests as they come in).\n\nThis demo uses `us-east-1` AWS region. Substitute all references to `us-east-1` with a region of your choice.\n\n## Deploy AWS resources for demo\n\nThe provided CloudFormation stack will deploy these AWS resources for the demo: A VPC, subnets, 2 ALBs and a S3 bucket.\n\nOne ALB comes preconfigured to send logs to S3.\nThis demo will walk you through in setting up ShadowReader to parse those logs then replay it to the 2nd ALB.\n\n```\nDeploy the stack:\ncurl https://raw.githubusercontent.com/edmunds/shadowreader/master/docs/demo-cf.yml --output demo-cf.yml\naws cloudformation deploy --stack-name sr-demo-resources  --template-file demo-cf.yml --region us-east-1\n```\n\nIf you'd rather not deploy a new VPC and subnets, this CF will allow you to choose already existing VPC/Subnets.\n```\ncurl https://raw.githubusercontent.com/edmunds/shadowreader/master/docs/demo-cf-choose.yml --output demo-cf-choose.yml\naws cloudformation deploy --stack-name sr-demo-resources  --template-file demo-cf-choose.yml --region us-east-1 --parameter-overrides subnet1=$SUBNET_ID1 subnet2=$SUBNET_ID2 vpcid=$VPCID\n```\n\n## Set up shadowreader.yml\n\nCopy `shadowreader.example.yml` to `shadowreader.yml`\n\n```\ncp shadowreader.example.yml shadowreader.yml\n```\n\nSet `access_logs_bucket` in `shadowreader.yml` like below.\n\nReplace AWS_ACCOUNT_ID with your account id (should be an integer like `123450493079`).\nReplace AWS_REGION with the region you deployed the CloudFormation to.\n\n```\nenvironment:\n    access_logs_bucket: sr-access-logs-$AWS_REGION-$AWS_ACCOUNT_ID/AWSLogs/$AWS_ACCOUNT_ID/elasticloadbalancing/$AWS_REGION/\n```\n\nSet `replay_mode` to `replay_live`\n\n```\nplugins:\n  loader_middleware: loader_middleware\n  replay_mode: replay_live\n```\n\n## Set up serverless.yml\n\nCopy `serverless.example.yml` to `serverless.yml`.\n\n```\ncp serverless.example.yml serverless.yml\n```\n\n(Both `serverless.yml` and `shadowreader.yml` must be configured before deployment via the [Serverless framework](https://serverless.com/).)\n\nUpdate `my_project_name` in `serverless.yml`. This is your project name.\nIt is to ensure that the S3 bucket used by ShadowReader has unique naming (S3 bucket names must be globally unique).\n\n```\ncustom:\n  my_project_name: my-unique-project-name\n```\n\nFind out the DNS name for the ALB we will be load testing.\n\n```\n$ aws elbv2 describe-load-balancers --names SR-Demo-ALB-receiving --region us-east-1 | grep DNSName\n\u003e\u003e \"DNSName\": \"SR-Demo-ALB-receiving-123459376.us-east-1.elb.amazonaws.com\",\n```\n\nFind `orchestrator-past` in `serverless.yml` and edit `base_url` to be `http://{DNSName}`.\n\n```\norchestrator-past:\n  handler: functions/orchestrator_past.lambda_handler\n  events:\n    - schedule: rate(1 minute)\n  environment:\n    test_params: '{\n                    \"base_url\": \"http://SR-Demo-ALB-receiving-123459376.us-east-1.elb.amazonaws.com\",\n                    \"rate\": 100,\n                    \"replay_start_time\": \"2018-08-06T01:00\",\n                    \"replay_end_time\": \"2018-08-06T02:00\",\n                    \"identifier\": \"oss\"\n                }'\n    timezone: US/Pacific\n```\n\n## 3. Install the Serverless framework\n\n```sh\nnpm install -g serverless\nserverless plugin install -n serverless-python-requirements\n```\n\nA more detailed guide here:\nhttps://serverless.com/framework/docs/getting-started/\n\n## 3.5 Set up virtual env (optional)\n\n```sh\npython3 -m venv ~/.virtualenvs/sr-env\nsource ~/.virtualenvs/sr-env/bin/activate\n```\n\n## 4. Deploy to AWS\n\n```\n# Deploy ShadowReader to your AWS account\nserverless deploy --stage dev --region us-east-1\n```\n\nIf you installed Python using Brew, you may run into this error:\n\n```\nFile \"/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py\", line 248, in finalize_options\n  \"must supply either home or prefix/exec-prefix -- not both\")\ndistutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both\n```\n\nRun this to fix it.\n[More details at StackOverflow](https://stackoverflow.com/questions/24257803/distutilsoptionerror-must-supply-either-home-or-prefix-exec-prefix-not-both)\n\n```\n(while in same directory as serverless.yml)\necho '[install]\\nprefix=' \u003e setup.cfg\n```\n\n## Start replaying traffic\n\nNow we will start querying one ALB, which will generate access logs for ShadowReader to replay in real-time.\n\nFind the DNSName for the ALB we will hit.\n\n```\n$ aws elbv2 describe-load-balancers --names SR-Demo-ALB-log-generator --region us-east-1 | grep DNSName\n\u003e\u003e \"DNSName\": \"SR-Demo-ALB-log-generator-1234567.us-east-1.elb.amazonaws.com\"\n```\n\nUse [watch](https://linux.die.net/man/1/watch) to start sending requests to our ALB, this will generate access logs for ShadowReader to replay.\n\n(`brew install watch` on mac)\n\n```\nwatch -n 1 curl http://SR-Demo-ALB-log-generator-1234567.us-east-1.elb.amazonaws.com\n```\n\nYou should now start seeing ALB logs being deposited to the `sr-access-logs` S3 bucket.\n\nIn about 8 minutes, requests sent to SR-Demo-ALB-log-generator will be replayed to SR-Demo-ALB-receiving\n\n## See the results\n\nCheck the CloudWatch ` HTTP fixed response count` metrics for SR-Demo-ALB-receiving. It should be similar to the one for SR-Demo-ALB-log-generator.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/edmunds/shadowreader/blob/master/docs/imgs/live-replay1.png?raw=true\" alt=\"shadow-reader-live-demo-results\" width=\"80%\" height=\"80%\"/\u003e\n\u003c/p\u003e\n\n## Next steps\n\nTo start replaying actual application traffic, try attaching an ECS service or EC2 server to log-generator ALB.\n\n\n## Design\n\nThis diagram details the AWS components Shadow Reader uses and how they interact. More details in the above case study.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/edmunds/shadowreader/blob/master/docs/imgs/shadow-reader-architecture.png?raw=true\" alt=\"shadow-reader-design\" width=\"90%\" height=\"90%\"/\u003e\n\u003c/p\u003e\n\n## Contributing and Deveploment\n\nPlease see the [contributing guide](CONTRIBUTING.md) for more specifics.\n\n## Contact/Support\n\nWhile ShadowReader has been productionalized at Edmunds, the open source version is still under active development and we greatly appreciate any feedback, suggestions, or comments!\n\nEmail me: ysawa@edmunds.com or contact me on Twitter: [@yukisww](https://twitter.com/yukisww)\n\nor use the [Issues](https://github.com/edmunds/shadowreader/issues) page\n\n\n## License\n\nDistributed under the Apache License 2.0. See [`LICENSE`](LICENSE) for more information.\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedmunds%2Fshadowreader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedmunds%2Fshadowreader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedmunds%2Fshadowreader/lists"}