{"id":18783985,"url":"https://github.com/sparkpost/bouncy-sink","last_synced_at":"2025-04-13T12:27:51.990Z","repository":{"id":47007163,"uuid":"129149978","full_name":"SparkPost/bouncy-sink","owner":"SparkPost","description":"A PMTA-based sink application that does opens, clicks, bounces, OOBs and FBLs","archived":false,"fork":false,"pushed_at":"2022-08-19T16:09:11.000Z","size":1085,"stargazers_count":18,"open_issues_count":0,"forks_count":5,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-13T12:27:44.081Z","etag":null,"topics":["arf","bounce-messages","email","sparkpost"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SparkPost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-11T20:25:27.000Z","updated_at":"2025-04-05T01:50:31.000Z","dependencies_parsed_at":"2022-09-17T14:00:20.193Z","dependency_job_id":null,"html_url":"https://github.com/SparkPost/bouncy-sink","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fbouncy-sink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fbouncy-sink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fbouncy-sink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SparkPost%2Fbouncy-sink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SparkPost","download_url":"https://codeload.github.com/SparkPost/bouncy-sink/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248713538,"owners_count":21149716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arf","bounce-messages","email","sparkpost"],"created_at":"2024-11-07T20:41:30.397Z","updated_at":"2025-04-13T12:27:51.969Z","avatar_url":"https://github.com/SparkPost.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://www.sparkpost.com\"\u003e\u003cimg src=\"https://www.sparkpost.com/sites/default/files/attachments/SparkPost_Logo_2-Color_Gray-Orange_RGB.svg\" width=\"200px\"/\u003e\u003c/a\u003e\n\n[Sign up](https://app.sparkpost.com/join?plan=free-0817?src=Social%20Media\u0026sfdcid=70160000000pqBb\u0026pc=GitHubSignUp\u0026utm_source=github\u0026utm_medium=social-media\u0026utm_campaign=github\u0026utm_content=sign-up) for a SparkPost account and visit our [Developer Hub](https://developers.sparkpost.com) for even more content.\n\n# Bouncy Sink for SparkPost traffic\n\n[![Build Status](https://travis-ci.org/SparkPost/bouncy-sink.svg?branch=master)](https://travis-ci.org/SparkPost/bouncy-sink)\n\nThe Bouncy Sink is a place where you can send your test email to. It simulates some aspects of real-world\nemail user and ISP behaviour, such as in-band and out-of-band [bounces](https://www.sparkpost.com/blog/wild-west-bounce-codes-deliverability/),\n[FBLs](https://www.sparkpost.com/blog/why-feedback-loops-matter/), [opens and clicks](https://www.sparkpost.com/blog/email-user-engagement-metrics/).\n\nPlease note that traffic to the sink **still counts as usage on your account.**\n\n**Why use this?** Sending test traffic to real ISP domains (such as gmail.com, hotmail and so on) that isn't actually\nfor humans to open, such as to \"made up\" addresses that bounce, can quickly damage your [email reputation](https://www.sparkpost.com/blog/email-reputation-matters/). \n\n## Generating traffic toward the sink\n\nTo use the sink, simply send SparkPost traffic to it.\nAny valid method can be used to generate traffic from SparkPost toward the sink.\n\nHere is [a traffic generator](https://github.com/tuck1s/sparkpost-traffic-gen)\nwhich can easily be deployed to Heroku, to generate random traffic through your SparkPost account toward the \"bouncy sink\" domains.\n\n## Recipient Domains\n\nDifferent response behaviours are available, through choice of recipient subdomain.  The localpart of the address can be anything.\n\n|Response Behaviour|Use Recipient Address|\n|-------------|--------------------------|\n|Accept quietly, without opens or clicks|`any@accept.bouncy-sink.trymsys.net`|\n|Out-of-band bounce|`any@oob.bouncy-sink.trymsys.net`|\n|Spam Complaint (ARF format FBL) |`any@fbl.bouncy-sink.trymsys.net`|\n|Accepted and opened at least once|`any@openclick.bouncy-sink.trymsys.net`|\n|Statistical mix of responses|`any@bouncy-sink.trymsys.net`|\n\nThe subdomain part immediately after the `@` is checked, so `@fbl.bouncy-sink.trymsys.net` and `@fbl.fred.wilma.bouncy-sink.trymsys.net`\ntrigger the same behaviour.\n\nOther subdomains, for example `foo.bar.bouncy-sink.trymsys.net` will give the statistical\nmix of responses.\n\nOpen and click tracking requires a valid html part in your mail content, and the relevant tracking options\nto be enabled in your SparkPost account \u0026 transmission. \"Click\" tries to follow all links present in the html part.\n\n### Statistical model\n\nThis is the default setup:\n\n\u003cimg src=\"doc-img/bouncy-sink-statistical-model.svg\"/\u003e\n\nThis can be customised using the .ini file if you are deploying your own bouncy sink instance - see [here](CONFIGURING.md).\n\nClicks are done only if the mail was also opened, so the event sequence can be:\n\n```\n[ Open [Open] [Click [Click] ] ]\n```\n\n## In-band bounces\n\nThis sink uses PMTA's built-in facility to generate in-band bounces with both 4xx (tempfail) and 5xx (permfail) codes on a portion of traffic.\nThe codes are varied at random, and are typical of what a real ISP might send back.\nThe 4xx codes will show up on SparkPost reporting as \"delayed\" mails, and they will be retried.\n\n## Actions on the mail content\n\nFor these actions, messages must have a valid DKIM signature.\n\n### Opens and Clicks\n\nIf an HTML mail part is present, the sink opens (\"renders\") the mail by fetching `\u003cimg .. src=\"..\"\u003e` tags \npresent in the received mail that are served by SparkPost's engagement tracker endpoint.\n\nThe sink clicks links in similar manner by fetching any  `\u003ca .. href=\"..\"\u003e` tags present in the received mail.\n\nThe sink uses an `OPTIONS` http request (which will, conicidentally be rejected by the engagement tracker) to check the server type.\nIt does not actually follow the link redirect, or fetch the whole object.\n\nThe User-Agent is randomly selected from a realistic set of current, popular browsers.\n\n### FBLs (aka Spam Complaints)\n\nThe sink responds to a some mails with an FBL back to SparkPost in ARF format.  The reply is constructed as follows:\n\n- Additional checks below must pass\n- The FBL `From:` header address and `MAIL FROM` is the received mail `To:` header value, which must be present\n- The FBL `To:` header address and `RCPT TO` is derived by looking up the received mail `Return-Path:` MX, according to the below table\n- The `X-MSFBL` header is populated from the received mail\n- The ARF-format FBL mail is attempted directly over SMTP to the relevant MX (simply choosing the first MX, if there is more than one)\n- SMTP error responses are logged\n\n|Service |MX |fblTo |\n|--------|---|------|\n|SparkPost|smtp.sparkpostmail.com|`fbl@sparkpostmail.com`\n|SparkPost Enterprise|*tenant*.mail.e.sparkpost.com|`fbl@tenant.mail.e.sparkpost.com`\n|SparkPost EU|smtp.eu.sparkpostmail.com|`fbl@eu.sparkpostmail.com`\n\nThe FBLs show up as `spam_complaint` events in SparkPost.\n\n### Out-of-band bounces\n\nOOB bounce replies are constructed as follows:\n\n- Additional checks below must pass\n- The OOB `From:` header address and `MAIL FROM` is taken from the received mail `To:` header value\n- The OOB `To:` header address and `RCPT TO` is taken from the received mail `Return-Path:` header\n- The OOB mail is attempted directly over SMTP to the relevant MX (choosing the first MX if there is more than one)\n- Endpoint error responses are logged\n\nThe OOBs show up as `out_of_band` events in SparkPost.\n\n### Additional checks on OOB and FBL actions\n\nTo reduce the effect of bad actors trying to use the sink to mount a [backscatter spam](https://en.wikipedia.org/wiki/Backscatter_(email)) attack, \nthe direct OOB and FBL actions also require SPF to pass (so we know the originating IP is valid for the domain).\n\nAll OOB and FBL actions require the `Return-Path:` MX to resolve back to a known SparkPost endpoint.\n\n### Internal application logfile\n\nAll actions are logged, with configurable logfile retention and midnight rotation.\n\n### Redis \u0026 Web reporting\n\nThe main script increments counters in `redis` which can be queried directly using `redis-cli`:\n\n```\n$ redis-cli\n127.0.0.1:6379\u003e keys *\n 1) \"consume-mail:0:int_fbl_sent\"\n 2) \"consume-mail:0:startedRunning\"\n 3) \"consume-mail:0:int_open_again\"\n 4) \"consume-mail:0:int_open\"\n 5) \"consume-mail:0:int_fail_dkim\"\n 6) \"consume-mail:0:int_oob_sent\"\n 7) \"consume-mail:0:int_accept\"\n 8) \"consume-mail:0:int_click_again\"\n 9) \"consume-mail:0:int_total_messages\"\n10) \"consume-mail:0:int_click\"\n11) \"consume-mail:0:int_open_url_not_sparkpost\"\n127.0.0.1:6379\u003e get consume-mail:0:int_total_messages\n\"13315977\"\n```\n`webReporter.py` is a simple Flask-based reporting app to present these counters.\n`gunicorn` is started on private port number 8888 on reboot by `crontab` which calls script `starting-gun.sh`.\n\nOn your client, open an SSH connection with port 8888 tunneled:\n```\nssh -i ##YourPrivateKeyHere## -L 8888:localhost:8888 YourUser@example.com\n```\n\nOpen page `localhost:8888` on your client browser:\n\n\u003cimg src=\"doc-img/bouncy-sink-private-web-monitor.png\"/\u003e\n\nYou can also fetch the stats in JSON format:\n```\n$ curl -s localhost:8888/json | jq .\n{\n  \"fbl_sent\": 2607,\n  \"fail_dkim\": 7,\n  \"oob_sent\": 133559,\n  \"startedRunning\": \"2018-05-31T00:30:58+00:00\",\n  \"open_again\": 2640376,\n  \"click\": 2642225,\n  \"total_messages\": 13340946,\n  \"open\": 7922352,\n  \"open_url_not_sparkpost\": 1,\n  \"accept\": 5282421,\n  \"click_again\": 792616\n}\n```\n\n### SparkPost suppression list cleaning\n\nBounces will populate your suppression list. It's good practice to purge those entries relating to the sink domains when you've finished.\n[Here is a tool](https://www.sparkpost.com/blog/suppression-list-python/) that you can use to clean up.\n\n### Manually restarting the consume-mail task\n\nIdentify the current process number and kill the task:\n```\nps aux | grep consume\nsudo kill _processID_\n```\n\nRestart the task:\n```\nsudo src/consume-mail.py /var/spool/mail/inbound/ -f \u003e/dev/null 2\u003e\u00261 \u0026\n```\n\n### User Agent values (on opens and clicks)\n\n`consume-mail.ini` specifies a file that should contain the user-agent strings (in .CSV format), similar to those\nfrom [here](https://developers.whatismybrowser.com/useragents/explore/software_type_specific/web-browser/).\n\n```\nSoftware,OS,Layout engine,Popularity,\n\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\",Chrome 60,Windows,Blink,Very common\n:\netc\n```\nOnly the \"Software\" column of this file is used. A user-agent is picked at random for each mail, prior to open and click processing.\n\n## See Also\n\n[Internal configuration details](CONFIGURING.md)\n\n[Simple \"accept\" sink built in to SparkPost](https://www.sparkpost.com/docs/faq/using-sink-server/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparkpost%2Fbouncy-sink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsparkpost%2Fbouncy-sink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsparkpost%2Fbouncy-sink/lists"}