{"id":31556321,"url":"https://github.com/having-fun-serverless/backup-twitter","last_synced_at":"2025-10-04T22:52:36.403Z","repository":{"id":140835536,"uuid":"562165566","full_name":"having-fun-serverless/backup-twitter","owner":"having-fun-serverless","description":"Backup your twitter account on a daily basis","archived":false,"fork":false,"pushed_at":"2022-11-06T07:46:25.000Z","size":1528,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-05T18:39:53.778Z","etag":null,"topics":["aws","kinesis-firehose","lambda","lambda-extensions","sam"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/having-fun-serverless.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-11-05T14:08:30.000Z","updated_at":"2024-04-05T18:39:56.001Z","dependencies_parsed_at":"2024-04-05T18:50:10.904Z","dependency_job_id":null,"html_url":"https://github.com/having-fun-serverless/backup-twitter","commit_stats":null,"previous_names":["having-fun-serverless/backup-twitter"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/having-fun-serverless/backup-twitter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/having-fun-serverless%2Fbackup-twitter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/having-fun-serverless%2Fbackup-twitter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/having-fun-serverless%2Fbackup-twitter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/having-fun-serverless%2Fbackup-twitter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/having-fun-serverless","download_url":"https://codeload.github.com/having-fun-serverless/backup-twitter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/having-fun-serverless%2Fbackup-twitter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278386109,"owners_count":25978109,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","kinesis-firehose","lambda","lambda-extensions","sam"],"created_at":"2025-10-04T22:52:34.293Z","updated_at":"2025-10-04T22:52:36.396Z","avatar_url":"https://github.com/having-fun-serverless.png","language":"Python","readme":"[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n\u003c!-- PROJECT LOGO --\u003e\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"images/logo.png\" alt=\"The following logo was created by Dall-E 2 with the following description _3d render of the twitter logo being uploaded to an aluminum bucket_\"\u003e\n\n\u003ch3 align=\"center\"\u003eBacking up Twitter\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    Backing up Twitter tweets became a hot topic in the last couple of days. I've decided to create a simple POC using Serverless components.\n    \u003cbr /\u003e\n    \u003cbr /\u003e\n    \u003ca href=\"https://github.com/aws-hebrew-book/backup-twitter/issues\"\u003eReport Bug\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/aws-hebrew-book/backup-twitter/issues\"\u003eRequest Feature\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#high-level-architecture\"\u003eHigh level architecture\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#prerequisites\"\u003ePrerequisites\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#testing\"\u003eTesting\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#monitoring\"\u003eMonitoring\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#landmarks\"\u003eLandmarks\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#dynamic-partitioning\"\u003eDynamic Partitioning\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#parameters-and-secret-store-extension\"\u003eParameters and Secret store extension\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#batch-processing\"\u003eBatch processing\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contact\"\u003eContact\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#logo\"\u003eLogo\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## About The Project\nTaking a real-world issue and trying to tackle it using Serverless components is an excellent way to learn about Serverless. The following Serverless application uses a scheduled task that runs once per day and backs up all the Twitter handles that you've configured.\n\nAfter deployment you can configure the relevant Twitter handles you want to backup by changing the value of a parameter called TwitterAccounts found under the [Parameters Store](https://us-east-1.console.aws.amazon.com/systems-manager/parameters?region=us-east-1)..\n\nEach day the Twitter handles are backed up at 10 AM UTC time. Due to Twitter API restrictions, only the previous day's tweets are being backup up. You can find your tweets under S3.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"images/s3-bucket.png\" alt=\"Tweets under S3\"\u003e\n\u003c/div\u003e\n\n### High level architecture\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"images/twitter-backup.png\" alt=\"Architecture diagram\"\u003e\n\u003c/div\u003e\n\n1. We have an evenbridge as a cron scheduler.\n2. A Lambda is being triggered every day at 10 AM UTC.\n3. In order to pull the configuration, the [AWS Parameters and Secrets Lambda Extension](https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html) is used.\n4. Each Twitter handle is being pushed as a separate message into SQS.\n5. Using the [batch processing utility](https://awslabs.github.io/aws-lambda-powertools-python/2.1.0/utilities/batch/) in the Python Lambda Power tools. Each message is processed\n6. For each handle, we are making a Twitter API call to get the Twitter account id and then the tweets from the last day\n7. The Twitter bearer token, which is required for authentication, is pulled from the AWS secret manager using [AWS Parameters and Secrets Lambda Extension](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html)\n8. Tweets are written into AWS Kinesis Firehose which writes them into S3. \n9. [Dynamic partitioning](https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html) is used, therefore the files created under S3 have handle prefixes.\n\n\n## Getting started\n### Prerequisites\n* Make sure your machine is ready to work with [AWS SAM](https://aws.amazon.com/serverless/sam/)\n* Create a [twitter API account](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api)\n* And save locally your `Bearer Token`, you should receive it at the end of the registration process\n\n### Installation\n* Clone this repository.\n* Run `sam build` and then `sam deploy --guided`. Accept the default values, except for \n    * _Parameter TwitterBearerToken_ - paste the token value you've recevied from Twitter. \n    * _Parameter TwitterAccountsValues_ - Choose the Twitter handles you want to back up. Of course, you can use the default here.\n\nAfter the deployment is complete you can always change the Twitter handles you want to back up by changing the value found under https://us-east-1.console.aws.amazon.com/systems-manager/parameters?region=us-east-1\n\n### Testing\n* You can test the application manully by executing the `ScheduleBackupFunction` Lambda directly from the console.\n\n### Monitoring\nMonitoring is done by using [Lumigo](https://platform.lumigo.io/auth/signup)\n\n## Landmarks\n### Dynamic Partitioning \n* IAM Policy for allowing the FH to write into S3 - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L109\n* The FH configuration which includes the Dynamic Partitioning configuration - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L138\n\n### Parameters and Secret store extension\n* Create relevant parms in SAM's yaml - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L167\n* Attached extension to the Lambda - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L40\n* Pull parameters via code - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/shared/parameters_utils.py#L7\n\n### Batch processing\n* Use SQS batch processor - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/pull_twitter_stream/app.py#L44\n## Contributing\n\nContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.\n\nIf you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag \"enhancement\".\nDon't forget to give the project a star! Thanks again!\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n\n\u003c!-- LICENSE --\u003e\n## License\n\nDistributed under the Apache License Version 2.0 License. See `LICENSE` for more information.\n\n\u003c!-- CONTACT --\u003e\n## Contact\n\nEfi Merdler-Kravitz - [@TServerless](https://twitter.com/TServerless)\n\n\n\n## Logo\nThe project's logo was created by Dall-E 2 with the following description _3d render of the twitter logo being uploaded to an aluminum bucket_\n\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaving-fun-serverless%2Fbackup-twitter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhaving-fun-serverless%2Fbackup-twitter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaving-fun-serverless%2Fbackup-twitter/lists"}