https://github.com/having-fun-serverless/backup-twitter
Backup your twitter account on a daily basis
https://github.com/having-fun-serverless/backup-twitter
aws kinesis-firehose lambda lambda-extensions sam
Last synced: 3 months ago
JSON representation
Backup your twitter account on a daily basis
- Host: GitHub
- URL: https://github.com/having-fun-serverless/backup-twitter
- Owner: having-fun-serverless
- License: apache-2.0
- Created: 2022-11-05T14:08:30.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2022-11-06T07:46:25.000Z (about 3 years ago)
- Last Synced: 2024-04-05T18:39:53.778Z (over 1 year ago)
- Topics: aws, kinesis-firehose, lambda, lambda-extensions, sam
- Language: Python
- Homepage:
- Size: 1.46 MB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://opensource.org/licenses/Apache-2.0)
Backing up Twitter
Backing up Twitter tweets became a hot topic in the last couple of days. I've decided to create a simple POC using Serverless components.
Report Bug
·
Request Feature
Table of Contents
## About The Project
Taking a real-world issue and trying to tackle it using Serverless components is an excellent way to learn about Serverless. The following Serverless application uses a scheduled task that runs once per day and backs up all the Twitter handles that you've configured.
After deployment you can configure the relevant Twitter handles you want to backup by changing the value of a parameter called TwitterAccounts found under the [Parameters Store](https://us-east-1.console.aws.amazon.com/systems-manager/parameters?region=us-east-1)..
Each day the Twitter handles are backed up at 10 AM UTC time. Due to Twitter API restrictions, only the previous day's tweets are being backup up. You can find your tweets under S3.
### High level architecture
1. We have an evenbridge as a cron scheduler.
2. A Lambda is being triggered every day at 10 AM UTC.
3. In order to pull the configuration, the [AWS Parameters and Secrets Lambda Extension](https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html) is used.
4. Each Twitter handle is being pushed as a separate message into SQS.
5. Using the [batch processing utility](https://awslabs.github.io/aws-lambda-powertools-python/2.1.0/utilities/batch/) in the Python Lambda Power tools. Each message is processed
6. For each handle, we are making a Twitter API call to get the Twitter account id and then the tweets from the last day
7. The Twitter bearer token, which is required for authentication, is pulled from the AWS secret manager using [AWS Parameters and Secrets Lambda Extension](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html)
8. Tweets are written into AWS Kinesis Firehose which writes them into S3.
9. [Dynamic partitioning](https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html) is used, therefore the files created under S3 have handle prefixes.
## Getting started
### Prerequisites
* Make sure your machine is ready to work with [AWS SAM](https://aws.amazon.com/serverless/sam/)
* Create a [twitter API account](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api)
* And save locally your `Bearer Token`, you should receive it at the end of the registration process
### Installation
* Clone this repository.
* Run `sam build` and then `sam deploy --guided`. Accept the default values, except for
* _Parameter TwitterBearerToken_ - paste the token value you've recevied from Twitter.
* _Parameter TwitterAccountsValues_ - Choose the Twitter handles you want to back up. Of course, you can use the default here.
After the deployment is complete you can always change the Twitter handles you want to back up by changing the value found under https://us-east-1.console.aws.amazon.com/systems-manager/parameters?region=us-east-1
### Testing
* You can test the application manully by executing the `ScheduleBackupFunction` Lambda directly from the console.
### Monitoring
Monitoring is done by using [Lumigo](https://platform.lumigo.io/auth/signup)
## Landmarks
### Dynamic Partitioning
* IAM Policy for allowing the FH to write into S3 - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L109
* The FH configuration which includes the Dynamic Partitioning configuration - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L138
### Parameters and Secret store extension
* Create relevant parms in SAM's yaml - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L167
* Attached extension to the Lambda - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L40
* Pull parameters via code - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/shared/parameters_utils.py#L7
### Batch processing
* Use SQS batch processor - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/pull_twitter_stream/app.py#L44
## Contributing
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!
1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
## License
Distributed under the Apache License Version 2.0 License. See `LICENSE` for more information.
## Contact
Efi Merdler-Kravitz - [@TServerless](https://twitter.com/TServerless)
## Logo
The project's logo was created by Dall-E 2 with the following description _3d render of the twitter logo being uploaded to an aluminum bucket_