Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/yousafkhamza/backup-to-s3-pyscript
It's a simple python script to take backup of directories (compressing) then the same to move your mentioned S3 bucket with the help of AWS IAM User.
https://github.com/yousafkhamza/backup-to-s3-pyscript
aws-s3 backup-script boto3 pip python-script python3
Last synced: 2 months ago
JSON representation
It's a simple python script to take backup of directories (compressing) then the same to move your mentioned S3 bucket with the help of AWS IAM User.
- Host: GitHub
- URL: https://github.com/yousafkhamza/backup-to-s3-pyscript
- Owner: yousafkhamza
- Created: 2021-07-28T23:24:38.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-07-30T04:48:45.000Z (over 3 years ago)
- Last Synced: 2023-09-20T19:52:37.669Z (over 1 year ago)
- Topics: aws-s3, backup-script, boto3, pip, python-script, python3
- Language: Python
- Homepage:
- Size: 16.6 KB
- Stars: 3
- Watchers: 2
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Directory Backup Moved to S3 (Pyscript)
[](https://travis-ci.org/joemccann/dillinger)
---
## Description
Here it's a python script that needs to use this script simply create a directory backup and moved that compressed backup file to a configured S3 bucket with the help of python script and AWS IAM User with S3 full access. So, let's roll down.
----
## Feature
- Easy to configure for anyone
- It generates the directory compressed format
- Which one we entered directory converted to a backup/compressed form to S3
- All the steps I have provided including AWS IAM User and S3 Bucket Creation
---
## Modules used
- [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)
- [os](https://docs.python.org/3/library/os.html)
- [posixpath](https://www.oreilly.com/library/view/python-standard-library/0596000960/ch13s04.html)
- [sys](https://docs.python.org/3/library/sys.html)
- [tarfile](https://docs.python.org/3/library/tarfile.html)
----
## Pre-Requests
- Basic Knowledge of python
- Basic Knowledge of AWS IAM, S3 service
- Need to change your IAM user creds and bucket name at var.py
### IAM User Creation steps (_with screenshot_)
1. _log into your AWS account as a root user and go to IAM user_
2. _goto Access Managment >> Users_
3. _Click Add User_ (_top right corner_)
4. _Enter any username as you like and Choose "Programmatic access" >> Click Next Permissions_
5. _Set Permissions >> Select "Attach existing policies directly" >> Choose "AmazonS3FullAccess" >> Click Next Tags_
6. _Add Tags(Optional)_ >> _Enter a key and value as you like either you can leave as blank_
7. _Review your user details and click "Create User"_
8. _Store your credentials to your local_
_Reference URL_:: _IAM User creation [article](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html)_
----
### S3 Bucket Creation (_with screenshot_)
1. _Go to S3 > Click Create Bucket_
2. _Any bucket name as you wish and then please enable versioning (that you upload same file multiple times or modified versions uploaded that the S3 stored as a version bases like Git)_
3. _Click create bucket_
> Reference URL:: Creating S3 bucket please use this [doc](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) and you can secure your bucket with IAM user using S3 [bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-iam-policies.html)
----
### Pre-Requested (Dependency packages)
```sh
yum install -y git
yum install -y python3
yum install -y python3-pip
```
### How to get
```sh
git clone https://github.com/yousafkhamza/backup-to-s3-pyscript.git
cd backup-to-s3-pyscript
pip3 install -r requirements.txt
```
> Change your creds and bucket name in at var.py file
Command to run the script::
```
python3 to-S3.py Python/test
# Pyhon/test <------------ Directory to take backup and move the backup to S3
```
----
## Output be like
```sh
$ $ python3 to-S3.py Python/httpd
Start to Upload that the httpd.tar.gz your S3 Bucket yousaf-test
Backup successfully uploaded to your S3 bucket yousaf-test
```
## View of S3 bucket
_Screenshot_
----
## Behind the code
_vim to-S3.py_
```sh
import boto3
import tarfile
import os
import sys
import posixpath
import var
from boto3.s3.transfer import S3Transfer
directory = sys.argv[1]
dirname = os.path.split(directory)[-1]
if directory.endswith ('/'):
print ('Please remove / after the directory path you have entered')
else:
if posixpath.isdir(directory):
tarname = '/tmp/{}.tar.gz'.format(dirname)
tar = tarfile.open(tarname,'w:gz')
tar.add(directory)
tar.close()
print('Start to Upload that the', dirname+'.tar.gz', 'your S3 Bucket', var.BUCKET_NAME)
# S3 uploading started
client = boto3.client('s3', aws_access_key_id=var.AWS_ACCESS_KEY_ID,aws_secret_access_key=var.AWS_SECRET_ACCESS_KEY)
transfer = S3Transfer(client)
transfer.upload_file(tarname, var.BUCKET_NAME, 'backup/{}.tar.gz'.format(dirname))
print('Backup succesfully uploaded to your S3 bucket', var.BUCKET_NAME)
# Remove temporary backup from local
os.remove(tarname)
else:
print('Please enter a valid directory path')
```
var.py
```sh
AWS_ACCESS_KEY_ID = 'AKTBRI2N5IAT3ND' <-------------- Replace your acess key
AWS_SECRET_ACCESS_KEY = 'asUNmMPrC99HoiiQPjehetFtVsPv' <--------- Replace your secret key
BUCKET_NAME = 'yousaf-test' <----------- Replace your bucket name
```
----
## Conclusion
It's a simple python script to take backup of directories (compressing) then the same to move your mentioned S3 bucket with the help of AWS IAM User. this script may be helpful who had face issues moving backups to S3 so it might be useful for cloud engineers.
### ⚙️ Connect with Me
