{"id":18947646,"url":"https://github.com/yousafkhamza/backup-to-s3-bashscript","last_synced_at":"2025-04-15T23:30:23.625Z","repository":{"id":195965529,"uuid":"391949530","full_name":"yousafkhamza/backup-to-s3-bashscript","owner":"yousafkhamza","description":"Here it's a bash script that needs to use this script simply create a directory backup and moved that compressed backup file to a configured S3 bucket with the help of bash script and AWS IAM User with S3 full access. I have added a new feature the script will be suitable for the ubuntu/RedHat repository and the script will install the dependencies as itself. So, let's roll down","archived":false,"fork":false,"pushed_at":"2021-08-05T11:58:08.000Z","size":10,"stargazers_count":3,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-09-20T19:52:36.898Z","etag":null,"topics":["backup-script","bash-script","devops","linux","s3-bucket"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yousafkhamza.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-08-02T12:57:29.000Z","updated_at":"2023-09-20T19:52:44.721Z","dependencies_parsed_at":"2023-09-20T19:52:43.333Z","dependency_job_id":"42d6f24e-10a7-41b7-beae-e042d389726e","html_url":"https://github.com/yousafkhamza/backup-to-s3-bashscript","commit_stats":null,"previous_names":["yousafkhamza/backup-to-s3-bashscript"],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yousafkhamza%2Fbackup-to-s3-bashscript","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yousafkhamza%2Fbackup-to-s3-bashscript/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yousafkhamza%2Fbackup-to-s3-bashscript/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yousafkhamza%2Fbackup-to-s3-bashscript/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yousafkhamza","download_url":"https://codeload.github.com/yousafkhamza/backup-to-s3-bashscript/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223688710,"owners_count":17186298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup-script","bash-script","devops","linux","s3-bucket"],"created_at":"2024-11-08T13:10:49.285Z","updated_at":"2024-11-08T13:10:52.334Z","avatar_url":"https://github.com/yousafkhamza.png","language":"Shell","readme":"# Directory Backup Moved to S3 (BashScript)\n[![Build](https://travis-ci.org/joemccann/dillinger.svg?branch=master)](https://travis-ci.org/joemccann/dillinger)\n\n---\n## Description\nHere it's a bash script that needs to use this script simply create a directory backup and moved that compressed backup file to a configured S3 bucket with the help of bash script and AWS IAM User with S3 full access. I have added a new feature the script will be suitable for the ubuntu/RedHat repository and the script will install the dependencies as itself. So, let's roll down\n\n----\n## Feature\n- Easy to configure for anyone \n- It generates the directory compressed format\n- Which one we entered directory converted to a backup/compressed form to S3\n- All the steps I have provided including AWS IAM User and S3 Bucket Creation\n- Includes AWSCLI installation which you used debian or redhat repository the script will findout which you used and installed the same\n- All the values append your answers while running the script\n\n## Cons\n- This script is not suitable for cronjob and it needs manual job but it just ask some kind of questions other the all things append your answer\n\n----\n## Pre-Requests\n- Basic Knowledge of bash \n- Basic Knowledge of AWS IAM, S3 service\n- Need to change your IAM user creds and then please enter the same while script running time\n\n### IAM User Creation steps (_with screenshot_)\n1. _log into your AWS account as a root user and go to IAM user_\n2. _goto Access Managment \u003e\u003e Users_\n![alt_txt](https://i.ibb.co/Y7kzZmN/IAM-1.png)\n3. _Click Add User_ (_top right corner_)\n![alt_txt](https://i.ibb.co/wW38xvR/IAM-2.png)\n4. _Enter any username as you like and Choose \"Programmatic access\" \u003e\u003e Click Next Permissions_\n![alt_txt](https://i.ibb.co/TrCbpBh/IAM-3.png)\n5. _Set Permissions \u003e\u003e Select \"Attach existing policies directly\" \u003e\u003e Choose \"AmazonS3FullAccess\" \u003e\u003e Click Next Tags_\n![alt_txt](https://i.ibb.co/8BHhwmc/IAM-4.png)\n6. _Add Tags(Optional)_ \u003e\u003e _Enter a key and value as you like either you can leave as blank_\n![alt_txt](https://i.ibb.co/QQb9svy/IAM-5.png)\n7. _Review your user details and click \"Create User\"_\n![alt_txt](https://i.ibb.co/RcxL770/IAM-6.png)\n8. _Store your credentials to your local_\n![alt_txt](https://i.ibb.co/nPVWcXZ/IAM-7.png)\n\n_Reference URL_:: _IAM User creation [article](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html)_\n\n----\n### S3 Bucket Creation (_with screenshot_)\n1. _Go to S3 \u003e  Click Create Bucket_\n![alt_txt](https://i.ibb.co/bLky3Rb/S3-1.png)\n2. _Any bucket name as you wish and then please enable versioning (that you upload same file multiple times or modified versions uploaded that the S3 stored as a version bases like Git)_\n![alt_txt](https://i.ibb.co/kXCQJfQ/S3-3.png)\n3. _Click create bucket_\n\n![alt_txt](https://i.ibb.co/chwztWB/S3-4.png)\n\n\u003e Reference URL:: Creating S3 bucket please use this [doc](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) and you can secure your bucket with IAM user using S3 [bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-iam-policies.html)\n\n----\n### Pre-Requested (Dependency packages)\n```sh\nyum install -y git\n```\n\n### How to get\n```sh\ngit clone https://github.com/yousafkhamza/backup-to-s3-bashscript.git\ncd backup-to-s3-bashscript\nchmod +x backup-to-S3.sh\n```\n\u003e Change your creds and bucket name in at var.py file\n\nCommand to run the script::\n```\n[root@ip-172-31-10-180 backup-to-s3-bashscript]# ./backup-to-S3.sh\n# --------------------------- or --------------------------------- #\n[root@ip-172-31-10-180 backup-to-s3-bashscript]# bash backup-to-S3.sh\n```\n\n----\n## Output be like\n```sh\n[root@ip-172-31-10-180 backup-to-s3-bashscript]# ./backup-to-S3.sh\nAWS Package is installed.....\n\nStart the script...\n\nPlease configure your IAM user creds on the server\n\nAWS Access Key ID [None]:                                # \u003c---------  Enter your access_key here.\nAWS Secret Access Key [None]:                           # \u003c--------- Enter your secret key here.\nDefault region name [None]: ap-south-1                 # \u003c--------- Enter your region which you needs\nDefault output format [None]: json                    # \u003c----------  Default output format is json\n\nLet's roll to create your backup to S3\n\nPlease crosscheck the credentials are given below\n\naws_access_key_id = \u003c Your accesss_key confirmation on the part of script \u003e\naws_secret_access_key = \u003c Your secret_key confirmation on the part of script \u003e\n\nDo you need to reconfigure the same [Y/N]: n\n\nLet's roll to create your backup to S3\n\nEnter your directory path(The directory will be compressed as a tar.gz file): /root/Python\n\nTaking the directory path to your local as a temporary.........\n\ntar: Removing leading `/' from member names\n/root/Python/\n/root/Python/test/\n/root/Python/test/file.txt\n/root/Python/test/two.txt\n\nLocal Backup Takes successfully,,,,\n\nEnter your Bucket Name (S3 Destination): yousaf-test\nBackup Moving to S3.......\nupload: ../../tmp/Python-05082021.tar.gz to s3://yousaf-test/backup/Python-05082021.tar.gz\n\nRemoving local backup.....\nLocal backup removes successfully\n```\n\n## Output be like (ScreenShot)\n![alt_txt](https://i.ibb.co/swxt5S6/out.png)\n\n## View of S3 bucket\n![alt_txt](https://i.ibb.co/KNd0jcQ/Screenshot-56.png)\n\n----\n## Behind the code\n_vim backup-to-S3.sh_\n```\n#!/bin/bash\n\nUbuntu=$(which apt \u003e 2\u0026\u003e1; echo $?)\nRedhat=$(which yum \u003e 2\u0026\u003e1; echo $?)\nAWS=$(which aws \u003e 2\u0026\u003e1; echo $?)\ndate=$(date \"+%d%m%Y\")\n\nif [[ \"$AWS\" -eq 0 ]]; then\n\techo \"AWS Package is installed.....\"\n\techo \"\"\n\techo \"Start the script...\"\n\t\n\tif [[ $(ls ~/.aws/credentials \u003e 2\u0026\u003e1; echo $?) -ne 0 ]]; then\n\t\techo \"\"\n\t\techo \"Please configure your IAM user creds on the server\"\n\t\techo \"\"\n\t\taws configure --profile backups3\n\t\techo \"\"\n\t\techo \"Let's roll to create your backup to S3\"\n\tfi\nelif [[ \"$Ubuntu\" -eq 0 ]]; then\n\techo \"AWS Package is not installed on your debian distro. Installing AWS package....\"\n\tsleep 1\n\techo \"\"\n\tsudo apt install -y awscli\n\techo \"Please configure your IAM user creds on the server\"\n    echo \"\"\n    aws configure --profile backups3\n    echo \"\"\n    echo \"Let's roll to create your backup to S3\"\nelif [[ \"$Redhat\" -eq 0 ]]; then\n\techo \"AWS Package is not installed on your ReadHat distro. Installing AWS package....\"\n\tsleep 1\n\techo \"\"\n\tsudo yum install -y awscli\n\techo \"Please configure your IAM user creds on the server\"\n    echo \"\"\n    aws configure --profile backups3\n    echo \"\"\n    echo \"Let's roll to create your backup to S3\"\nelse\n\techo \"Please install AWS Package..... and retry the same\"\n\texit 1\nfi\n\nisInFileA=$(cat ~/.aws/credentials | grep -c \"backups3\")\nisInFileB=$(cat ~/.aws/config | grep -c \"backups3\")\nCredInServer=~/.aws/credentials\n# AWS Configuration on the server\nif [ -f $CredInServer ] \u0026\u0026 [ \"$isInFileA\" -eq 1 ] \u0026\u0026 [ \"$isInFileB\" -eq 1 ]; then\n    echo \"\"\n\techo \"Please crosscheck the credentials are given below\"\n    echo \"\"\n    cat $CredInServer | grep -A 2 \"backups3\" | tail -n2\n    echo \"\"\n    read -p \"Do you need to reconfigure the same [Y/N]: \" con1\n    if [[ \"$con1\" =~ ^([yY][eE][sS]|[yY])+$ ]]; then\n        aws configure --profile backups3\n    else\n        echo \"\"\n        echo \"Let's roll to create your backup to S3\"\n    fi\nelse\n\techo \"\"\n    echo \"Please configure your IAM user creds on the server\"\n    echo \"\"\n    aws configure --profile backups3\n    echo \"\"\n    echo \"Let's roll to create your backup to S3\"\nfi\n\nisInFileA=$(cat ~/.aws/credentials | grep -c \"backups3\")\nisInFileB=$(cat ~/.aws/config | grep -c \"backups3\")\nCredInServer=~/.aws/credentials\n# Taking a local Backup before S3 upload\nif [ -f $CredInServer ] \u0026\u0026 [ \"$isInFileA\" -eq 1 ] \u0026\u0026 [ \"$isInFileB\" -eq 1 ]; then\n\techo \"\"\n\tread -p \"Enter your directory path(The directory will be compressed as a tar.gz file): \" path\n\tBackupName=$(echo $path | awk -F \"/\" '{print $NF}')\t\t\t\n\tif [ -z $path ]; then \n\t\techo \"Please specify a a absalute directory path\"\n\t\texit 1\n\telse\n\t\tif [[ \"$path\" == */ ]]; then\n\t\t\techo \"Your entered directory path endswith /, so please remove the last / symbol\"\n\t\telse\n\t\t\tif [ -d $path ]; then\n\t\t\t\techo \"\"\n\t\t\t\techo \"Taking the directory path to your local as a temporary.........\"\n\t\t\t\techo \"\"\n\t\t\t\tsleep 2\n\t\t\t\trm -f /tmp/$BackupName-*.tar.gz \n\t\t\t\ttar -cvf /tmp/$BackupName-$date.tar.gz $path/\n\t\t\t\techo \"\"\n\t\t\t\techo \"Local Backup Takes successfully,,,,\"\n\t\t\t\t# Backup Copy to S3\n\t\t\t\techo \"\"\n\t\t\t\tread -p \"Enter your Bucket Name (S3 Destination): \" bucket\n\t\t\t\tif [ -z bucket ]; then\n\t\t\t\t\techo \"Please specify a Bucket name\"\n\t\t\t\t\texit 1\n\t\t\t\telse\t\t\t\t\n\t\t\t\t\tif [ $(aws s3 --profile backups3 ls | grep -w \"$bucket\" \u003e 2\u0026\u003e1; echo $?) -eq 0 ]; then\n\t\t\t\t\t\techo \"Backup Moving to S3.......\"\n\t\t\t\t\t\taws s3 --profile backups3 cp \"/tmp/$BackupName-$date.tar.gz\" s3://$bucket/backup/\n\t\t\t\t\t\techo \"\"\n\t\t\t\t\t\techo \"Removing local backup.....\"\n\t\t\t\t\t\trm -f /tmp/$BackupName-*.tar.gz \n\t\t\t\t\t\techo \"Local backup removes successfully\"\n\t\t\t\t\telse\n\t\t\t\t\t\techo \"\"\n\t\t\t\t\t\techo \"Please enter a valid bucket name\"\n\t\t\t\t\t\texit 1\n\t\t\t\t\tfi\n\t\t\t\tfi\n\t\t\telse\n\t\t\t\techo \"\"\n\t\t\t\techo \"Enter a valid directory absalute path\"\n\t\t\tfi\n\t\tfi\n\tfi\nfi\n```\n\n----\n## Conclusion\nIt's a simple bash script to take backup of directories (compressing) then the same to move your mentioned S3 bucket with the help of AWS IAM User. this script may be helpful who had face issues moving backups to S3 so it might be useful for cloud/linux/DevOps engineers.  \n\n### ⚙️ Connect with Me \n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"mailto:yousaf.k.hamza@gmail.com\"\u003e\u003cimg src=\"https://img.shields.io/badge/Gmail-D14836?style=for-the-badge\u0026logo=gmail\u0026logoColor=white\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://www.linkedin.com/in/yousafkhamza\"\u003e\u003cimg src=\"https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge\u0026logo=linkedin\u0026logoColor=white\"/\u003e\u003c/a\u003e \n\u003ca href=\"https://www.instagram.com/yousafkhamza\"\u003e\u003cimg src=\"https://img.shields.io/badge/Instagram-E4405F?style=for-the-badge\u0026logo=instagram\u0026logoColor=white\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://wa.me/%2B917736720639?text=This%20message%20from%20GitHub.\"\u003e\u003cimg src=\"https://img.shields.io/badge/WhatsApp-25D366?style=for-the-badge\u0026logo=whatsapp\u0026logoColor=white\"/\u003e\u003c/a\u003e\u003cbr /\u003e\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyousafkhamza%2Fbackup-to-s3-bashscript","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyousafkhamza%2Fbackup-to-s3-bashscript","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyousafkhamza%2Fbackup-to-s3-bashscript/lists"}