{"id":18718812,"url":"https://github.com/denis-source/file_backup","last_synced_at":"2025-11-10T16:30:21.372Z","repository":{"id":136600361,"uuid":"502344016","full_name":"Denis-Source/file_backup","owner":"Denis-Source","description":"Command line application to copy folder content from one location to another.","archived":false,"fork":false,"pushed_at":"2022-11-19T04:50:19.000Z","size":69,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-28T11:12:17.388Z","etag":null,"topics":["backup","command-line","filesystem","google-drive-api-v3","python","sftp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Denis-Source.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-11T12:35:46.000Z","updated_at":"2022-12-05T04:56:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"9b39b5a1-938f-48eb-a0d9-6e7337c45857","html_url":"https://github.com/Denis-Source/file_backup","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Denis-Source%2Ffile_backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Denis-Source%2Ffile_backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Denis-Source%2Ffile_backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Denis-Source%2Ffile_backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Denis-Source","download_url":"https://codeload.github.com/Denis-Source/file_backup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239581759,"owners_count":19662960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","command-line","filesystem","google-drive-api-v3","python","sftp"],"created_at":"2024-11-07T13:23:09.799Z","updated_at":"2025-11-10T16:30:21.324Z","avatar_url":"https://github.com/Denis-Source.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# File Backup\n\n## About\n\nCommand line application to copy the folder content from one location to another.\n\nLocation is not restricted to the local file system as there is an ability to choose both input and output handlers.\n\n\u003e All the handlers can be used in an arbitrary order.\n\nApplication has the following handlers:\n\n- \"vanilla\" file system;\n- remote SFTP server;\n- Google Drive;\n- DropBox.\n\n\u003e SFTP handler requires server configurations (IP address, key file location), GDrive handler requires credentials for the V3 API and Dropbox requires authorization token.\n* * *\n\n## Used libraries\n\n- `googleapiclient`\n- `pysftp`\n- `os`\n- `dropbox`\n\n`LocalHandler` mainly uses built-in `os` library as it provides tools for the folder creation, file system inromation retriving.\n`SFTPHandler` uses `pysftp` library that is very similar to the `os` library\n`GDriveHandler` uses [GoogleDrive V3 API](https://developers.google.com/drive/api/quickstart/quickstarts-overview).\n`DropBoxHandler` uses [DropBox Python SDK for API v2](https://www.dropbox.com/developers/documentation/python#overview)\n* * *\n\n## Installation\n\n```sh\ngit clone https://github.com/Denis-Source/file_backup.git\n```\n\n```sh\npython3 -m venv env/\nsource env/bin/activate\n```\n\n```sh\npython main.py -i {input_folder} -o {output_folder}\n```\n\n* * *\n\n## Showcase\n\nThe console application has the following signature:\n\n```sh\nusage: main.py [-h] [-ih INPUT_HANDLER] [-oh OUTPUT_HANDLER] [-i INPUT] -o OUTPUT [-v]\n\nFile Backup command line App\noptional arguments:\n  -h, --help                 show this help message and exit\n  -ih, --input_handler       type of a handler as an input\n  -oh, --output_handler\t\t\t\ttype of a handler as an output\n  -i, --input                input location\n  -o, --output               output location\n  -v, --validation           use validators defined in the configs\n```\n\n\u003e All the examples below were provided using [WSL](https://docs.microsoft.com/ru-ru/windows/wsl/install) to avoid mixing of UNIX and Windows file system path formats.\n\n***\n\n### Local Handler\nConsidering we have the following file tree structure:\n```sh\n~/samples$ tree\n.\n├── BuzzBuzz_edited.wav\n├── SX_Door_origin.wav\n├── SX_Phone_origin.wav\n└── edited\n    └── Wow_Edited.wav\n\n1 directory, 4 files\n```\n\n\u003e By Default if input or output handlers are not specified, the local handler will be used\n\nWe can copy the folder content from one location to another, using `-i` and `-o` keys:\n```sh\npython3 main.py -i ~/samples/ -o /mnt/c/file_backup/samples\n```\n\nThe folders content were copied to the specified output folder:\n```sh\ntree /mnt/c/file_backup/samples/\n/mnt/c/file_backup/samples/\n├── BuzzBuzz_edited.wav\n├── SX_Door_origin.wav\n├── SX_Phone_origin.wav\n└── edited\n    └── Wow_Edited.wav\n\n1 directory, 4 files\n```\n\nThe logging output:\n```sh\n2022-08-27 17:13:04,560 INFO    app             Backuping files from /home/den/samples/ using local handler to /mnt/c/file_backup/samples using local handler not using validators\n2022-08-27 17:13:04,560 INFO    local           Got information about a folder at /home/den/samples/\n2022-08-27 17:13:04,561 INFO    local           Got information about a file at /home/den/samples//SX_Door_origin.wav\n2022-08-27 17:13:04,561 INFO    local           Got information about a folder at /home/den/samples//edited\n2022-08-27 17:13:04,561 INFO    local           Got information about a file at /home/den/samples//SX_Phone_origin.wav\n2022-08-27 17:13:04,561 INFO    local           Got information about a file at /home/den/samples//BuzzBuzz_edited.wav\n2022-08-27 17:13:04,561 INFO    local           Got information about a file at /home/den/samples//edited/Wow_Edited.wav\n2022-08-27 17:13:04,561 INFO    local           Creating folder at /mnt/c/file_backup/samples\n2022-08-27 17:13:04,566 INFO    local           Got information about a folder at /mnt/c/file_backup/samples\n2022-08-27 17:13:04,567 INFO    local           Сopying File at /home/den/samples//SX_Door_origin.wav\n2022-08-27 17:13:04,569 INFO    local           File at /home/den/samples//SX_Door_origin.wav is read\n2022-08-27 17:13:04,571 INFO    local           Created new file at /mnt/c/file_backup/samples/SX_Door_origin.wav\n2022-08-27 17:13:04,572 INFO    local           Сopying Folder at /home/den/samples//edited\n2022-08-27 17:13:04,572 INFO    local           Creating folder at /mnt/c/file_backup/samples/edited\n2022-08-27 17:13:04,579 INFO    local           Got information about a folder at /mnt/c/file_backup/samples/edited\n2022-08-27 17:13:04,580 INFO    local           Сopying File at /home/den/samples//edited/Wow_Edited.wav\n2022-08-27 17:13:04,581 INFO    local           File at /home/den/samples//edited/Wow_Edited.wav is read\n2022-08-27 17:13:04,584 INFO    local           Created new file at /mnt/c/file_backup/samples/edited/Wow_Edited.wav\n2022-08-27 17:13:04,586 INFO    local           Сopying File at /home/den/samples//SX_Phone_origin.wav\n2022-08-27 17:13:04,588 INFO    local           File at /home/den/samples//SX_Phone_origin.wav is read\n2022-08-27 17:13:04,590 INFO    local           Created new file at /mnt/c/file_backup/samples/SX_Phone_origin.wav\n2022-08-27 17:13:04,591 INFO    local           Сopying File at /home/den/samples//BuzzBuzz_edited.wav\n2022-08-27 17:13:04,594 INFO    local           File at /home/den/samples//BuzzBuzz_edited.wav is read\n2022-08-27 17:13:04,596 INFO    local           Created new file at /mnt/c/file_backup/samples/BuzzBuzz_edited.wav\n2022-08-27 17:13:04,598 INFO    app             Done in 0:00:00!\n```\n***\n\n### SFTP Handler\nWe can specify `sftp` both as the input or output handlers using `-ih` and `-oh`:\n\n\u003e To use SFTP it is required to configure the following constants in the `config.py` file:  `HOST`,  `USERNAME`,  `KEY LOCATION`.\n\nTo upload the previous local `samples/` folder to the SFTP server, we should use the following command:\n```sh\npython main.py -i ~/samples/ -o /backups/samples -oh sftp\n```\nProgram logging:\n```sh\n2022-08-27 17:34:14,438 INFO    app             Backuping files from /home/den/samples/ using local handler to backups/samples using local handler not using validators\n2022-08-27 17:34:14,439 INFO    local           Got information about a folder at /home/den/samples/\n2022-08-27 17:34:14,439 INFO    local           Got information about a file at /home/den/samples//SX_Door_origin.wav\n2022-08-27 17:34:14,439 INFO    local           Got information about a folder at /home/den/samples//edited\n2022-08-27 17:34:14,439 INFO    local           Got information about a file at /home/den/samples//SX_Phone_origin.wav\n2022-08-27 17:34:14,439 INFO    local           Got information about a file at /home/den/samples//BuzzBuzz_edited.wav\n2022-08-27 17:34:14,439 INFO    local           Got information about a file at /home/den/samples//edited/Wow_Edited.wav\n2022-08-27 17:34:14,440 INFO    sftp            Creating folder at backups/samples\n2022-08-27 17:34:14,771 INFO    sftp            Got information about a folder at backups/samples\n2022-08-27 17:34:14,771 INFO    sftp            Сopying File at /home/den/samples//SX_Door_origin.wav\n2022-08-27 17:34:14,790 INFO    local           File at /home/den/samples//SX_Door_origin.wav is read\n2022-08-27 17:34:14,901 INFO    sftp            Created new file at backups/samples/SX_Door_origin.wav\n2022-08-27 17:34:14,901 INFO    sftp            Сopying Folder at /home/den/samples//edited\n2022-08-27 17:34:14,901 INFO    sftp            Creating folder at backups/samples/edited\n2022-08-27 17:34:15,036 INFO    sftp            Got information about a folder at backups/samples/edited\n2022-08-27 17:34:15,037 INFO    sftp            Сopying File at /home/den/samples//edited/Wow_Edited.wav\n2022-08-27 17:34:15,055 INFO    local           File at /home/den/samples//edited/Wow_Edited.wav is read\n2022-08-27 17:34:15,176 INFO    sftp            Created new file at backups/samples/edited/Wow_Edited.wav\n2022-08-27 17:34:15,177 INFO    sftp            Сopying File at /home/den/samples//SX_Phone_origin.wav\n2022-08-27 17:34:15,197 INFO    local           File at /home/den/samples//SX_Phone_origin.wav is read\n2022-08-27 17:34:15,259 INFO    sftp            Created new file at backups/samples/SX_Phone_origin.wav\n2022-08-27 17:34:15,259 INFO    sftp            Сopying File at /home/den/samples//BuzzBuzz_edited.wav\n2022-08-27 17:34:15,279 INFO    local           File at /home/den/samples//BuzzBuzz_edited.wav is read\n2022-08-27 17:34:15,459 INFO    sftp            Created new file at backups/samples/BuzzBuzz_edited.wav\n2022-08-27 17:34:15,460 INFO    app             Done in 0:00:01!\n```\n\nThe remote server has the following file structure:\n```sh\nroot@vaua0073687:~/backups# tree ~/backups/\n/root/backups/\n└── samples\n    ├── BuzzBuzz_edited.wav\n    ├── edited\n    │   └── Wow_Edited.wav\n    ├── SX_Door_origin.wav\n    └── SX_Phone_origin.wav\n\n2 directories, 4 files\n```\n\nWe can also swap the handlers:\n```sh\npython main.py -o ~/samples/ -i backups/samples -ih sftp -oh local\n```\n\n\u003e If `-ih` or `-oh` keys are not specified, the `local` handler will be used\n***\n\n### Google Drive Handler\n\nWe can use Google Drive Handler to upload or download folders.\n\n\u003eTo use GDrive handler, we should provide `secrets.json` file.\n\nTo upload `samples/` folder we can use the following command:\n\n```sh\npython main.py -i ~/samples/ -o backups/samples -oh gdrive\n```\n\nProgram output:\n```sh\n2022-08-27 17:43:17,764 INFO    app             Backuping files from /home/den/samples/ using local handler to backups/samples using local handler not using validators\n2022-08-27 17:43:17,764 INFO    local           Got information about a folder at /home/den/samples/\n2022-08-27 17:43:17,764 INFO    local           Got information about a file at /home/den/samples//SX_Door_origin.wav\n2022-08-27 17:43:17,765 INFO    local           Got information about a folder at /home/den/samples//edited\n2022-08-27 17:43:17,765 INFO    local           Got information about a file at /home/den/samples//SX_Phone_origin.wav\n2022-08-27 17:43:17,765 INFO    local           Got information about a file at /home/den/samples//BuzzBuzz_edited.wav\n2022-08-27 17:43:17,765 INFO    local           Got information about a file at /home/den/samples//edited/Wow_Edited.wav\n2022-08-27 17:43:17,765 INFO    gdrive          Creating folder at backups/samples\n2022-08-27 17:43:18,461 INFO    gdrive          Found records with name backups in folder 0AH3CXGph3HmOUk9PVA: 0\n2022-08-27 17:43:19,363 INFO    gdrive          Created folder backups at\n2022-08-27 17:43:19,640 INFO    gdrive          Found records with name samples in folder 1N9LMu8J4rQrLNwTYWcWh67FAF6a4Y3gs: 0\n2022-08-27 17:43:20,254 INFO    gdrive          Created folder samples at /backups\n2022-08-27 17:43:20,254 INFO    gdrive          Сopying File at /home/den/samples//SX_Door_origin.wav\n2022-08-27 17:43:20,595 INFO    gdrive          Found records with name SX_Door_origin.wav in folder 1SgfUNnq8JVNNvJL5c1c9rMseiZ0oVpd2: 0\n2022-08-27 17:43:20,595 INFO    local           File at /home/den/samples//SX_Door_origin.wav is read\n2022-08-27 17:43:21,692 INFO    gdrive          Сopying Folder at /home/den/samples//edited\n2022-08-27 17:43:21,692 INFO    gdrive          Creating folder at backups/samples/edited\n2022-08-27 17:43:22,304 INFO    gdrive          Found records with name backups in folder 0AH3CXGph3HmOUk9PVA: 1\n2022-08-27 17:43:22,304 INFO    gdrive          Folder backups is already exists at\n2022-08-27 17:43:22,569 INFO    gdrive          Found records with name samples in folder 1N9LMu8J4rQrLNwTYWcWh67FAF6a4Y3gs: 1\n2022-08-27 17:43:22,569 INFO    gdrive          Folder samples is already exists at /backups\n2022-08-27 17:43:22,839 INFO    gdrive          Found records with name edited in folder 1SgfUNnq8JVNNvJL5c1c9rMseiZ0oVpd2: 0\n2022-08-27 17:43:23,370 INFO    gdrive          Created folder edited at /backups/samples\n2022-08-27 17:43:23,370 INFO    gdrive          Сopying File at /home/den/samples//edited/Wow_Edited.wav\n2022-08-27 17:43:23,628 INFO    gdrive          Found records with name Wow_Edited.wav in folder 174fISaVT9lqcSKnHXLMTEoaTHifMiBXt: 0\n2022-08-27 17:43:23,629 INFO    local           File at /home/den/samples//edited/Wow_Edited.wav is read\n2022-08-27 17:43:24,820 INFO    gdrive          Сopying File at /home/den/samples//SX_Phone_origin.wav\n2022-08-27 17:43:25,085 INFO    gdrive          Found records with name SX_Phone_origin.wav in folder 1SgfUNnq8JVNNvJL5c1c9rMseiZ0oVpd2: 0\n2022-08-27 17:43:25,086 INFO    local           File at /home/den/samples//SX_Phone_origin.wav is read\n2022-08-27 17:43:26,264 INFO    gdrive          Сopying File at /home/den/samples//BuzzBuzz_edited.wav\n2022-08-27 17:43:26,498 INFO    gdrive          Found records with name BuzzBuzz_edited.wav in folder 1SgfUNnq8JVNNvJL5c1c9rMseiZ0oVpd2: 0\n2022-08-27 17:43:26,498 INFO    local           File at /home/den/samples//BuzzBuzz_edited.wav is read\n2022-08-27 17:43:27,634 INFO    app             Done in 0:00:10!\n```\n\nGoogle Drive folder contents:\n![google drive folder](https://i.ibb.co/RSj33QV/image.png)\n***\n#### DropBox\nWe can also use DropBox Handler to copy the previously mentioned samples from the Google Drive to DropBox.\n\n\u003e In order to function, the API token should be generated from application console with the needed permissions. The token itself should be specified in the configs.\n\nTo copy `samples/` folder we can use the following command:\n```sh\npython main.py  -i backups -ih gdrive -o backups -oh dropbox\n```\n\nProgram output:\n```\n2022-11-19 06:47:12,604 INFO    app             Backuping files from backups using gdrive handler to backups using gdrive handler not using validators\n2022-11-19 06:47:12,928 INFO    gdrive          Got id about the root folder: 0AH3CXGph3HmOUk9PVA\n2022-11-19 06:47:14,925 INFO    dropbox         deleted already created folder\n2022-11-19 06:47:16,355 INFO    gdrive          Request to read file backups/samples/BuzzBuzz_edited.wav is successful\n2022-11-19 06:47:16,355 INFO    gdrive          Downloading file in memory\n2022-11-19 06:47:17,130 INFO    gdrive          Download is successful\n2022-11-19 06:47:18,584 INFO    dropbox         Uploaded file backups/samples/BuzzBuzz_edited.wav\n2022-11-19 06:47:18,584 INFO    gdrive          Request to read file backups/samples/SX_Phone_origin.wav is successful\n2022-11-19 06:47:18,584 INFO    gdrive          Downloading file in memory\n2022-11-19 06:47:19,142 INFO    gdrive          Download is successful\n2022-11-19 06:47:20,109 INFO    dropbox         Uploaded file backups/samples/SX_Phone_origin.wav\n2022-11-19 06:47:20,972 INFO    gdrive          Request to read file backups/samples/edited/Wow_Edited.wav is successful\n2022-11-19 06:47:20,972 INFO    gdrive          Downloading file in memory\n2022-11-19 06:47:21,600 INFO    gdrive          Download is successful\n2022-11-19 06:47:23,160 INFO    dropbox         Uploaded file backups/samples/edited/Wow_Edited.wav\n2022-11-19 06:47:23,168 INFO    gdrive          Request to read file backups/samples/SX_Door_origin.wav is successful\n2022-11-19 06:47:23,168 INFO    gdrive          Downloading file in memory\n2022-11-19 06:47:23,817 INFO    gdrive          Download is successful\n2022-11-19 06:47:24,866 INFO    dropbox         Uploaded file backups/samples/SX_Door_origin.wav\n2022-11-19 06:47:24,866 INFO    app             Done in 0:00:12!\n```\n\nDropBox folder contents:\n\n![dropbox_example](https://i.ibb.co/Jd7DnhH/image.png)\n\n\n### Validators\nIf we try to copy the contents of the source folder:\n```sh\npython main.py -o /mnt/c/file_backup/source_code\n```\n\nWe will see that the folder has ~14k files:\n```sh\ntree /mnt/c/file_backup/source_code/ | wc -l\n14037\n```\n\n\u003eIf `-i` key is not specified, it will copy the current folder.\n\nWe can filter out unnecessary files (`.pyc` binaries, `.env` files, etc) by setting file\\folder filtering using `config.py` fle using regular expressions:\n\n```py\nEXCLUDED_NAMES = [\n    r\"^__\",\n    r\"^env\",\n    r\"^venv\",\n    r\"^\\.\",\n    \"^lib$\",\n]\nEXCLUDED_FORMATS = [\n    \"pyc\",\n    \"pyd\",\n]\n```\n\nTo use them, we simply provide `-v` key:\n```sh\npython main.py -o /mnt/c/file_backup/source_code -v\n```\n\nNow the output folder contains only 22 files:\n```sh\ntree /mnt/c/file_backup/source_code/ | wc -l\n22\n```\n***\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenis-source%2Ffile_backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenis-source%2Ffile_backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenis-source%2Ffile_backup/lists"}