{"id":22632467,"url":"https://github.com/uploadcare/migro","last_synced_at":"2025-06-15T07:04:52.819Z","repository":{"id":22004461,"uuid":"92167226","full_name":"uploadcare/migro","owner":"uploadcare","description":"Uploadcare migration tool.","archived":false,"fork":false,"pushed_at":"2025-06-03T20:18:11.000Z","size":251,"stargazers_count":11,"open_issues_count":7,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-04T05:13:16.871Z","etag":null,"topics":["cdn","file-upload","image-processing","migration-tool","python","python3"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uploadcare.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-05-23T11:52:59.000Z","updated_at":"2025-06-03T20:17:05.000Z","dependencies_parsed_at":"2024-04-17T14:45:49.830Z","dependency_job_id":"b09d82ec-8bd9-4a72-aa02-490313f4a8df","html_url":"https://github.com/uploadcare/migro","commit_stats":{"total_commits":146,"total_committers":11,"mean_commits":"13.272727272727273","dds":0.5616438356164384,"last_synced_commit":"9bff9e5b1df443dc7c19e971df4d8e64304b1959"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/uploadcare/migro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uploadcare%2Fmigro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uploadcare%2Fmigro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uploadcare%2Fmigro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uploadcare%2Fmigro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uploadcare","download_url":"https://codeload.github.com/uploadcare/migro/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uploadcare%2Fmigro/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259935601,"owners_count":22934386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdn","file-upload","image-processing","migration-tool","python","python3"],"created_at":"2024-12-09T02:17:41.213Z","updated_at":"2025-06-15T07:04:52.798Z","avatar_url":"https://github.com/uploadcare.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"================================\nMIGRO: Uploadcare Migration Tool\n================================\n\n\nMigro assists in migrating files to Uploadcare from AWS S3 and other cloud\nfile management services such as Filestack, Cloudinary, etc.\nYou can also migrate from additional services by providing URLs for file import.\n\nMigro utilizes a local SQLite database to store migration statuses to prevent duplicates,\nthereby avoiding re-uploading files in case of an interruption or failure.\n\n\nHow migration works\n-------------------\n\nMigro does not directly download any files. Instead, it utilizes\nthe `Uploading API`_, specifically the ``From URL`` method_.\nThe migration process is straightforward: you either provide a list of file\nURLs or set your AWS S3 credentials, and those files are uploaded to your Uploadcare project.\n\nUpon completion, you will receive a listing of all uploaded files.\nYou will also receive details on the status of each processed file and any errors that may have occurred.\n\n\nInstallation\n------------\n\nThis utility requires Python version 3.8 or later.\n\nIn order to install ``migro``, run:\n\n.. code-block:: console\n\n  $ pip install uploadcare-migro\n\n\nGet started\n-----------\n\nTo begin using Migro, you must have your Uploadcare project `public key`_.\n\nOptionally, you can store the public key and other necessary credentials in environment variables.\nThis approach is beneficial if you must migrate many files, allowing you to perform multiple migrations efficiently.\n\n.. code-block:: console\n\n  $ migro init \u003cOPTIONS\u003e\n\nAfter running the initialization command, your credentials will be saved in a local ``.env``\nfile, allowing you to use Migro without specifying them in the command line each time.\nOnce the ``.env`` file is created, you can edit it manually if needed.\n\n.. code-block:: console\n\n  $ migro init --uc_public_key \u003cPUBLIC_KEY\u003e --uc_secret_key \u003cSECRET_KEY\u003e\n\nIf you run the command twice or more times, the tool will update the existing\ncredentials or add new ones. For instance, if you run the following command:\n\n.. code-block:: console\n\n  $ migro init --uc_secret_key \u003cSECRET_KEY\u003e\n\nThe tool will update the secret key but leave the public key unchanged.\n\nDifferent credentials are needed depending on the source of files you want to migrate.\nRefer to the necessary options for each source below.\n\n\nMigrating files\n---------------\n\nMigro supports migration from the following sources:\n\n- AWS S3: The tool scans the bucket, generates temporary signed URLs, and migrates all files.\n- File with URLs: The tool reads a file containing URLs and migrates all files listed.\n\nEach migration source requires the following arguments:\n\n``\u003cPUBLIC_KEY\u003e`` — your Uploadcare project `public key`_.\n\n``[\u003cSECRET_KEY\u003e]`` — your Uploadcare project secret key.\nThis is optional and required only if the signed uploads feature is enabled in your project.\n\nAnd the following options:\n\n.. code-block::\n\n  --upload_base_url TEXT            Base URL for uploads.  [default:\n                                    https://upload.uploadcare.com/]\n\n  --upload_timeout FLOAT            Number of seconds to wait till the file will be\n                                    processed by `from_url` upload.  [default: 30]\n\n  --concurrent_uploads INTEGER      Maximum number of upload requests running in\n                                    'parallel'.  [default: 20]\n\n  --status_check_interval FLOAT     Number of seconds in between status check\n                                    requests.\n\nEach option can be preset using the `migro init` command.\n\n\nUsage with AWS S3\n-----------------\n\nHow it works:\n  1. Migro verifies the credentials provided and checks if the bucket policy is correct.\n  2. The tool then scans the bucket and generates temporary signed URLs for all files.\n  3. Migro proceeds to upload all files to Uploadcare.\n\n\nSet policy for a bucket\n~~~~~~~~~~~~~~~~~~~~~~~\n\nTo ensure proper functionality, set the following minimal permissions for your AWS S3 bucket policy:\n\n.. code-block::\n\n    {\n        \"Version\": \"2012-10-17\",\n        \"Statement\": [\n            {\n                \"Principal\": \"*\",\n                \"Effect\": \"Allow\",\n                \"Action\": [\n                    \"s3:GetObject\",\n                    \"s3:ListBucket\"\n                ],\n                \"Resource\": [\n                    \"arn:aws:s3:::\u003cYOUR BUCKET NAME\u003e\",\n                    \"arn:aws:s3:::\u003cYOUR BUCKET NAME\u003e/*\"\n                ]\n            }\n        ]\n    }\n\nRemember to replace \u003cYOUR BUCKET NAME\u003e with your actual bucket name.\n\nTo initiate the migration, execute the following command:\n\n.. code-block:: console\n\n    $ migro s3 \u003cBUCKET_NAME\u003e \u003cPUBLIC_KEY\u003e [\u003cSECRET_KEY\u003e] [\u003cOPTIONS\u003e]\n\nFor example:\n\n.. code-block:: console\n\n    $ migro s3 \u003cBUCKET_NAME\u003e \u003cPUBLIC_KEY\u003e --s3_access_key_id \u003cACCESS_KEY_ID\u003e --s3_secret_access_key \u003cSECRET_ACCESS_KEY\u003e --s3_region \u003cREGION\u003e\n\n\nOptions:\n\n.. code-block::\n\n\n  -h, --help                        Show this help and quit.\n\n  --s3_access_key_id STRING         AWS Access Key ID for accessing the S3 bucket.\n\n  --s3_secret_access_key STRING     AWS Secret Access Key for accessing the S3 bucket.\n\n  --s3_region STRING                AWS region where the S3 bucket is located.\n\nEach option can be set beforehand using the `migro init` command.\n\nNote:\n    Utilizing ``boto3``, Migro attempts to use the\n    `default AWS credentials \u003chttps://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials\u003e`_\n    if they are not specified during the initialization step or via command line.\n\n\nUsage with file list\n--------------------\n\nTo migrate files using a list of URLs, execute the following command:\n\n.. code-block:: console\n\n    $ migro urls \u003cINPUT_FILE\u003e [\u003cPUBLIC_KEY\u003e] [\u003cSECRET_KEY\u003e]\n\nWhere:\n\n``\u003cINPUT_FILE\u003e`` — path to a text file containing a list of file URLs\nto be uploaded to your Uploadcare project.\n\nOptions:\n\n.. code-block::\n\n  -h, --help                  Show this help and quit.\n\n\nResults file\n------------\n\nOnce the migration is complete, you'll receive a ``.csv`` file containing the results,\nwhich will be located in the logs folder.\n\n.. code-block::\n\n    Path                , File Size, Uploadcare UUID                       , Status  , Error\n    ------------------- , ---------, ------------------------------------- , ------- , ----------------------------------------------------------------------\n    kittens.jpg         , 3478134  , 0897e21a-7b7f-4037-95ec-9b70cbdf6d30  , uploaded,\n    raccoons.jpg        , 2063823  , 14af9b5a-5388-4f48-a38b-61380e8c7332  , uploaded,\n    invalid_format.csv  , 339898   ,                                       , error   , \"File validation error: Uploading of these file types is not allowed.\"\n\n\nThe output file format is straightforward. Each line represents an input file, with five columns separated by commas.\n\nColumn Details:\n * The first column contains the input file URL or S3 key.\n * The second column displays the file size in bytes.\n * The third column holds the uploaded file UUID or remains empty if the file was not uploaded.\n * The fourth column indicates the status of the file, which can be \"uploaded\" or \"error\".\n * The fifth column provides an error message if the file was not uploaded.\n\n\nExamples\n--------\n\nHere's how you set up the tool for the first time:\n\n.. code-block:: console\n\n    $ migro init --uc_public_key \u003cPUBLIC_KEY\u003e --s3_access_key_id \u003cS3_ACCESS_KEY\u003e --s3_secret_access_key \u003cS3_SECRET_KEY\u003e --s3_bucket_name \u003cS3_BUCKET_NAME\u003e --s3_region \u003cS3_REGION\u003e\n\n.. code-block::\n\n    Configuration saved successfully to /Users/username/path/to/migro/.env\n\nand run the utility:\n\n.. code-block:: console\n\n    $ migro s3\n\nAnd that's what you get:\n\n.. code-block::\n\n    Checking the credentials...\n    Credentials are correct.\n    Collecting files...\n    Starting upload...\n\n\n    Upload progress: 100%|████████████████████████████████████████████████| 6/6 [00:03\u003c00:00,  1.74s/file]\n    File uploading has been finished!\n    Uploaded files: 5\n    Failed files: 1\n    Check the results in \"/Users/username/path/to/migro/logs/Attempt 1 - 2024-04-23 17-13-38 - s3.csv\"\n    Thanks for your interest in Uploadcare.\n    Hit us up at help@uploadcare.com in case of any questions.\n\n\nAfter the migration\n-------------------\n\nOnce the migration is complete, execute the ``migro drop`` command to\nremove the `.env` file containing credentials, clear the local database, and the logs folder.\n\n.. code-block:: console\n\n    $ migro drop\n\n\nNote for Windows users\n----------------------\n\nCurrently, there is an issue with terminating the program using CTRL+C on Windows.\nAs a result, the program cannot be terminated correctly using this method.\n\nThis issue stems from platform-dependent behavior in the Python programming language.\n\n\nAlternatives\n------------\n\nYou can use our libs_\nto migrate your files from any source.\n\n.. _Uploading API: https://uploadcare.com/documentation/upload/\n.. _method: https://uploadcare.com/documentation/upload/#from-url\n.. _public key: https://uploadcare.com/documentation/keys/\n.. _libs: https://uploadcare.com/documentation/libs/\n\n\nNeed help?\n----------\n\nDo you need to migrate files from another service? Feel free to create an issue.\nAlternatively, you can reach us at help@uploadcare.com.\nWe'll be happy to assist you with the migration.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuploadcare%2Fmigro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuploadcare%2Fmigro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuploadcare%2Fmigro/lists"}