{"id":19435662,"url":"https://github.com/code-kern-ai/refinery-submodule-s3","last_synced_at":"2025-08-26T06:37:40.231Z","repository":{"id":45843361,"uuid":"513891504","full_name":"code-kern-ai/refinery-submodule-s3","owner":"code-kern-ai","description":"S3 related AWS and Minio logic.","archived":false,"fork":false,"pushed_at":"2024-10-18T08:24:34.000Z","size":33,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"dev","last_synced_at":"2025-02-25T06:44:54.721Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.kern.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/code-kern-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-14T12:24:46.000Z","updated_at":"2024-10-18T08:24:36.000Z","dependencies_parsed_at":"2024-05-14T13:29:38.333Z","dependency_job_id":"13d040bb-7245-4cce-8ee2-1b5b8d1b88bf","html_url":"https://github.com/code-kern-ai/refinery-submodule-s3","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/code-kern-ai/refinery-submodule-s3","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-kern-ai%2Frefinery-submodule-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-kern-ai%2Frefinery-submodule-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-kern-ai%2Frefinery-submodule-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-kern-ai%2Frefinery-submodule-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/code-kern-ai","download_url":"https://codeload.github.com/code-kern-ai/refinery-submodule-s3/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/code-kern-ai%2Frefinery-submodule-s3/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272186211,"owners_count":24888333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-26T02:00:07.904Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T15:07:27.199Z","updated_at":"2025-08-26T06:37:40.209Z","avatar_url":"https://github.com/code-kern-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# refinery-submodule-s3\n[![refinery repository](https://uploads-ssl.webflow.com/61e47fafb12bd56b40022a49/62c2f30f935f4d37dc864eeb_Kern%20refinery.png)](https://github.com/code-kern-ai/refinery)\n\nS3 related AWS and Minio logic.\n\n*Caution! This is only meant as a submodule not as a library*\n\nIf you like what we're working on, please leave a ⭐ for [refinery](https://github.com/code-kern-ai/refinery)!\n\n## Usage\n\nImplement by using:\n`git submodule add https://github.com/code-kern-ai/refinery-submodule-s3.git submodules/s3`\n\nThis means the requirements of the project need to include:\n\n```\nboto3\nminio\n```\n\n### Also some os variables are required to access minio:\n\n- `S3_ENDPOINT` --\u003e adress to access (e.g. http://$HOST_IP:7053 -- usually for aws)\n- `S3_ENDPOINT_LOCAL` --\u003e local adress to access (e.g. object-storage:9000 -- usually for minio)\n- `S3_ACCESS_KEY` --\u003e S3 User name\n- `S3_SECRET_KEY` --\u003e S3 Password\n\n### Also some os variables are required to access aws:\nThey are stored in a different set of os variables to be able to migration from minio to aws\n\n- `S3_TARGET ` --\u003e set to AWS if AWS should be used other value or no value at all defaults to minio\n- `S3_AWS_ENDPOINT` --\u003e adress to access\n- `S3_AWS_REGION` --\u003e region e.g. eu-west-1\n- `S3_AWS_ACCESS_KEY` --\u003e AWS User name\n- `S3_AWS_SECRET_KEY ` --\u003e AWS Password\n- `STS_ENDPOINT` --\u003e Security Token Service (STS) \n\n\n## Common submodule logic applies\n[git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules)\n\nSubmodules can be a bit irritating - key insights:\n\n1. Paths need to be relative inside the module (e.g. `from . import constants` )\n2. They are their own repositories so they need to be handled as such (committing etc. for changes inside their own branch)\n3. Cloning needs to be done recursively to get the files\n4. Adding submodules without changing the folder will result in errors because the import of paths with `-` doesn't work\n5. Drone builds appear to work fine\n6. Drone builds need to include the \"collection\" of the submodules data so another step needs to be introduces to the .yml file:\n\n```\n- name: submodules\n  image: alpine/git\n  commands:\n    - git submodule init\n    - 'git config --global url.\"https://github.com/\".insteadOf git@github.com:'\n    - \"git submodule update --recursive\"\n```\n\n**\"remember\" to pull / push for the submodules!!**\n\n## Example Commands for submodules\n\n### Clone reposity with submodules\n\n`git clone \u003cgit@github ...\u003e --recursive`\nExample\n`git clone git@github.com:code-kern-ai/refinery-gateway.git --recursive`\n\n(if you missed the --recursive part or switch to a branch with a new submodule):\n`git submodule update --init`\n\n### Add submodule\n\n(no path specified -\u003e the repository will be a new folder on the top level - **DON'T DO THIS**)\n\n`git submodule add \u003cgit@github ...\u003e snipmate-snippets/snippets/`\n\n(path specified -\u003e this one we usually want since repository names might have some issues)\n\n`git submodule add \u003cgit@github ...\u003e submodules/\u003crepo_name_with_underscores\u003e`\n\n### Working with submodules after initial setup\n\n- Either navigate to the submodule folder and use it like every other git:\n  - `cd submodules/\u003crepo_name_with_underscores\u003e`\n  - `git pull`\n  - `git add .`\n  - `git commit ....`\n  - `git push`\n- Or use submodule command like:\n  - `git submodule foreach git pull`\n  - `git submodule foreach git add .`\n  - `git submodule foreach git commit -m '\u003cmessage\u003e'`\n  - `git submodule foreach git push`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcode-kern-ai%2Frefinery-submodule-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcode-kern-ai%2Frefinery-submodule-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcode-kern-ai%2Frefinery-submodule-s3/lists"}