{"id":37403448,"url":"https://github.com/oslokommune/okdata-data-uploader","last_synced_at":"2026-01-16T05:48:23.223Z","repository":{"id":37964020,"uuid":"379235243","full_name":"oslokommune/okdata-data-uploader","owner":"oslokommune","description":"AWS Lambda function for generating presigned URLs and form fields used to upload files to S3","archived":false,"fork":false,"pushed_at":"2026-01-12T08:25:02.000Z","size":1072,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-01-12T18:07:57.305Z","etag":null,"topics":["dataplatform"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oslokommune.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-06-22T10:49:38.000Z","updated_at":"2026-01-12T08:25:05.000Z","dependencies_parsed_at":"2024-04-12T05:59:30.043Z","dependency_job_id":"056ae295-a768-4e21-a626-558fc69f18b0","html_url":"https://github.com/oslokommune/okdata-data-uploader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oslokommune/okdata-data-uploader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fokdata-data-uploader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fokdata-data-uploader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fokdata-data-uploader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fokdata-data-uploader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oslokommune","download_url":"https://codeload.github.com/oslokommune/okdata-data-uploader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fokdata-data-uploader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28477418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T03:13:13.607Z","status":"ssl_error","status_checked_at":"2026-01-16T03:11:47.863Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataplatform"],"created_at":"2026-01-16T05:48:23.153Z","updated_at":"2026-01-16T05:48:23.210Z","avatar_url":"https://github.com/oslokommune.png","language":"Python","readme":"# Data uploader\n\nREST API for creating presigned URLs and form fields that can be used to POST a file to S3.\n\n## Setup\n\n1. [Install Serverless Framework](https://serverless.com/framework/docs/getting-started/)\n2. Install plugins:\n```\nmake init\n```\n\n### Setup for development\n\nGrab yourself a virtualenv:\n```\npython -m venv .venv\n```\n\nTo activate the virtualenv run:\n```\nsource ./venv/bin/activate\n```\n\nInside the virtualenv you can install packages locally:\n```\npip install -r requirements.txt\n```\n\nTo exit from a virtualenv run:\n```\ndeactivate\n```\n\n## Running tests\n\n```\n$ make test\n```\n\nTests are run using [tox](https://pypi.org/project/tox/).\n\n## Build image\n\n```\n$ make build\n```\nCreates an image with tag: okdata/okdata-uploader, used to test locally if a build in ci/cd fails.\n\n## Deploy\n\nDeploy to both dev and prod is automatic via GitHub Actions on push to\n`main`. You can alternatively deploy from local machine with: `make deploy` or\n`make deploy-prod`.\n\n## Code formatting\n\n```\nmake format\n```\n\nRuns [black](https://black.readthedocs.io/en/stable/) to format python code.\n\n## Upload size\n\nA single PUT can be up to 5GB for S3 signed URLs (and is then our current limitation), over that and a multi-part upload must be created\n\n## TODO\n\n - Revisit the upload flow\n   - Today: frontend checks dataset/schema and creates edition, then POSTs file\n   - Alternative: frontend POSTs filename/metadata, backend checks dataset/schema\n     - Alt 1: return signed s3 url, frontend POSTs, (new) backend waits for S3 event and then creates edition (where should metadata be stored in the meantime?)\n     - Alt 2: create edition, return s3 url\n - Create script to get signed URLs for multipart, and script to upload these parts and combine them\n   - https://github.com/sandyghai/AWS-S3-Multipart-Upload-Using-Presigned-Url\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foslokommune%2Fokdata-data-uploader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foslokommune%2Fokdata-data-uploader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foslokommune%2Fokdata-data-uploader/lists"}