{"id":17030629,"url":"https://github.com/vsoch/django-nginx-upload","last_synced_at":"2025-04-12T12:12:07.596Z","repository":{"id":36320877,"uuid":"137612664","full_name":"vsoch/django-nginx-upload","owner":"vsoch","description":"example of using nginx upload module with Django!","archived":false,"fork":false,"pushed_at":"2024-02-07T19:40:49.000Z","size":149,"stargazers_count":78,"open_issues_count":5,"forks_count":17,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-12T12:12:02.857Z","etag":null,"topics":["django","nginx","nginx-upload","upload"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vsoch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-16T21:53:29.000Z","updated_at":"2024-03-05T13:37:29.000Z","dependencies_parsed_at":"2025-01-13T16:31:25.212Z","dependency_job_id":"47dc9b3d-ca67-4f97-8f1a-de7b5e6a452c","html_url":"https://github.com/vsoch/django-nginx-upload","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsoch%2Fdjango-nginx-upload","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsoch%2Fdjango-nginx-upload/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsoch%2Fdjango-nginx-upload/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsoch%2Fdjango-nginx-upload/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vsoch","download_url":"https://codeload.github.com/vsoch/django-nginx-upload/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248565078,"owners_count":21125417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django","nginx","nginx-upload","upload"],"created_at":"2024-10-14T08:07:42.567Z","updated_at":"2025-04-12T12:12:07.577Z","avatar_url":"https://github.com/vsoch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Django Nginx Upload\n\nThe [nginx upload](https://www.nginx.com/resources/wiki/modules/upload/) module offers a nice way to bypass the Django application for file uploads.  This is an example Django application that \ndemonstrates a basic setup with a custom nginx image and a webserver, both as Docker images for\ndeployment with docker compose.\n\n![img/upload.png](img/upload.png)\n\nIf this work is useful to you, please reference the DOI\n\n[![DOI](https://zenodo.org/badge/137612664.svg)](https://zenodo.org/badge/latestdoi/137612664)\n\n## Rationale\nI am creating this toy example after much frustration trying to implement (something) that would be able to handle uploads of [Singularity](https://singularityware.github.io) containers. Specifically, \nwhen you set up an nginx web server and you have file uploads, you usually need to set a maximum body\nsize so the server doesn't poop out:\n\n```bash\n  client_max_body_size 8000M;\n  client_body_buffer_size 8000M;\n  client_body_timeout 120;\n```\n\nIt's a game we all play - when you first create the application, you forget to set this\nentirely. At some point you get a suite of errors related to \"this file is too big,\" you do a google\nsearch, find that you can define the max body (and buffer and timeout) and add to your configuration.\nBut if you have REALLY big files, you can still have the configuration above, and have the upload fail.\nThen you start looking for other solutions, and here we are!\n\nThere were several issues ([#108](https://github.com/singularityhub/sregistry/issues/108), [#109](https://github.com/singularityhub/sregistry/issues/109), [#123](https://github.com/singularityhub/sregistry/issues/123)) with discussion that ranged from using [chunked upload](https://github.com/juliomalegria/django-chunked-upload), and then this nginx module, and in retrospect you can read the issues to\nsee how I stumbled trying to find the best approach. I was so determined to make this easier for all future dinosaur programmers that I started working on it early one Saturday morning, and had an entire new thing (this repository) about 8 hours later. This is my dinosaur programmer superpower!\n\n\n### Chunked Upload\n\nThe idea of a chunked upload speaks for itself - you break a file into chunks, and upload them separately instead of the entire thing in one go. This module to implement chunked uploading for Django worked beautifully in the browser, and if you need a good \"out of the box\" module this is a good approach. It was very easy for me to plug the uploader into a pretty progress bar, and my result looked like this:\n\n![img/progress.png](img/progress.png)\n\nThe author was also very wise to provide an [example](https://github.com/juliomalegria/django-chunked-upload-demo). But getting the same functionality from the command line would be a series of hacks. I [posted an issue](https://github.com/juliomalegria/django-chunked-upload/issues/41) but did not get any response, even after many weeks. I think this solution could be very good if the work is extended to include an integration with django restful (or similar) for interaction from the command line.\n\n### Chunk Upload (Restful)\nAnother user created a [django restful framework chunked upload](https://github.com/jkeifer/drf-chunked-upload) but did not provide substantial documentation to get it working. We [came very close](https://github.com/jkeifer/drf-chunked-upload/issues/6)\nbut I wasn't able to guess what the PUT or POST calls should look like, and I sensed we weren't going to progress very quickly. \n\n### Nginx Module\nI at first didn't like the idea of having a file uploaded directly to my server, but actually\nyou can integrate authentication at several levels so my concern was assuaged. For this demo I won't\nimplement any kind of authentication, but I'll point you to resources to figure it out (I would\nimagine the authentication schema / details would vary based on the use case).\n\n\n# Usage\n\nI've provided the application Dockerized so you don't need to install dependencies, beyond using Docker.\nTo bring up the server:\n\n```bash\ndocker-compose up -d\n```\n\nThen you need to create the nginx subfolders (this folder is bound to both containers):\n\n```bash\nsudo mkdir -p images/_upload/{0..9} \u0026\u0026 sudo chmod 777 -R images/_upload\n```\n\nThen to upload, you can use either the web interface or a command line utility [push.py](push.py)\n\n## Web Interface\n\nThen you can navigate to [http://127.0.0.1](http://127.0.0.1) to see the portal. It's the massive\npurpely/pink box you see in the picture above! You can click or drop to add a file to upload.\n\nWhen the file finishes uploading, you will see the table! You can always navigate back to the\nmain page by clicking the home icon in the upper left.\n\n![img/table.png](img/table.png)\n\n## Command Line\n\nI wrote you a little client to push an image to your server! Here is the usage:\n\n```bash\nusage: push.py [-h] [--host HOST] [--port PORT] [--schema SCHEMA] file\n\nDinosaur Nginx Upload Example\n\npositional arguments:\n  file                  full path to file to push\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --host HOST           the host where the server is running\n  --port PORT, -p PORT  the port where the server is running\n  --schema SCHEMA, -s SCHEMA\n                        http:// or https://\nusage: push.py [-h] [--host HOST] [--port PORT] [--schema SCHEMA] file\npush.py: error: the following arguments are required: file\n```\n\nYou shouldn't need to change the host or port or schema given running the Docker containers\nlocally, but I've added those arguments in case you want to extend the script to some other \nserver.\n\nAnd here is an example to upload a container. Note that you will need requests and requests tool belt instlled.\n\n```bash\npip install requests\npip install requests-toolbelt\n```\n```bash\n./push.py /home/vanessa/dfnworks.simg \nPUSH /home/vanessa/dfnworks.simg\n[3682 of 3682 MB]\n[Return status 200 Upload Complete]\n```\n\n## Deployment\nIf you deploy this, there are some customizations you need to take into account!\n\n - **Authentication** is removed for this demo. You should add it back. You can add this at the level of the nginx module, or via a view for the application.\n - **https** of course this should be served with https. The server block is largely the same, but you would have another for port 443.\n\nMy preference for a lot of these scaled problems is to \"outsource\" to another service (e.g., use Google Storage and their APIs) but given that we still need to deploy local modular applications on our own filesystems, this seems like a reasonable solution. If you have any questions or need help, please don't be afraid to [reach out](https://www.github.com/vsoch/django-nginx-upload/issues). I hope that this is helpful for you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvsoch%2Fdjango-nginx-upload","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvsoch%2Fdjango-nginx-upload","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvsoch%2Fdjango-nginx-upload/lists"}