{"id":15653993,"url":"https://github.com/abrookins/streaming_django","last_synced_at":"2025-04-30T22:24:33.586Z","repository":{"id":137289043,"uuid":"64275659","full_name":"abrookins/streaming_django","owner":"abrookins","description":"An explanation of how Django's streaming HTTP responses work","archived":false,"fork":false,"pushed_at":"2016-07-29T19:23:38.000Z","size":18,"stargazers_count":29,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-30T20:33:43.158Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abrookins.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-27T03:55:35.000Z","updated_at":"2025-03-12T18:02:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"b981007c-81ee-4e8d-becb-df06a50d8a22","html_url":"https://github.com/abrookins/streaming_django","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abrookins%2Fstreaming_django","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abrookins%2Fstreaming_django/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abrookins%2Fstreaming_django/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abrookins%2Fstreaming_django/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abrookins","download_url":"https://codeload.github.com/abrookins/streaming_django/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251790555,"owners_count":21644236,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T12:48:44.703Z","updated_at":"2025-04-30T22:24:33.565Z","avatar_url":"https://github.com/abrookins.png","language":"Python","readme":"# How does Django's `StreamingHttpResponse` work, exactly?\n\nThis repository exists to explain just what goes on when you use Django's\n`StreamingHttpResponse`.\n\nI will discuss what happens in your Django application, what happens at the\nPython Web Server Gateway Interface (WSGI) layer, and look at some examples.\n\n## How to use this repository\n\nJust read this document (README.md).\n\nIf you want to experiment with running `curl` requests against a streaming vs.\nnon-streaming Django view, follow the next section, \"Running the\n`streaming_django` project,\" to install the included example Django project.\n\n### Running the `streaming_django` project\n\nFirst, [install docker](https://www.docker.com/), including `docker-compose`,\nand then get a machine started.\n\nWhen you have a Docker machine running, do the following:\n\n\t$ git clone git@github.com:abrookins/streaming_django.git\n\t$ cd streaming_django\n\t$ docker-compose build\n\t$ docker-compose up\n\nNow you're ready to make a request:\n\n\t$ curl -vv --raw \"http://192.168.99.100/download_csv_streaming\"\n\nOr:\n\n\t$ curl -vv --raw \"http://192.168.99.100/download_csv\" \n\n**Pro tip**: The `--raw` flag is important if you want to see that a response\nis actually streaming. Without it, you won't see much difference between a\nstreaming and non-streaming response.\n\n## So, what even is a `StreamingHttpResponse`?\n\nMost Django responses use `HttpResponse`. At a high level, this means that the\nbody of the response is built in memory and sent to the HTTP client in a single\npiece.\n\nHere's a short example of using `HttpResponse`:\n\n```python\n    def my_view(request):\n        message = 'Hello, there!'\n        response =  HttpResponse(message)\n        response['Content-Length'] = len(message)\n\n        return response\n```\n\nA `StreamingHttpResponse`, on the other hand, is a response whose body is sent\nto the client in multiple pieces, or \"chunks.\"\n\nHere's a short example of using `StreamingHttpResponse`:\n\n```python\n    def hello():\n        yield 'Hello,'\n        yield 'there!'\n\n    def my_view(request):\n        # NOTE: No Content-Length header!\n        return StreamingHttpResponse(hello)\n```\n\nYou can read more about how to use these two classes in [Django's\ndocumentation](https://docs.djangoproject.com/en/1.9/ref/request-response/).\nThe interesting part is what happens next -- *after* you return the response.\n\n## When would you use a `StreamingHttpResponse`?\n\nBut before we talk about what happens *after* you return the response, let us\ndigress for a moment: why would you even use a `StreamingHttpResponse`?\n\nOne of the best use cases for streaming responses is to send large files, e.g.\na large CSV file.\n\nWith an `HttpResponse`, you would typically load the entire file into memory\n(produced dynamically or not) and then send it to the client. For a large file,\nthis costs memory on the server and \"time to first byte\" (TTFB) sent to the\nclient.\n\nWith a `StreamingHttpResponse`, you can load parts of the file into memory, or\nproduce parts of the file dynamically, and begin sending these parts to the\nclient immediately. **Crucially,** there is no need to load the entire file\ninto memory.\n\n## A quick note about WSGI\n\nNow we're approaching the part of our journey that lies just beyond most Django\ndevelopers' everyday experience of working with Django's response classes.\n\nYes, we're about to discuss the [Python Web Server Gateway Interface (WSGI)\nspecification](https://www.python.org/dev/peps/pep-3333/).\n\nSo, a quick note if you aren't familiar with WSGI. WSGI is a specification that\nproposes rules that web frameworks and web servers should follow in order that\nyou, the framework user, can swap out one WSGI server (like uWSGI) for another\n(Gunicorn) and expect your Python web application to continue to function.\n\n## Django and WSGI\n\nAnd now, back to our journey into deeper knowledge!\n\nSo, what happens after your Django view returns a `StreamingHttpResponse`? In\nmost Python web applications, the response is passed off to a WSGI server like\nuWSGI or Gunicorn (AKA, Green Unicorn).\n\nAs with `HttpResponse`, Django ensures that `StreamingHttpResponse` conforms to\nthe WSGI spec, which states this:\n\n\u003e When called by the server, the application object must return an iterable\n\u003e yielding zero or more bytestrings. This can be accomplished in a variety of\n\u003e ways, such as by returning a list of bytestrings, or by the application being a\n\u003e generator function that yields bytestrings, or by the application being a class\n\u003e whose instances are iterable.\n\nHere's how `StreamingHttpResponse` satisfies these requirements ([full\nsource](https://docs.djangoproject.com/en/1.9/_modules/django/http/response/#StreamingHttpResponse)):\n\n```python\n    @property\n    def streaming_content(self):\n        return map(self.make_bytes, self._iterator)\n# ...\n\n    def __iter__(self):\n        return self.streaming_content\n```\n\nYou give the class a generator and it coerces the values that it produces into\nbytestrings.\n\nCompare that with the approach in `HttpResponse` ([full source](https://docs.djangoproject.com/en/1.9/_modules/django/http/response/#HttpResponse)):\n\n```python\n    @content.setter\n    def content(self, value):\n        # ...\n        self._container = [value]\n\n    def __iter__(self):\n        return iter(self._container)\n```\n\nAh ha! An iterator with a single item. Very interesting. Now, let's take a look\nat what a WSGI server does with these two different responses.\n\n## The WSGI server\n\nGunicorn's synchronous worker offers a good example of what happens after\nDjango returns a response object. The code is [relatively\nshort](https://github.com/benoitc/gunicorn/blob/39f62ac66beaf83ceccefbfabd5e3af7735d2aff/gunicorn/workers/sync.py#L176-L183)\n-- here's the important part (for our purposes):\n\n```python\nrespiter = self.wsgi(environ, resp.start_response)\ntry:\n    if isinstance(respiter, environ['wsgi.file_wrapper']):\n        resp.write_file(respiter)\n    else:\n        for item in respiter:\n            resp.write(item)\n    resp.close()\n```\n\nWhether your response is streaming or not, Gunicorn iterates over it and writes\neach string the response yields. If that's the case, then what makes your\nstreaming response actually \"stream\"?\n\nFirst, some conditions must be true:\n\n* The client must be speaking HTTP/1.1 or newer\n* The request method wasn't a HEAD\n* The response does not include a Content-Length header\n* The response status wasn't 204 or 304\n\nIf these conditions are true, then Gunicorn will add a `Transfer-Encoding:\nchunked` header to the response, signaling to the client that the response will\nstream in chunks.\n\nIn fact, Gunicorn will respond with `Transfer-Encoding: chunked` even if you\nused an `HttpResponse`, if those conditions are true!\n\nTo really stream a response, that is, to send it to the client in pieces, the\nconditions must be true, *and* your response needs to be an iterable with\nmultiple items.\n\n### What does the client get?\n\nIf the streaming response worked, the client should get an HTTP 1.1 response\nwith the `Transfer-Encoding: chunked` header, and instead of a single piece of\ncontent with a `Content-Length`, the client should see each bytestring that\nyour generator/iterator yielded, sent with the length of that chunk.\n\nHere is an example that uses the code in this repository:\n\n```\n(streaming_django) ❯ curl -vv --raw \"http://192.168.99.100/download_csv_streaming\"\n*   Trying 192.168.99.100...\n* Connected to 192.168.99.100 (192.168.99.100) port 80 (#0)\n\u003e GET /download_csv_streaming HTTP/1.1\n\u003e Host: 192.168.99.100\n\u003e User-Agent: curl/7.43.0\n\u003e Accept: */*\n\u003e\n\u003c HTTP/1.1 200 OK\n\u003c Server: nginx/1.11.1\n\u003c Date: Fri, 29 Jul 2016 14:27:58 GMT\n\u003c Content-Type: text/csv\n\u003c Transfer-Encoding: chunked\n\u003c Connection: keep-alive\n\u003c X-Frame-Options: SAMEORIGIN\n\u003c Content-Disposition: attachment; filename=big.csv\n\u003c\nf\nOne,Two,Three\n\nf\nHello,world,1\n\n...\n\n10\nHello,world,99\n\n0\n\n* Connection #0 to host 192.168.99.100 left intact\n```\n\nSo there you have it. We journeyed from considering when to use\n`StreamingHttpResponse` over `HttpResponse`, to an example of using the class\nin your Django project, then into the dungeons of WSGI and WSGI servers, and\nfinally to the client's experience. And we managed to stream a response -- go\nus!\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabrookins%2Fstreaming_django","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabrookins%2Fstreaming_django","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabrookins%2Fstreaming_django/lists"}