{"id":16228932,"url":"https://github.com/kubukoz/dropbox-demo","last_synced_at":"2025-03-19T13:31:35.119Z","repository":{"id":52296896,"uuid":"350922969","full_name":"kubukoz/dropbox-demo","owner":"kubukoz","description":"A showcase project for my talk about typelevel stack architecture.","archived":false,"fork":false,"pushed_at":"2022-01-22T04:39:59.000Z","size":833,"stargazers_count":9,"open_issues_count":0,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-09T01:43:48.986Z","etag":null,"topics":["cats-effect","cats-effect-3"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kubukoz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-24T02:35:08.000Z","updated_at":"2024-02-10T18:59:11.000Z","dependencies_parsed_at":"2022-09-05T22:20:37.450Z","dependency_job_id":null,"html_url":"https://github.com/kubukoz/dropbox-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubukoz%2Fdropbox-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubukoz%2Fdropbox-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubukoz%2Fdropbox-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubukoz%2Fdropbox-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kubukoz","download_url":"https://codeload.github.com/kubukoz/dropbox-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243996257,"owners_count":20380977,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cats-effect","cats-effect-3"],"created_at":"2024-10-10T12:56:54.920Z","updated_at":"2025-03-19T13:31:34.718Z","avatar_url":"https://github.com/kubukoz.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dropbox-demo\n\nAn application accompanying [my talk about structuring functional applications in Scala](https://speakerdeck.com/kubukoz/connecting-the-dots-building-and-structuring-a-functional-application-in-scala).\n\n## Run application\n\n### Backend\n\nPrerequisites: sbt, Elasticsearch running on `localhost:9200`, `tesseract` binary available on the `PATH` and runnable.\n\nElasticsearch is available in the attached `docker-compose` setup. `tesseract` is available if you enter the attached `nix-shell`. The shell will also load environment variables from the `env.sh` file, if you have one (it's ignored in git).\n\nOnce you have these: `sbt run` - at time of writing the application starts on `0.0.0.0:4000`, this can be configured with the `HTTP_HOST`/`HTTP_PORT` environment variables.\n\n### Frontend\n\nPrerequisites: Node 14.x, npm (provided in a nix-shell in the `frontend` directory).\n\n```shell\ncd frontend\nnpm start\n```\n\n## Project goals\n\nSearch images from some storage by the text on them.\n\nI have tens of thousands of images to search, and the OCR (optical character recognition) process takes ~0.5 seconds per image for relatively small images, so live decoding is a no-no.\nInstead, we will allow the user to index a path from the store, and later search the database populated in that process.\n\nThe indexing will happen in the background, without the user having to wait for it to complete before getting a response.\nWe'll run the whole process in constant memory (except this is kind of up to Tesseract, I haven't investigated its memory usage on large images yet) - so both downloading the list of images to index, and their actual bytes, is done in a streaming fashion, thanks to fs2.\n\n## Infrastructure\n\nAt the time of writing:\n\n- OCR is performed by [Tesseract](https://github.com/tesseract-ocr/tesseract)\n- Image source is Dropbox, we'll be using the [User API](https://www.dropbox.com/developers/documentation/http/documentation)\n- Indexing and full text search are possible thanks to [Open Distro for Elasticsearch](https://opendistro.github.io).\n\nRight now, this only runs on a local machine. Tesseract is provided via the [nix shell](https://nixos.org/), Open Distro runs in [docker](https://www.docker.com/). The application can be started using [bloop](https://scalacenter.github.io/bloop/).\n\n## Tech stack\n\nThe backend is built in [Scala](https://scala-lang.org) (obviously - that was the point of the talk), using the following libraries:\n\n- [Cats Effect 3](https://typelevel.org/cats-effect), for several things, such as monadic composition of asynchronous tasks (e.g. Elasticsearch client) and interop with other libraries from the ecosystem\n- [fs2](https://fs2.io) - for streaming data, so that we can run the indexing process in constant memory, as long as the OCR implementation can do so\n- [http4s](https://http4s.org/) - for the HTTP server, as well as a custom client for Dropbox\n- [ciris](https://cir.is) - for compositionally loading configuration\n- [circe](https://circe.github.io/circe) - for decoding/encoding JSON\n- [log4cats](https://typelevel.org/log4cats/) - for logging\n- [Elasticsearch high-level Java client](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html) - for talking to Elasticsearch.\n  Normally you could use something like [elastic4s](https://github.com/sksamuel/elastic4s/), but I only needed a subset of its functionality and wanted to show how this can be wrapped in `cats.effect.IO`\n- [weaver](https://disneystreaming.github.io/weaver-test/) - for testing\n- [chimney](https://scalalandio.github.io/chimney/) - for transforming similar datatypes\n\nThe frontend is built with [React](https://reactjs.org/) + [TypeScript](https://www.typescriptlang.org/). You'll need [Node 14.x and npm](https://nodejs.org/en/), both are provided with the attached nix shell in the `frontend` directory.\n\n## Architecture\n\nFirst of all, the data flow in processes that can be triggered by the user:\n\n![Data flows](docs/data-flow.png)\n\nThere are three main processes:\n\n### Indexing\n\n- Wait for the user to provide a directory to index (a path to the directory in Dropbox)\n- Download metadata of all images within that directory (recursively)\n- For each entry, get a stream of bytes of its content, push it to Tesseract\n- Pass the metadata and the OCR decoding result to the indexer (Elasticsearch)\n\nAPI:\n\n```shell\n# paths must start with /\nhttp :4000/index path=\"/images\"\n```\n\n```http\nPOST /index HTTP/1.1\nAccept: application/json, */*;q=0.5\nAccept-Encoding: gzip, deflate\nConnection: keep-alive\nContent-Length: 27\nContent-Type: application/json\nHost: localhost:4000\nUser-Agent: HTTPie/2.4.0\n\n{\n    \"path\": \"/images\"\n}\n\n\nHTTP/1.1 202 Accepted\nContent-Length: 0\nDate: Sun, 02 May 2021 17:57:19 GMT\n```\n\n### Search\n\n- Ask the user for a query\n- Pass the query to Elasticsearch with some level of fuzziness\n- Pass results back to the user (metadata of matching files)\n\nAPI:\n\n```shell\nhttp :4000/search\\?query=test\n```\n\n```http\nGET /search?query=test HTTP/1.1\nAccept: */*\nAccept-Encoding: gzip, deflate\nConnection: keep-alive\nHost: localhost:4000\nUser-Agent: HTTPie/2.4.0\n\n\n\nHTTP/1.1 200 OK\nContent-Type: application/json\nDate: Sun, 02 May 2021 17:58:41 GMT\nTransfer-Encoding: chunked\n\n[\n    {\n        \"content\": \"Some decoded text\",\n        \"imageUrl\": \"http://127.0.0.1:4000/view/%2Fimages%2Ffile1.jpg\",\n        \"thumbnailUrl\": \"http://127.0.0.1:4000/view/%2Fimages%2Ffile1.jpg\"\n    },\n    {\n        \"content\": \"Another file with test text\",\n        \"imageUrl\": \"http://127.0.0.1:4000/view/%2Fimages%2Ffile2.jpg\",\n        \"thumbnailUrl\": \"http://127.0.0.1:4000/view/%2Fimages%2Ffile2.jpg\"\n    }\n]\n```\n\n### Download\n\n- Take a concrete file's path from the user (from file metadata)\n- Return stream of bytes for that file\n\nAPI:\n\n```shell\nhttp :4000/view/%2Fimages%2Ffile2.jpg\n```\n\n```http\nGET /view/%2Fimages%2Ffile2.jpg HTTP/1.1\nAccept: */*\nAccept-Encoding: gzip, deflate\nConnection: keep-alive\nHost: localhost:4000\nUser-Agent: HTTPie/2.4.0\n\n\n\nHTTP/1.1 200 OK\nContent-Type: image/jpeg\nDate: Sun, 02 May 2021 18:01:30 GMT\nTransfer-Encoding: chunked\n\n\n\n+-----------------------------------------+\n| NOTE: binary data not shown in terminal |\n+-----------------------------------------+\n```\n\n### Module graph\n\n![Modules](docs/modules.png)\n\nThe blue boxes correspond to the processes outlined above (core logic), green boxes are high-level adapters for the underlying vendor-specific implementations - this is similar to the [hexagonal architecture](https://www.youtube.com/watch?v=sOaS83Ir8Ck) / Ports\u0026Adapters patterns.\n\nThese correspond almost directly to the interfaces (Tagless Final algebras) in the project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubukoz%2Fdropbox-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkubukoz%2Fdropbox-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubukoz%2Fdropbox-demo/lists"}