{"id":46714075,"url":"https://github.com/dwyl/image-classifier","last_synced_at":"2026-03-09T10:02:46.492Z","repository":{"id":204789760,"uuid":"709711262","full_name":"dwyl/image-classifier","owner":"dwyl","description":"πŸ–ΌοΈ Classify images and extract data from or describe their contents using machine learning","archived":false,"fork":false,"pushed_at":"2026-01-02T20:12:14.000Z","size":24486,"stargazers_count":31,"open_issues_count":5,"forks_count":5,"subscribers_count":7,"default_branch":"main","last_synced_at":"2026-01-06T21:34:12.228Z","etag":null,"topics":["ai","artificial-intelligence","classification","hugging-face","huggingface","image","image-classification","image-recognition","learn","machine-learning","machinelearning","tutorial"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dwyl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-10-25T08:42:36.000Z","updated_at":"2026-01-04T06:13:09.000Z","dependencies_parsed_at":"2025-09-01T17:35:55.463Z","dependency_job_id":null,"html_url":"https://github.com/dwyl/image-classifier","commit_stats":null,"previous_names":["dwyl/image-classifier"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dwyl/image-classifier","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dwyl","download_url":"https://codeload.github.com/dwyl/image-classifier/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30290933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T02:57:19.223Z","status":"ssl_error","status_checked_at":"2026-03-09T02:56:26.373Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","classification","hugging-face","huggingface","image","image-classification","image-recognition","learn","machine-learning","machinelearning","tutorial"],"created_at":"2026-03-09T10:02:42.037Z","updated_at":"2026-03-09T10:02:46.478Z","avatar_url":"https://github.com/dwyl.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Image Captioning \u0026 Semantic Search in `Elixir`\n\n![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/dwyl/image-classifier/ci.yml?label=build\u0026style=flat-square\u0026branch=main)\n[![codecov.io](https://img.shields.io/codecov/c/github/dwyl/image-classifier/main.svg?style=flat-square)](https://codecov.io/github/dwyl/image-classifier?branch=main)\n[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat-square)](https://github.com/dwyl/image-classifier/issues)\n[![HitCount](https://hits.dwyl.com/dwyl/image-classifier.svg?style=flat-square\u0026show=unique)](https://hits.dwyl.com/dwyl/image-classifier)\n\nLet's use `Elixir` machine learning capabilities\nto build an application\nthat performs **image captioning**\nand **semantic search**\nto look for uploaded images\nwith your voice! πŸŽ™οΈ\n\n\u003c/div\u003e\n\n\u003cbr /\u003e\n\n- [Image Captioning \\\u0026 Semantic Search in `Elixir`](#image-captioning--semantic-search-in-elixir)\n - [Why? 🀷](#why-)\n - [What? πŸ’­](#what-)\n - [Who? πŸ‘€](#who-)\n - [How? πŸ’»](#how-)\n - [Prerequisites](#prerequisites)\n - [πŸŒ„ Image Captioning in `Elixir`](#-image-captioning-in-elixir)\n - [0. Creating a fresh `Phoenix` project](#0-creating-a-fresh-phoenix-project)\n - [1. Installing initial dependencies](#1-installing-initial-dependencies)\n - [2. Adding `LiveView` capabilities to our project](#2-adding-liveview-capabilities-to-our-project)\n - [3. Receiving image files](#3-receiving-image-files)\n - [4. Integrating `Bumblebee`](#4-integrating-bumblebee)\n - [4.1 `Nx` configuration](#41-nx-configuration)\n - [4.2 `Async` processing the image for classification](#42-async-processing-the-image-for-classification)\n - [4.2.1 Considerations regarding `async` processes](#421-considerations-regarding-async-processes)\n - [4.2.2 Alternative for better testing](#422-alternative-for-better-testing)\n - [4.3 Image pre-processing](#43-image-pre-processing)\n - [4.4 Updating the view](#44-updating-the-view)\n - [4.5 Check it out!](#45-check-it-out)\n - [4.6 Considerations on user images](#46-considerations-on-user-images)\n - [5. Final Touches](#5-final-touches)\n - [5.1 Setting max file size](#51-setting-max-file-size)\n - [5.2 Show errors](#52-show-errors)\n - [5.3 Show image preview](#53-show-image-preview)\n - [6. What about other models?](#6-what-about-other-models)\n - [7. How do I deploy this thing?](#7-how-do-i-deploy-this-thing)\n - [8. Showing example images](#8-showing-example-images)\n - [8.1 Creating a hook in client](#81-creating-a-hook-in-client)\n - [8.2 Handling the example images list event inside our LiveView](#82-handling-the-example-images-list-event-inside-our-liveview)\n - [8.3 Updating the view](#83-updating-the-view)\n - [8.4 Using URL of image instead of base64-encoded](#84-using-url-of-image-instead-of-base64-encoded)\n - [8.5 See it running](#85-see-it-running)\n - [9. Store metadata and classification info](#9-store-metadata-and-classification-info)\n - [9.1 Installing dependencies](#91-installing-dependencies)\n - [9.2 Adding `Postgres` configuration files](#92-adding-postgres-configuration-files)\n - [9.3 Creating `Image` schema](#93-creating-image-schema)\n - [9.4 Changing our LiveView to persist data](#94-changing-our-liveview-to-persist-data)\n - [10. Adding double MIME type check and showing feedback to the person in case of failure](#10-adding-double-mime-type-check-and-showing-feedback-to-the-person-in-case-of-failure)\n - [10.1 Showing a toast component with error](#101-showing-a-toast-component-with-error)\n - [11. Benchmarking image captioning models](#11-benchmarking-image-captioning-models)\n - [πŸ” Semantic search](#-semantic-search)\n - [0. Overview of the process](#0-overview-of-the-process)\n - [0.1 Audio transcription](#01-audio-transcription)\n - [0.2 Creating embeddings](#02-creating-embeddings)\n - [0.3 Semantical search](#03-semantical-search)\n - [1. Pre-requisites](#1-pre-requisites)\n - [2. Transcribe an audio recording](#2-transcribe-an-audio-recording)\n - [1.1 Adding a loading spinner](#11-adding-a-loading-spinner)\n - [2.2 Defining `Javascript` hook](#22-defining-javascript-hook)\n - [2.3 Handling audio upload in `LiveView`](#23-handling-audio-upload-in-liveview)\n - [2.4 Serving the `Whisper` model](#24-serving-the-whisper-model)\n - [2.5 Handling the model's response and updating elements in the view](#25-handling-the-models-response-and-updating-elements-in-the-view)\n - [3. Embeddings and semantic search](#3-embeddings-and-semantic-search)\n - [3.1 The `HNSWLib` Index (GenServer)](#31-the-hnswlib-index-genserver)\n - [3.2 Saving the `HNSWLib` Index in the database](#32-saving-the-hnswlib-index-in-the-database)\n - [3.2 The embeding model](#32-the-embeding-model)\n - [4. Using the Index and embeddings](#4-using-the-index-and-embeddings)\n - [4.0 Check the folder \"hnswlib\"](#40-check-the-folder-hnswlib)\n - [4.1 Computing the embeddings in our app](#41-computing-the-embeddings-in-our-app)\n - [4.1.1 Changing the `Image` schema so it's embeddable](#411-changing-the-image-schema-so-its-embeddable)\n - [4.1.2 Using embeddings in semantic search](#412-using-embeddings-in-semantic-search)\n - [4.1.2.1 Mount socket assigns](#4121-mount-socket-assigns)\n - [4.1.2.2 Consuming image uploads](#4122-consuming-image-uploads)\n - [4.1.2.3 Using the embeddings to semantically search images](#4123-using-the-embeddings-to-semantically-search-images)\n - [4.1.2.4 Creating embeddings when uploading images](#4124-creating-embeddings-when-uploading-images)\n - [4.1.2.5 Update the LiveView view](#4125-update-the-liveview-view)\n - [5. Tweaking our UI](#5-tweaking-our-ui)\n - [_Please_ star the repo! ⭐️](#please-star-the-repo-️)\n\n\u003cbr /\u003e\n\n## Why? 🀷\n\nWhilst building our [app](https://github.com/dwyl/app),\nwe consider `images` an _essential_ medium of communication.\n\nWe needed a fully-offline capable (no 3rd party APIs/Services) image captioning service\nusing state-of-the-art pre-trained image and embedding models to describe images uploaded in our\n[`App`](https://github.com/dwyl/app).\n\nBy adding a way of captioning images, we make it _easy_ for people to suggest meta tags that describe images so they become **searchable**.\n\n## What? πŸ’­\n\nA step-by-step tutorial building a fully functional\n`Phoenix LiveView` web application that allows anyone\nto upload an image and have it described\nand searchable.\n\nIn addition to this,\nthe app will allow the person to record an audio\nwhich describes the image they want to find.\n\nThe audio will be transcribed into text\nand be semantically queryable.\nWe do this by encoding the image captions\nas vectors and running `knn search` on them.\n\nWe'll be using three different models:\n\n- Salesforce's BLIP model [`blip-image-captioning-large`](https://huggingface.co/Salesforce/blip-image-captioning-large)\n for image captioning.\n- OpenAI's speech recognition model\n [`whisper-small`](https://huggingface.co/openai/whisper-small).\n- [`sentence-transformers/paraphrase-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2)\n embedding model.\n\n## Who? πŸ‘€\n\nThis tutorial is aimed at `Phoenix` beginners\nwho want to start exploring the machine-learning capabilities\nof the Elixir language within a `Phoenix` application.\nWe propose to use pre-trained models from Hugging Face via `Bumblebee`\nand grasp how to:\n\n- run a model, in particular image captioning.\n- how to use embeddings.\n- how to run a semantic search using an\n [Approximate Nearest Neighbour](https://towardsdatascience.com/comprehensive-guide-to-approximate-nearest-neighbors-algorithms-8b94f057d6b6)\n algorithm.\n\nIf you are completely new to `Phoenix` and `LiveView`,\nwe recommend you follow the **`LiveView` _Counter_ Tutorial**:\n\n[dwyl/phoenix-liveview-counter-tutorial](https://github.com/dwyl/phoenix-liveview-counter-tutorial)\n\n## How? πŸ’»\n\nIn these chapters, we'll go over the development process of this small application.\nYou'll learn how to do this _yourself_, so grab some coffee and let's get cracking!\n\nThis section will be divided into two sections.\nOne will go over **image captioning**\nwhile the second one will expand the application\nby adding **semantic search**.\n\n## Prerequisites\n\nThis tutorial requires you to have `Elixir` and `Phoenix` installed.\n\nIf you don't, please see [how to install Elixir](https://github.com/dwyl/learn-elixir#installation) and [Phoenix](https://hexdocs.pm/phoenix/installation.html#phoenix).\n\nThis guide assumes you know the basics of `Phoenix`\nand have _some_ knowledge of how it works.\nIf you don't, we _highly suggest_ you follow our other tutorials first, e.g: [github.com/dwyl/**phoenix-chat-example**](https://github.com/dwyl/phoenix-chat-example)\n\nIn addition to this, **_some_ knowledge of `AWS`** - what it is, what an `S3` bucket is/does - **is assumed**.\n\n\u003e [!NOTE]\n\u003e If you have questions or get stuck,\n\u003e please open an issue!\n\u003e [/dwyl/image-classifier/issues](https://github.com/dwyl/image-classifier/issues)\n\n\u003cdiv align=\"center\"\u003e\n\n## πŸŒ„ Image Captioning in `Elixir`\n\n\u003e In this section, we'll start building our application\n\u003e with `Bumblebee` that supports Transformer models.\n\u003e At the end of this section,\n\u003e you'll have a fully functional application\n\u003e that receives an image,\n\u003e processes it accordingly\n\u003e and captions it.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/image-classifier/assets/17494745/05d0b510-ef9a-4a51-8425-d27902b0f7ad\"\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n### 0. Creating a fresh `Phoenix` project\n\nLet's create a fresh `Phoenix` project.\nRun the following command in a given folder:\n\n```sh\nmix phx.new . --app app --no-dashboard --no-ecto --no-gettext --no-mailer\n```\n\nWe're running [`mix phx.new`](https://hexdocs.pm/phoenix/Mix.Tasks.Phx.New.html) to generate a new project without a dashboard and mailer (email) service,\nsince we don't need those features in our project.\n\nAfter this, if you run `mix phx.server` to run your server, you should be able to see the following page.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/fields/assets/194400/891e890e-c94a-402e-baee-ee47fd3725a7\"\u003e\n\u003c/p\u003e\n\nWe're ready to start building.\n\n### 1. Installing initial dependencies\n\nNow that we're ready to go, let's start by adding some dependencies.\n\nHead over to `mix.exs`and add the following dependencies\nto the `deps` section.\n\n```elixir\n{:bumblebee, \"~\u003e 0.5.0\"},\n{:exla, \"~\u003e 0.7.0\"},\n{:nx, \"~\u003e 0.7.0 \"},\n{:hnswlib, \"~\u003e 0.1.5\"},\n```\n\n- [**`bumblebee`**](https://github.com/elixir-nx/bumblebee) is a framework that will allows us to integrate\n [`Transformer Models`](https://huggingface.co/docs/transformers/index) in `Phoenix`.\n The `Transformers` (from [Hugging Face](https://huggingface.co/))\n are APIs that allow us to easily download and use\n [pre-trained models](https://blogs.nvidia.com/blog/2022/12/08/what-is-a-pretrained-ai-model).\n The `Bumblebee` package aims to support all Transformer Models, even if some are still lacking.\n You may check which ones are supported by visiting\n `Bumblebee`'s repository or by visiting https://jonatanklosko-bumblebee-tools.hf.space/apps/repository-inspector\n and checking if the model is currently supported.\n\n- [**`Nx`**](https://hexdocs.pm/nx/Nx.html) is a library that allows us to work with\n [`Numerical Elixir`](https://github.com/elixir-nx/), the Elixir's way of doing [numerical computing](https://www.hilarispublisher.com/open-access/introduction-to-numerical-computing-2168-9679-1000423.pdf). It supports tensors and numericla computations.\n\n- [**`EXLA`**](https://hexdocs.pm/exla/EXLA.html) is the Elixir implementation of [Google's XLA](https://www.tensorflow.org/xla/),\n a compiler that provides faster linear algebra calculations\n with `TensorFlow` models.\n This backend compiler is needed for `Nx`.\n We are installing `EXLA` because it allows us to compile models _just-in-time_ and run them on CPU and/or GPU.\n\n- [**`Vix`**](https://hexdocs.pm/vix/readme.html) is an Elixir extension for [libvips](https://www.libvips.org/), an image processing library.\n\nIn `config/config.exs`, let's add our `:nx` configuration\nto use `EXLA`.\n\n```elixir\nconfig :nx, default_backend: EXLA.Backend\n```\n\n### 2. Adding `LiveView` capabilities to our project\n\nAs it stands, our project is not using `LiveView`.\nLet's fix this.\n\nThis will launch a super-powered process that establishes a WebSocket connection\nbetween the server and the browser.\n\nIn `lib/app_web/router.ex`, change the `scope \"/\"` to the following.\n\n```elixir\n scope \"/\", AppWeb do\n pipe_through :browser\n\n live \"/\", PageLive\n end\n```\n\nInstead of using the `PageController`,\nwe are going to be creating `PageLive`,\na `LiveView` file.\n\nLet's create our `LiveView` files.\nInside `lib/app_web`,\ncreate a folder called `live`\nand create the following file\n`page_live.ex`.\n\n```elixir\n#/lib/app_web/live/page_live.ex\ndefmodule AppWeb.PageLive do\n use AppWeb, :live_view\n\n @impl true\n def mount(_params, _session, socket) do\n {:ok, socket}\n end\nend\n\n```\n\nThis is a simple `LiveView` controller.\n\nIn the same `live` folder,\ncreate a file called `page_live.html.heex`\nand use the following code.\n\n```html\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-28 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv\n class=\"flex justify-center items-center mx-auto max-w-xl w-[50vw] lg:mx-0\"\n \u003e\n \u003cform\u003e\n \u003cdiv class=\"space-y-12\"\u003e\n \u003cdiv\u003e\n \u003ch2 class=\"text-base font-semibold leading-7 text-gray-900\"\u003e\n Image Classifier\n \u003c/h2\u003e\n \u003cp class=\"mt-1 text-sm leading-6 text-gray-600\"\u003e\n Drag your images and we'll run an AI model to caption it!\n \u003c/p\u003e\n\n \u003cdiv class=\"mt-10 grid grid-cols-1 gap-x-6 gap-y-8 sm:grid-cols-6\"\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cspan\u003eUpload a file\u003c/span\u003e\n \u003cinput\n id=\"file-upload\"\n name=\"file-upload\"\n type=\"file\"\n class=\"sr-only\"\n /\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/form\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nThis is a simple HTML form that uses\n[`Tailwind CSS`](https://github.com/dwyl/learn-tailwind)\nto enhance the presentation of the upload form.\nWe'll also remove the unused header of the page layout,\nwhile we're at it.\n\nLocate the file `lib/app_web/components/layouts/app.html.heex`\nand remove the `\u003cheader\u003e` class.\nThe file should only have the following code:\n\n```html\n\u003cmain class=\"px-4 py-20 sm:px-6 lg:px-8\"\u003e\n \u003cdiv class=\"mx-auto max-w-2xl\"\u003e\n \u003c.flash_group flash={@flash} /\u003e \u003c%= @inner_content %\u003e\n \u003c/div\u003e\n\u003c/main\u003e\n```\n\nNow you can safely delete the `lib/app_web/controllers` folder,\nwhich is no longer used.\n\nIf you run `mix phx.server`,\nyou should see the following screen:\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/imgup/assets/17494745/5a3438fe-fa45-47f9-8cb2-9d6d405f55a0\"\u003e\n\u003c/p\u003e\n\nThis means we've successfully added `LiveView`\nand changed our view!\n\n### 3. Receiving image files\n\nNow, let's start by receiving some image files.\n\nWith `LiveView`,\nwe can easily do this by using\n[`allow_upload/3`](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#allow_upload/3)\nwhen mounting our `LiveView`.\nWith this function, we can easily accept\nfile uploads with progress.\nWe can define file types, max number of entries,\nmax file size,\nvalidate the uploaded file and much more!\n\nFirstly,\nlet's make some changes to\n`lib/app_web/live/page_live.html.heex`.\n\n```html\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-28 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv\n class=\"flex justify-center items-center mx-auto max-w-xl w-[50vw] lg:mx-0\"\n \u003e\n \u003cdiv class=\"space-y-12\"\u003e\n \u003cdiv class=\"border-gray-900/10 pb-12\"\u003e\n \u003ch2 class=\"text-base font-semibold leading-7 text-gray-900\"\u003e\n Image Classification\n \u003c/h2\u003e\n \u003cp class=\"mt-1 text-sm leading-6 text-gray-600\"\u003e\n Do simple captioning with this\n \u003ca\n href=\"https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html\"\n class=\"font-mono font-medium text-sky-500\"\n \u003eLiveView\u003c/a\n \u003e\n demo, powered by\n \u003ca\n href=\"https://github.com/elixir-nx/bumblebee\"\n class=\"font-mono font-medium text-sky-500\"\n \u003eBumblebee\u003c/a\n \u003e.\n \u003c/p\u003e\n\n \u003c!-- File upload section --\u003e\n \u003cdiv class=\"mt-10 grid grid-cols-1 gap-x-6 gap-y-8 sm:grid-cols-6\"\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n phx-drop-target=\"{@uploads.image_list.ref}\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform phx-change=\"validate\" phx-submit=\"save\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nWe've added a few features:\n\n- we used\n [`\u003c.live_file_input/\u003e`](https://hexdocs.pm/phoenix_live_view/Phoenix.Component.html#live_file_input/1)\n for `LiveView` file upload.\n We've wrapped this component\n with an element that is annotated with the `phx-drop-target` attribute\n pointing to the DOM `id` of the file input.\n- because we used `\u003c.live_file_input/\u003e`,\n we need to annotate its wrapping element\n with `phx-submit` and `phx-change`,\n as per\n [hexdocs.pm/phoenix_live_view/uploads.html#render-reactive-elements](https://hexdocs.pm/phoenix_live_view/uploads.html#render-reactive-elements)\n\nBecause we've added these bindings,\nwe need to add the event handlers in\n`lib/app_web/live/page_live.ex`.\nOpen it and update it to:\n\n```elixir\ndefmodule AppWeb.PageLive do\n use AppWeb, :live_view\n\n @impl true\n def mount(_params, _session, socket) do\n {:ok,\n socket\n |\u003e assign(label: nil, upload_running?: false, task_ref: nil)\n |\u003e allow_upload(:image_list,\n accept: ~w(image/*),\n auto_upload: true,\n progress: \u0026handle_progress/3,\n max_entries: 1,\n chunk_size: 64_000\n )}\n end\n\n @impl true\n def handle_event(\"validate\", _params, socket) do\n {:noreply, socket}\n end\n\n @impl true\n def handle_event(\"remove-selected\", %{\"ref\" =\u003e ref}, socket) do\n {:noreply, cancel_upload(socket, :image_list, ref)}\n end\n\n @impl true\n def handle_event(\"save\", _params, socket) do\n {:noreply, socket}\n end\n\n defp handle_progress(:image_list, entry, socket) when entry.done? do\n uploaded_file =\n consume_uploaded_entry(socket, entry, fn %{path: _path} = _meta -\u003e\n {:ok, entry}\n end)\n {:noreply, socket}\n end\n\n defp handle_progress(:image_list, _, socket), do: {:noreply, socket}\nend\n```\n\n- when `mount/3`ing the LiveView,\n we are creating three socket assigns:\n `label` pertains to the model prediction;\n `upload_running?` is a boolean referring to whether the model is running or not;\n `task_ref` refers to the reference of the task that was created for image classification\n (we'll delve into this further later down the line).\n Additionally, we are using the `allow_upload/3` function to define our upload configuration.\n The most important settings here are `auto_upload` set to `true`\n and the `progress` fields.\n By configuring these two properties,\n we are telling `LiveView` that _whenever the person uploads a file_,\n **it is processed immediately and consumed**.\n\n- the `progress` field is handled by the `handle_progress/3` function.\n It receives chunks from the client with a build-in `UploadWriter` function\n (as explained in the [docs](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.UploadWriter.html)).\n When the chunks are all consumed, we get the boolean `entry.done? == true`.\n We consume the file in this function by using\n [`consume_uploaded_entry/3`](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#consume_uploaded_entry/3).\n The anonymous function returns `{:ok, data}` or `{:postpone, message}`.\n Whilst consuming the entry/file,\n we can access its path and then use its content.\n _For now_, we don't need to use it.\n But we will in the future to feed our image classifier with it!\n After the callback function is executed,\n this function \"consumes the entry\",\n essentially deleting the image from the temporary folder\n and removing it from the uploaded files list.()\n\n- the `\"validate\"`, `\"remove-selected\"`, `\"save\"` event handlers\n are called whenever the person uploads the image,\n wants to remove it from the list of uploaded images\n and when wants to submit the form,\n respectively.\n You may see that we're not doing much with these handlers;\n we're simply replying with a `:noreply`\n because we don't need to do anything with them.\n\nAnd that's it!\nIf you run `mix phx.server`, nothing will change.\n\n### 4. Integrating `Bumblebee`\n\nNow here comes the fun part!\nIt's time to do some image captioning! πŸŽ‰\n\n#### 4.1 `Nx` configuration\n\nWe first need to add some initial setup in the\n`lib/app/application.ex` file.\nHead over there and change\nthe `start` function like so:\n\n```elixir\n@impl true\ndef start(_type, _args) do\n children = [\n # Start the Telemetry supervisor\n AppWeb.Telemetry,\n # Start the PubSub system\n {Phoenix.PubSub, name: App.PubSub},\n {Nx.Serving, serving: serving(), name: ImageClassifier},\n # Start the Endpoint (http/https)\n AppWeb.Endpoint\n ]\n\n opts = [strategy: :one_for_one, name: App.Supervisor]\n Supervisor.start_link(children, opts)\nend\n\ndef serving do\n {:ok, model_info} = Bumblebee.load_model({:hf, \"microsoft/resnet-50\"})\n {:ok, featurizer} = Bumblebee.load_featurizer({:hf, \"microsoft/resnet-50\"})\n\n Bumblebee.Vision.image_classification(model_info, featurizer,\n top_k: 1,\n compile: [batch_size: 10],\n defn_options: [compiler: EXLA]\n )\nend\n```\n\nWe are using [`Nx.Serving`](https://hexdocs.pm/nx/Nx.Serving.html), which simply allows us to encapsulate tasks; it can be networking, machine learning, data processing or any other task.\n\nIn this specific case, we are using it to **batch requests**.\nThis is extremely useful and important because we are using models that typically run on\n[GPU](https://en.wikipedia.org/wiki/Graphics_processing_unit).\nThe GPU is _really good_ at **parallelizing tasks**.\nTherefore, instead of sending an image classification request one by one, we can _batch them_/bundle them together as much as we can and then send it over.\n\nWe can define the `batch_size` and `batch_timeout` with `Nx.Serving`.\nWe're going to use the default values, hence why we're not explicitly defining them.\n\nWith `Nx.Serving`, we define a `serving/0` function\nthat is then used by it, which in turn is executed in the supervision tree since we declare it as a child in the Application module.\n\nIn the `serving/0` function, we are loading the[`ResNet-50`](https://huggingface.co/microsoft/resnet-50) model and its featurizer.\n\n\u003e [!NOTE]\n\u003e A `featurizer` can be seen as a\n\u003e [`Feature Extractor`](https://huggingface.co/docs/transformers/main_classes/feature_extractor).\n\u003e It is essentially a component that is responsible for converting input data\n\u003e into a format that can be processed by a pre-trained language model.\n\u003e\n\u003e It takes raw information and performs various transformations,\n\u003e such as\n\u003e [tokenization](https://neptune.ai/blog/tokenization-in-nlp),\n\u003e [padding](https://www.baeldung.com/cs/deep-neural-networks-padding),\n\u003e and encoding to prepare the data for model training or inference.\n\nLastly, this function returns a serving for image classification by calling [`image_classification/3`](https://hexdocs.pm/bumblebee/Bumblebee.Vision.html#image_classification/3), where we can define our compiler and task batch size.\nWe gave our serving function the name `ImageClassifier` as declared in the Application module.\n\n#### 4.2 `Async` processing the image for classification\n\nNow we're ready to send the image to the model\nand get a prediction of it!\n\nEvery time we upload an image,\nwe are going to run **async processing**.\nThis means that the task responsible for image classification will be run in another process, thus asynchronously, meaning that the LiveView _won't have to wait_ for this task to finish\nto continue working.\n\nFor this scenario, we are going to be using the\n[`Task` module](https://hexdocs.pm/elixir/1.14/Task.html) to spawn processes to complete this task.\n\nGo to `lib/app_web/live/page_live.ex`\nand change the following code.\n\n```elixir\ndef handle_progress(:image_list, entry, socket) when entry.done? do\n # Consume the entry and get the tensor to feed to classifier\n tensor = consume_uploaded_entry(socket, entry, fn %{} = meta -\u003e\n {:ok, vimage} = Vix.Vips.Image.new_from_file(meta.path)\n pre_process_image(vimage)\n end)\n\n # Create an async task to classify the image\n task = Task.async(fn -\u003e Nx.Serving.batched_run(ImageClassifier, tensor) end)\n\n # Update socket assigns to show spinner whilst task is running\n {:noreply, assign(socket, upload_running?: true, task_ref: task.ref)}\nend\n\n@impl true\ndef handle_info({ref, result}, %{assigns: %{task_ref: ref}} = socket) do\n # This is called everytime an Async Task is created.\n # We flush it here.\n Process.demonitor(ref, [:flush])\n\n # And then destructure the result from the classifier.\n %{predictions: [%{label: label}]} = result\n\n # Update the socket assigns with result and stopping spinner.\n {:noreply, assign(socket, label: label, upload_running?: false)}\nend\n```\n\n\u003e [!NOTE]\n\u003e The `pre_process_image/1` function is yet to be defined.\n\u003e We'll do that in the following section.\n\nIn the `handle_progress/3` function,\nwhilst we are consuming the image,\nwe are first converting it to a\n[`Vix.Vips.Image`](https://hexdocs.pm/vix/Vix.Vips.Image.html) `Struct`\nusing the file path.\nWe then feed this image to the `pre_process_image/1` function that we'll implement later.\n\nWhat's important is to notice this line:\n\n```elixir\ntask = Task.async(fn -\u003e Nx.Serving.batched_run(ImageClassifier, tensor) end)\n```\n\nWe are using\n[`Task.async/1`](https://hexdocs.pm/elixir/1.12/Task.html#async/1)\nto call our `Nx.Serving` build function `ImageClassifier` we've defined earlier,\nthus initiating a batched run with the image tensor.\nWhile the task is spawned,\nwe update the socket assigns with the reference to the task (`:task_ref`)\nand update the `:upload_running?` assign to `true`,\nso we can show a spinner or a loading animation.\n\nWhen the task is spawned using `Task.async/1`,\na couple of things happen in the background.\nThe new process is monitored by the caller (our `LiveView`),\nwhich means that the caller will receive a\n`{:DOWN, ref, :process, object, reason}`\nmessage once the process it is monitoring dies.\nAnd, a link is created between both processes.\n\nTherefore,\nwe **don't need to use**\n[**`Task.await/2`**](https://hexdocs.pm/elixir/1.12/Task.html#await/2).\nInstead, we create a new handler to receive the aforementioned.\nThat's what we're doing in the\n`handle_info({ref, result}, %{assigns: %{task_ref: ref}} = socket)` function.\nThe received message contains a `{ref, result}` tuple,\nwhere `ref` is the monitor’s reference.\nWe use this reference to stop monitoring the task,\nsince we received the result we needed from our task\nand we can discard an exit message.\n\nIn this same function, we destructure the prediction\nfrom the model and assign it to the socket assign `:label`\nand set `:upload_running?` to `false`.\n\nQuite beautiful, isn't it?\nWith this, we don't have to worry if the person closes the browser tab.\nThe process dies (as does our `LiveView`),\nand the work is automatically cancelled,\nmeaning no resources are spent\non a process for which nobody expects a result anymore.\n\n##### 4.2.1 Considerations regarding `async` processes\n\nWhen a task is spawned using `Task.async/2`,\n**it is linked to the caller**.\nWhich means that they're related:\nif one dies, the other does too.\n\nWe ought to take this into account when developing our application.\nIf we don't have control over the result of the task,\nand we don't want our `LiveView` to crash if the task crashes,\nwe must use a different alternative to spawn our task -\n[`Task.Supervisor.async_nolink/3`](https://hexdocs.pm/elixir/1.14/Task.Supervisor.html#async_nolink/3)\ncan be used for this effect,\nmeaning we can use it if we want to make sure\nour `LiveView` won't die and the error is reported,\neven if the task crashes.\n\nWe've chosen `Task.async/2` for this very reason.\nWe are doing something **that takes time/is expensive**\nand we **want to stop the task if `LiveView` is closed/crashes**.\nHowever, if you are building something\nlike a report that has to be generated even if the person closes the browser tab,\nthis is not the right solution.\n\n##### 4.2.2 Alternative for better testing\n\nWe are spawning async tasks by calling `Task.async/1`.\nThis is creating an **_unsupervised_ task**.\nAlthough it's plausible for this simple app,\nit's best for us to create a\n[**`Supervisor`**](https://hexdocs.pm/elixir/1.15.7/Supervisor.html)\nthat manages their child tasks.\nThis gives more control over the execution\nand lifetime of the child tasks.\n\nAdditionally, it's better to have these tasks supervised\nbecause it makes it possible to create tests for our `LiveView`.\nFor this, we need to make a couple of changes.\n\nFirst, head over to `lib/app/application.ex`\nand add a supervisor to the `start/2` function children array.\n\n```elixir\ndef start(_type, _args) do\n children = [\n AppWeb.Telemetry,\n {Phoenix.PubSub, name: App.PubSub},\n {Nx.Serving, serving: serving(), name: ImageClassifier},\n {Task.Supervisor, name: App.TaskSupervisor}, # add this line\n AppWeb.Endpoint\n ]\n\n opts = [strategy: :one_for_one, name: App.Supervisor]\n Supervisor.start_link(children, opts)\nend\n```\n\nWe are creating a [`Task.Supervisor`](https://hexdocs.pm/elixir/Supervisor.html)\nwith the name `App.TaskSupervisor`.\n\nNow, in `lib/app_web/live/page_live.ex`,\nwe create the async task like so:\n\n```elixir\ntask = Task.Supervisor.async(App.TaskSupervisor, fn -\u003e Nx.Serving.batched_run(ImageClassifier, tensor) end)\n```\n\nWe are now using\n[`Task.Supervisor.async`](https://hexdocs.pm/elixir/1.15.7/Task.Supervisor.html#async/3),\npassing the name of the supervisor defined earlier.\n\nAnd that's it!\nWe are creating async tasks like before,\nthe only difference is that they're now **supervised**.\n\nIn tests, you can create a small module that waits for the tasks to be completed.\n\n```elixir\ndefmodule AppWeb.SupervisorSupport do\n\n @moduledoc \"\"\"\n This is a support module helper that is meant to wait for all the children of a supervisor to complete.\n If you go to `lib/app/application.ex`, you'll see that we created a `TaskSupervisor`, where async tasks are spawned.\n This module helps us to wait for all the children to finish during tests.\n \"\"\"\n\n @doc \"\"\"\n Find all children spawned by this supervisor and wait until they finish.\n \"\"\"\n def wait_for_completion() do\n pids = Task.Supervisor.children(App.TaskSupervisor)\n Enum.each(pids, \u0026Process.monitor/1)\n wait_for_pids(pids)\n end\n\n defp wait_for_pids([]), do: nil\n defp wait_for_pids(pids) do\n receive do\n {:DOWN, _ref, :process, pid, _reason} -\u003e wait_for_pids(List.delete(pids, pid))\n end\n end\nend\n```\n\nYou can call `AppWeb.SupervisorSupport.wait_for_completion()`\nin unit tests so they wait for the tasks to complete.\nIn our case,\nwe do that until the _prediction is made_.\n\n#### 4.3 Image pre-processing\n\nAs we've noted before,\nwe need to **pre-process the image before passing it to the model**.\nFor this, we have three main steps:\n\n- removing the [`alpha` ](https://en.wikipedia.org/wiki/Alpha_compositing)\n out of the image, flattening it out.\n- convert the image to `sRGB` [colourspace](https://en.wikipedia.org/wiki/Color_space).\n This is needed to ensure that the image is consistent\n and aligns with the model's training data images.\n- set the representation of the image as a `Tensor`\n to `height, width, bands`.\n The image tensor will then be organized as a three-dimensional array,\n where the first dimension represents the height of the image,\n the second refers to the width of the image,\n and the third pertains to the different\n [spectral bands/channels of the image](https://en.wikipedia.org/wiki/Multispectral_imaging).\n\nOur `pre_process_image/1` function will implement these three steps.\nLet's implement it now! \u003cbr /\u003e\nIn `lib/app_web/live/page_live.ex`,\nadd the following:\n\n```elixir\n defp pre_process_image(%Vimage{} = image) do\n\n # If the image has an alpha channel, flatten it:\n {:ok, flattened_image} = case Vix.Vips.Image.has_alpha?(image) do\n true -\u003e Vix.Vips.Operation.flatten(image)\n false -\u003e {:ok, image}\n end\n\n # Convert the image to sRGB colourspace ----------------\n {:ok, srgb_image} = Vix.Vips.Operation.colourspace(flattened_image, :VIPS_INTERPRETATION_sRGB)\n\n # Converting image to tensor ----------------\n {:ok, tensor} = Vix.Vips.Image.write_to_tensor(srgb_image)\n\n # We reshape the tensor given a specific format.\n # In this case, we are using {height, width, channels/bands}.\n %Vix.Tensor{data: binary, type: type, shape: {x, y, bands}} = tensor\n format = [:height, :width, :bands]\n shape = {x, y, bands}\n\n final_tensor =\n binary\n |\u003e Nx.from_binary(type)\n |\u003e Nx.reshape(shape, names: format)\n\n {:ok, final_tensor}\n end\n```\n\nThe function receives a `Vix` image,\nas detailed earlier.\nWe use [`flatten/1`](https://hexdocs.pm/vix/Vix.Vips.Operation.html#flatten/2)\nto flatten the alpha out of the image.\n\nThe resulting image has its colourspace changed\nby calling [`colourspace/3`](https://hexdocs.pm/vix/Vix.Vips.Operation.html#colourspace/3),\nwhere we change the to `sRGB`.\n\nThe colourspace-altered image is then converted to a\n[tensor](https://hexdocs.pm/vix/Vix.Tensor.html),\nby calling\n[`write_to_tensor/1`](https://hexdocs.pm/vix/Vix.Vips.Image.html#write_to_tensor/1).\n\nWe then\n[reshape](https://hexdocs.pm/nx/Nx.html#reshape/3)\nthe tensor according to the format that was previously mentioned.\n\nThis function returns the processed tensor,\nthat is then used as input to the model.\n\n### 4.4 Updating the view\n\nAll that's left is updating the view\nto reflect these changes we've made to the `LiveView`.\nHead over to `lib/app_web/live/page_live.html.heex`\nand change it to this.\n\n```html\n\u003c.flash_group flash={@flash} /\u003e\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-28 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv\n class=\"flex justify-center items-center mx-auto max-w-xl w-[50vw] lg:mx-0\"\n \u003e\n \u003cdiv class=\"space-y-12\"\u003e\n \u003cdiv class=\"border-gray-900/10 pb-12\"\u003e\n \u003ch2 class=\"text-base font-semibold leading-7 text-gray-900\"\u003e\n Image Classification\n \u003c/h2\u003e\n \u003cp class=\"mt-1 text-sm leading-6 text-gray-600\"\u003e\n Do simple classification with this\n \u003ca\n href=\"https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html\"\n class=\"font-mono font-medium text-sky-500\"\n \u003eLiveView\u003c/a\n \u003e\n demo, powered by\n \u003ca\n href=\"https://github.com/elixir-nx/bumblebee\"\n class=\"font-mono font-medium text-sky-500\"\n \u003eBumblebee\u003c/a\n \u003e.\n \u003c/p\u003e\n\n \u003c!-- File upload section --\u003e\n \u003cdiv class=\"mt-10 grid grid-cols-1 gap-x-6 gap-y-8 sm:grid-cols-6\"\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n phx-drop-target=\"{@uploads.image_list.ref}\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\n \u003c!-- Prediction text --\u003e\n \u003cdiv\n class=\"mt-6 flex space-x-1.5 items-center font-bold text-gray-900 text-xl\"\n \u003e\n \u003cspan\u003eDescription: \u003c/span\u003e\n \u003c!-- Spinner --\u003e\n \u003c%= if @upload_running? do %\u003e\n \u003cdiv role=\"status\"\u003e\n \u003cdiv\n class=\"relative w-6 h-6 animate-spin rounded-full bg-gradient-to-r from-purple-400 via-blue-500 to-red-400 \"\n \u003e\n \u003cdiv\n class=\"absolute top-1/2 left-1/2 transform -translate-x-1/2 -translate-y-1/2 w-3 h-3 bg-gray-200 rounded-full border-2 border-white\"\n \u003e\u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% else %\u003e \u003c%= if @label do %\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @label %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light\"\u003eWaiting for image input.\u003c/span\u003e\n \u003c% end %\u003e \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nIn these changes,\nwe've added the output of the model in the form of text.\nWe are rendering a spinner\nif the `:upload_running?` socket assign is set to true.\nOtherwise,\nwe add the `:label`, which holds the prediction made by the model.\n\nYou may have also noticed that\nwe've changed the `phx` event handlers\nto `noop`.\nThis is simply to simplify the `LiveView`.\n\nHead over to `lib/app_web/live/page_live.ex`.\nYou can now remove the `\"validate\"`, `\"save\"`\nand `\"remove-selected\"` handlers,\nbecause we're not going to be needing them.\nReplace them with this handler:\n\n```elixir\n @impl true\n def handle_event(\"noop\", _params, socket) do\n {:noreply, socket}\n end\n```\n\n#### 4.5 Check it out!\n\nAnd that's it!\nOur app is now _functional_ πŸŽ‰.\n\nIf you run the app,\nyou can drag and drop or select an image.\nAfter this, a task will be spawned that will run the model\nagainst the image that was submitted.\n\nOnce a prediction is made, display it!\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/aws-sdk-mock/assets/17494745/894b988e-4f60-4781-8838-c7fd95e571f0\" /\u003e\n\u003c/p\u003e\n\nYou can and **should** try other models.\n`ResNet-50` is just one of the many that are supported by `Bumblebee`.\nYou can see the supported models at https://github.com/elixir-nx/bumblebee#model-support.\n\n#### 4.6 Considerations on user images\n\nTo keep the app as simple as possible,\nwe are receiving the image from the person as is.\nAlthough we are processing the image,\nwe are doing it so **it is processable by the model**.\n\nWe have to understand that:\n\n- in most cases, **full-resolution images are not necessary**,\n because neural networks work on much smaller inputs\n (e.g. `ResNet-50` works with `224px x 224px` images).\n This means that a lot of data is unnecessarily uploaded over the network,\n increasing workload on the server to potentially downsize a large image.\n- decoding an image requires an additional package,\n meaning more work on the server.\n\nWe can avoid both of these downsides by moving this work to the client.\nWe can leverage the\n[`Canvas API` ](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API)\nto decode and downsize this image on the client-side,\nreducing server workload.\n\nYou can see an example implementation of this technique\nin `Bumblebee`'s repository\nat https://github.com/elixir-nx/bumblebee/blob/main/examples/phoenix/image_classification.exs\n\nHowever, since we are not using `JavaScript` for anything,\nwe can (and _should_!) properly downsize our images\nso they better fit the training dataset of the model we use.\nThis will allow the model to process faster\nsince larger images carry over more data that is ultimately unnecessary\nfor models to make predictions.\n\nOpen\n`lib/app_web/live/page_live.ex`,\nfind the `handle_progress/3` function\nand change resize the image _before processing it_.\n\n```elixir\n file_binary = File.read!(meta.path)\n\n # Get image and resize\n # This is dependant on the resolution of the model's dataset.\n # In our case, we want the width to be closer to 640, whilst maintaining aspect ratio.\n width = 640\n {:ok, thumbnail_vimage} = Vix.Vips.Operation.thumbnail(meta.path, width, size: :VIPS_SIZE_DOWN)\n\n # Pre-process it\n {:ok, tensor} = pre_process_image(thumbnail_vimage)\n\n #...\n```\n\nWe are using\n[`Vix.Vips.Operation.thumbnail/3`](https://hexdocs.pm/vix/Vix.Vips.Operation.html#thumbnail/3)\nto resize our image to a fixed width\nwhilst maintaining aspect ratio.\nThe `width` variable can be dependent on the model that you use.\nFor example, `ResNet-50` is trained on `224px224` pictures,\nso you may want to resize the image to this width.\n\n\u003e **Note**: We are using the `thumbnail/3` function\n\u003e instead of `resize/3` because it's _much_ faster. \u003cbr /\u003e\n\u003e Check\n\u003e https://github.com/libvips/libvips/wiki/HOWTO----Image-shrinking\n\u003e to know why.\n\n### 5. Final Touches\n\nAlthough our app is functional,\nwe can make it **better**. 🎨\n\n#### 5.1 Setting max file size\n\nIn order to better control user input,\nwe should add a limit to the size of the image that is being uploaded.\nIt will be easier on our server and ultimately save costs.\n\nLet's add a cap of `5MB` to our app!\nFortunately for you, this is super simple!\nYou just need to add the `max_file_size`\nto the `allow_uploads/2` function\nwhen mounting the `LiveView`!\n\n```elixir\n def mount(_params, _session, socket) do\n {:ok,\n socket\n |\u003e assign(label: nil, upload_running?: false, task_ref: nil)\n |\u003e allow_upload(:image_list,\n accept: ~w(image/*),\n auto_upload: true,\n progress: \u0026handle_progress/3,\n max_entries: 1,\n chunk_size: 64_000,\n max_file_size: 5_000_000 # add this\n )}\n end\n```\n\nAnd that's it!\nThe number is in `bytes`,\nhence why we set it as `5_000_000`.\n\n#### 5.2 Show errors\n\nIn case a person uploads an image that is too large,\nwe should show this feedback to the person!\n\nFor this, we can leverage the\n[`upload_errors/2`](https://hexdocs.pm/phoenix_live_view/Phoenix.Component.html#upload_errors/2)\nfunction.\nThis function will return the entry errors for an upload.\nWe need to add a handler for one of these errors to show it first.\n\nHead over `lib/app_web/live/page_live.ex`\nand add the following line.\n\n```elixir\n def error_to_string(:too_large), do: \"Image too large. Upload a smaller image up to 5MB.\"\n```\n\nNow, add the following section below the upload form\ninside `lib/app_web/live/page_live.html.heex`.\n\n```html\n\u003c!-- Show errors --\u003e\n\u003c%= for entry \u003c- @uploads.image_list.entries do %\u003e\n\u003cdiv class=\"mt-2\"\u003e\n \u003c%= for err \u003c- upload_errors(@uploads.image_list, entry) do %\u003e\n \u003cdiv class=\"rounded-md bg-red-50 p-4 mb-2\"\u003e\n \u003cdiv class=\"flex\"\u003e\n \u003cdiv class=\"flex-shrink-0\"\u003e\n \u003csvg\n class=\"h-5 w-5 text-red-400\"\n viewBox=\"0 0 20 20\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M10 18a8 8 0 100-16 8 8 0 000 16zM8.28 7.22a.75.75 0 00-1.06 1.06L8.94 10l-1.72 1.72a.75.75 0 101.06 1.06L10 11.06l1.72 1.72a.75.75 0 101.06-1.06L11.06 10l1.72-1.72a.75.75 0 00-1.06-1.06L10 8.94 8.28 7.22z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003c/div\u003e\n \u003cdiv class=\"ml-3\"\u003e\n \u003ch3 class=\"text-sm font-medium text-red-800\"\u003e\n \u003c%= error_to_string(err) %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n\u003c/div\u003e\n\u003c% end %\u003e\n```\n\nWe are iterating over the errors returned by `upload_errors/2`\nand invoking `error_to_string/1`,\nwhich we've just defined in our `LiveView`.\n\nNow, if you run the app\nand try to upload an image that is too large,\nan error will show up.\n\nAwesome! πŸŽ‰\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=800 src=\"https://github.com/dwyl/aws-sdk-mock/assets/17494745/1bf903eb-31d5-48a4-9da9-1f5f64932b6e\" /\u003e\n\u003c/p\u003e\n\n#### 5.3 Show image preview\n\nAs of now, even though our app predicts the given images,\nit is not showing a preview of the image the person submitted.\nLet's fix this πŸ› οΈ.\n\nLet's add a new socket assign variable\npertaining to the [base64](https://en.wikipedia.org/wiki/Base64) representation\nof the image in `lib/app_web/live_page/live.ex`\n\n```elixir\n |\u003e assign(label: nil, upload_running?: false, task_ref: nil, image_preview_base64: nil)\n```\n\nWe've added `image_preview_base64`\nas a new socket assign,\ninitializing it as `nil`.\n\nNext, we need to _read the file while consuming it_,\nand properly update the socket assign\nso we can show it to the person.\n\nIn the same file,\nchange the `handle_progress/3` function to the following.\n\n```elixir\n def handle_progress(:image_list, entry, socket) when entry.done? do\n # Consume the entry and get the tensor to feed to classifier\n %{tensor: tensor, file_binary: file_binary} = consume_uploaded_entry(socket, entry, fn %{} = meta -\u003e\n file_binary = File.read!(meta.path)\n\n {:ok, vimage} = Vix.Vips.Image.new_from_file(meta.path)\n {:ok, tensor} = pre_process_image(vimage)\n {:ok, %{tensor: tensor, file_binary: file_binary}}\n end)\n\n # Create an async task to classify the image\n task = Task.Supervisor.async(App.TaskSupervisor, fn -\u003e Nx.Serving.batched_run(ImageClassifier, tensor) end)\n\n # Encode the image to base64\n base64 = \"data:image/png;base64, \" \u003c\u003e Base.encode64(file_binary)\n\n # Update socket assigns to show spinner whilst task is running\n {:noreply, assign(socket, upload_running?: true, task_ref: task.ref, image_preview_base64: base64)}\n end\n```\n\nWe're using [`File.read!/1`](https://hexdocs.pm/elixir/1.13/File.html#read/1)\nto retrieve the binary representation of the image that was uploaded.\nWe use [`Base.encode64/2`](https://hexdocs.pm/elixir/1.12/Base.html#encode64/2)\nto encode this file binary\nand assign the newly created `image_preview_base64` socket assign\nwith this base64 representation of the image.\n\nNow, all that's left to do\nis to _render the image on our view_.\nIn `lib/app_web/live/page_live.html.heex`,\nlocate the line:\n\n```html\n\u003cdiv class=\"text-center\"\u003e\u003c/div\u003e\n```\n\nWe are going to update this `\u003cdiv\u003e`\nto show the image with the `image_preview_base64` socket assign.\n\n```html\n\u003cdiv class=\"text-center\"\u003e\n \u003c!-- Show image preview --\u003e\n \u003c%= if @image_preview_base64 do %\u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list} class=\"hidden\" /\u003e\n \u003cimg src=\"{@image_preview_base64}\" /\u003e\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c% else %\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list} class=\"hidden\" /\u003e\n Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003ePNG, JPG, GIF up to 5MB\u003c/p\u003e\n \u003c% end %\u003e\n\u003c/div\u003e\n```\n\nAs you can see,\nwe are checking if `@image_preview_base64` is defined.\nIf so, we simply show the image with it as `src` 😊.\n\nNow, if you run the application,\nyou'll see that after dragging the image,\nit is previewed and shown to the person!\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=800 src=\"https://github.com/dwyl/image-classifier/assets/17494745/2835c24f-f4ba-48bc-aab0-6b39830156ce\" /\u003e\n\u003c/p\u003e\n\n### 6. What about other models?\n\nMaybe you weren't happy with the results from this model.\n\nThat's fair. `ResNet-50` is a smaller, \"older\" model compared to other\nimage captioning/classification models.\n\nWhat if you wanted to use others?\nWell, as we've mentioned before,\n`Bumblebee` uses\n[**Transformer models from `HuggingFace`**](https://huggingface.co/docs/transformers/index).\nTo know if one is supported\n(as shown in [`Bumblebee`'s docs](https://github.com/elixir-nx/bumblebee#model-support)),\nwe need to check the `config.json` file\nin the model repository\nand copy the class name under `\"architectures\"`\nand search it on `Bumblebee`'s codebase.\n\nFor example,\nhere's one of the more popular image captioning models -\nSalesforce's `BLIP` -\nhttps://huggingface.co/Salesforce/blip-image-captioning-large/blob/main/config.json.\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"48%\" src=\"https://github.com/elixir-nx/bumblebee/assets/17494745/33dc869f-37f7-4d18-b126-3a0bd0d578d3\" /\u003e\n \u003cimg width=\"48%\" src=\"https://github.com/elixir-nx/bumblebee/assets/17494745/8f1d115c-171b-42bf-b974-08172c957a09\" /\u003e\n\u003c/p\u003e\n\nIf you visit `Bumblebee`'s codebase\nand search for the class name,\nyou'll find it is supported.\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"800\" src=\"https://github.com/elixir-nx/bumblebee/assets/17494745/500eb97b-c20a-4c9a-846e-327cdcd1c37c\" /\u003e\n\u003c/p\u003e\n\nAwesome!\nNow we can use it!\n\nIf you dig around `Bumblebee`'s docs as well\n(https://hexdocs.pm/bumblebee/Bumblebee.Vision.html#image_to_text/5),\nyou'll see that we've got to use `image_to_text/5` with this model.\nIt needs a `tokenizer`, `featurizer` and a `generation-config`\nso we can use it.\n\nLet's do it!\nHead over to `lib/app/application.ex`,\nand change the `serving/0` function.\n\n```elixir\n def serving do\n {:ok, model_info} = Bumblebee.load_model({:hf, \"Salesforce/blip-image-captioning-base\"})\n {:ok, featurizer} = Bumblebee.load_featurizer({:hf, \"Salesforce/blip-image-captioning-base\"})\n {:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, \"Salesforce/blip-image-captioning-base\"})\n {:ok, generation_config} = Bumblebee.load_generation_config({:hf, \"Salesforce/blip-image-captioning-base\"})\n\n Bumblebee.Vision.image_to_text(model_info, featurizer, tokenizer, generation_config,\n compile: [batch_size: 10],\n defn_options: [compiler: EXLA]\n )\n end\n```\n\nAs you can see, we're using the repository name of `BLIP`'s model\nfrom the HuggingFace website.\n\nIf you run `mix phx.server`,\nyou'll see that it will download the new models,\ntokenizers, featurizer and configs to run the model.\n\n```sh\n|======================================================================| 100% (989.82 MB)\n[info] TfrtCpuClient created.\n|======================================================================| 100% (711.39 KB)\n[info] Running AppWeb.Endpoint with cowboy 2.10.0 at 127.0.0.1:4000 (http)\n[info] Access AppWeb.Endpoint at http://localhost:4000\n[watch] build finished, watching for changes...\n```\n\nYou may think we're done here.\nBut we are not! βœ‹\n\nThe **destructuring of the output of the model may not be the same**. \u003cbr /\u003e\nIf you try to submit a photo,\nyou'll get this error:\n\n```sh\nno match of right hand side value:\n%{results: [%{text: \"a person holding a large blue ball on a beach\"}]}\n```\n\nThis means that we need to make some changes\nwhen parsing the output of the model πŸ˜€.\n\nHead over to `lib/app_web/live/page_live.ex`\nand change the `handle_info/3` function\nthat is called after the async task is completed.\n\n```elixir\n def handle_info({ref, result}, %{assigns: %{task_ref: ref}} = socket) do\n Process.demonitor(ref, [:flush])\n\n %{results: [%{text: label}]} = result # change this line\n\n {:noreply, assign(socket, label: label, upload_running?: false)}\n end\n```\n\nAs you can see, we are now correctly destructuring the result from the model.\nAnd that's it!\n\nIf you run `mix phx.server`,\nyou'll see that we got far more accurate results!\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"800\" src=\"https://github.com/elixir-nx/bumblebee/assets/17494745/0cae2db0-5ca4-4434-9c63-76aadb7d578b\" /\u003e\n\u003c/p\u003e\n\nAwesome! πŸŽ‰\n\n\u003e [!NOTE]\n\u003e Be aware that `BLIP`\n\u003e is a _much_ larger model than `ResNet-50`.\n\u003e There are more accurate and even larger models out there\n\u003e (e.g:\n\u003e [`blip-image-captioning-large`](https://huggingface.co/Salesforce/blip-image-captioning-large),\n\u003e the larger version of the model we've just used).\n\u003e This is a balancing act: the larger the model, the longer a prediction may take\n\u003e and more resources your server will need to have to handle this heavier workload.\n\n\u003e [!WARNING]\n\u003e\n\u003e We've created a small module that allows you to have multiple models\n\u003e cached and downloaded and keep this logic contained.\n\u003e\n\u003e For this, check the [`deployment guide`](./deployment.md#5-a-better-model-management).\n\n### 7. How do I deploy this thing?\n\nThere are a few considerations you may want to have\nbefore considering deploying this.\nLuckily for you,\nwe've created a small document\nthat will **guide you through deploying this app in `fly.io`**!\n\nCheck the [`deployment.md`](./deployment.md) file for more information.\n\n### 8. Showing example images\n\n\u003e [!WARNING]\n\u003e\n\u003e This section assumes you've made the changes made in the previous section.\n\u003e Therefore, you should follow the instructions in\n\u003e [`7. How do I deploy this thing?`](#7-how-do-i-deploy-this-thing)\n\u003e and come back after you're done.\n\nWe have a fully functioning application that predicts images.\nNow we can add some cool touches to show the person\nsome examples if they are inactive.\n\nFor this,\nwe are going to need to make **three changes**.\n\n- create a hook in the **client** (`Javascript`)\n to send an event when there's inactivity after\n a given number of seconds.\n- change `page_live.ex` **LiveView** to accommodate\n this new event.\n- change the **view** m `page_live.html.heex`\n to show these changes to the person.\n\nLet's go over each one!\n\n#### 8.1 Creating a hook in client\n\nWe are going to detect the inactivity of the person\nwith some `Javascript` code.\n\nHead over to `assets/js/app.js`\nand change it to the following.\n\n```js\n// If you want to use Phoenix channels, run `mix help phx.gen.channel`\n// to get started and then uncomment the line below.\n// import \"./user_socket.js\"\n\n// You can include dependencies in two ways.\n//\n// The simplest option is to put them in assets/vendor and\n// import them using relative paths:\n//\n// import \"../vendor/some-package.js\"\n//\n// Alternatively, you can `npm install some-package --prefix assets` and import\n// them using a path starting with the package name:\n//\n// import \"some-package\"\n//\n\n// Include phoenix_html to handle method=PUT/DELETE in forms and buttons.\nimport \"phoenix_html\";\n// Establish Phoenix Socket and LiveView configuration.\nimport { Socket } from \"phoenix\";\nimport { LiveSocket } from \"phoenix_live_view\";\nimport topbar from \"../vendor/topbar\";\n\n// Hooks to track inactivity\nlet Hooks = {};\nHooks.ActivityTracker = {\n mounted() {\n // Set the inactivity duration in milliseconds\n const inactivityDuration = 8000; // 8 seconds\n\n // Set a variable to keep track of the timer and if the process to predict example image has already been sent\n let inactivityTimer;\n let processHasBeenSent = false;\n\n let ctx = this;\n\n // Function to reset the timer\n function resetInactivityTimer() {\n // Clear the previous timer\n clearTimeout(inactivityTimer);\n\n // Start a new timer\n inactivityTimer = setTimeout(() =\u003e {\n // Perform the desired action after the inactivity duration\n // For example, send a message to the Elixir process using Phoenix Socket\n if (!processHasBeenSent) {\n processHasBeenSent = true;\n ctx.pushEvent(\"show_examples\", {});\n }\n }, inactivityDuration);\n }\n\n // Call the function to start the timer initially\n resetInactivityTimer();\n\n // Reset the timer whenever there is user activity\n document.addEventListener(\"mousemove\", resetInactivityTimer);\n document.addEventListener(\"keydown\", resetInactivityTimer);\n },\n};\n\nlet csrfToken = document\n .querySelector(\"meta[name='csrf-token']\")\n .getAttribute(\"content\");\nlet liveSocket = new LiveSocket(\"/live\", Socket, {\n hooks: Hooks,\n params: { _csrf_token: csrfToken },\n});\n\n// Show progress bar on live navigation and form submits\ntopbar.config({ barColors: { 0: \"#29d\" }, shadowColor: \"rgba(0, 0, 0, .3)\" });\nwindow.addEventListener(\"phx:page-loading-start\", (_info) =\u003e topbar.show(300));\nwindow.addEventListener(\"phx:page-loading-stop\", (_info) =\u003e topbar.hide());\n\n// connect if there are any LiveViews on the page\nliveSocket.connect();\n\n// expose liveSocket on window for web console debug logs and latency simulation:\n// \u003e\u003e liveSocket.enableDebug()\n// \u003e\u003e liveSocket.enableLatencySim(1000) // enabled for duration of browser session\n// \u003e\u003e liveSocket.disableLatencySim()\nwindow.liveSocket = liveSocket;\n```\n\n- we have added a `Hooks` variable,\n with a property **`ActivityTracker`**.\n This hook has the `mounted()` function\n that is executed when the component that is hooked with this hook is mounted.\n You can find more information at https://hexdocs.pm/phoenix_live_view/js-interop.html.\n- inside the `mounted()` function,\n we create a `resetInactivityTimer()` function\n that is executed every time the\n **mouse moves** (`mousemove` event)\n and a **key is pressed**(`keydown`).\n This function resets the timer\n that is run whilst there is a lack of inactivity.\n- if the person is inactive for _8 seconds_,\n we create an event `\"show_examples\"`.\n We will create a handler in the LiveView\n to handle this event later.\n- we add the `Hooks` variable to the `hooks`\n property when initializing the `livesocket`.\n\nAnd that's it!\n\nFor this hook to _actually be executed_,\nwe need to create a component that uses it\ninside our view file.\n\nFor this, we can simply create a hidden component\non top of the `lib/app_web/live/page_live.html.heex` file.\n\nAdd the following hidden component.\n\n```html\n\u003cdiv class=\"hidden\" id=\"tracker_el\" phx-hook=\"ActivityTracker\" /\u003e\n```\n\nWe use the `phx-hook` attribute\nto bind the hook we've created in our `app.js` file\nto the component so it's executed.\nWhen this component is mounted,\nthe `mounted()` function inside the hook\nis executed.\n\nAnd that's it! πŸ‘\n\nYour app won't work yet because\nwe haven't created a handler\nto handle the `\"show_examples\"` event.\n\nLet's do that right now!\n\n### 8.2 Handling the example images list event inside our LiveView\n\nNow that we have our client sorted,\nlet's head over to our LiveView\nat `lib/app_web/live/page_live.ex`\nand make the needed changes!\n\nBefore anything,\nlet's add socket assigns that we will need throughout our adventure!\nOn top of the file,\nchange the socket assigns to the following.\n\n```elixir\n socket\n |\u003e assign(\n # Related to the file uploaded by the user\n label: nil,\n upload_running?: false,\n task_ref: nil,\n image_preview_base64: nil,\n\n # Related to the list of image examples\n example_list_tasks: [],\n example_list: [],\n display_list?: false\n )\n```\n\nWe've added **three new assigns**:\n\n- **`example_list_tasks`** is a list of\n the async tasks that are created for each example image.\n- **`example_list`** is a list of the example images\n with their respective predictions.\n- **`display_list?`** is a boolean that\n tells us if the list is to be shown or not.\n\nAwesome! Let's continue.\n\nAs we've mentioned before,\nwe need to create a handler for our `\"show_examples\"` event.\n\nAdd the following function to the file.\n\n```elixir\n\n @image_width 640\n\n def handle_event(\"show_examples\", _data, socket) do\n\n # Only run if the user hasn't uploaded anything\n if(is_nil(socket.assigns.task_ref)) do\n # Retrieves a random image from Picsum with a given `image_width` dimension\n random_image = \"https://picsum.photos#{@image_width}/#{@image_width}\"\n\n # Spawns prediction tasks for example image from random Picsum image\n tasks = for _ \u003c- 1..2 do\n {:req, body} = {:req, Req.get!(random_image).body}\n predict_example_image(body)\n end\n\n\n # List to change `example_list` socket assign to show skeleton loading\n display_example_images = Enum.map(tasks, fn obj -\u003e %{predicting?: true, ref: obj.ref} end)\n\n # Updates the socket assigns\n {:noreply, assign(socket, example_list_tasks: tasks, example_list: display_example_images)}\n\n else\n {:noreply, socket}\n end\n end\n\n```\n\n\u003e [!WARNING]\n\u003e\n\u003e We are using the\n\u003e [`req`](https://github.com/wojtekmach/req) package\n\u003e to download the file binary from the URL.\n\u003e Make sure to install it in the `mix.exs` file.\n\n- we are using the [Picsum API](https://picsum.photos),\n an _awesome_ image API with lots of photos!\n They provide a `/random` URL that yields a random photo.\n In this URL we can inclusively define the dimensions we want!\n That's what we're doing in the first line of the function.\n We are using a module constant `@image_width 640` on top of the file,\n so add that to the top of the file.\n This function is relevant because we preferably want to deal\n with images that are in the same resolution as the dataset the model was trained in.\n- we are creating **two async tasks** that retrieve the binary of the image\n and pass it on to a `predict_example_image/1` function\n (we will create this function next).\n- the two tasks that we've created are in an array `tasks`.\n We create _another array_ `display_example_images` with the same number of elements as `tasks`,\n which will have two properties:\n **`predicting`**, meaning if the image is being predicted by the model;\n and **`ref`**, the reference of the task.\n- we assign the `tasks` array to the `example_list_tasks` socket assign\n and `display_example_images` array to the `example_list` array.\n So `example_list` will temporarily hold\n objects with `:predicting` and `:ref` properties\n whilst the model is being executed.\n\nAs we've just mentioned,\nwe are making use of a function called\n`predict_example_image/1` to make predictions\nof a given file binary.\n\nLet's implement it now!\nIn the same file, add:\n\n```elixir\n def predict_example_image(body) do\n with {:vix, {:ok, img_thumb}} \u003c-\n {:vix, Vix.Vips.Operation.thumbnail_buffer(body, @image_width)},\n {:pre_process, {:ok, t_img}} \u003c-\n {:pre_process, pre_process_image(img_thumb)} do\n\n # Create an async task to classify the image from Picsum\n Task.Supervisor.async(App.TaskSupervisor, fn -\u003e\n Nx.Serving.batched_run(ImageClassifier, t_img)\n end)\n |\u003e Map.merge(%{base64_encoded_url: \"data:image/png;base64, \" \u003c\u003e Base.encode64(body)})\n\n else\n {stage, error} -\u003e {stage, error}\n end\n end\n```\n\nFor the body of the image to be executed by the model,\nit needs to go through some pre-processing.\n\n- we are using [`thumbnail_buffer/3`](https://hexdocs.pm/vix/Vix.Vips.Operation.html#thumbnail_buffer/3)\n to make sure it's properly resized\n and then feeding the result to our own implemented\n `pre_process_image/1` function\n so it can be converted to a parseable tensor by the model.\n- after these two operations are successfully completed,\n we spawn two async tasks (like we've done before)\n and feed it to the model.\n We add the base64-encoded image\n to the return value so it can later be shown to the person.\n- if these operations fail, we return an error.\n\nGreat job! πŸ‘\n\nOur example images async tasks have successfully been created\nand are on their way to the model!\n\nNow we need to handle these newly created async tasks\nonce they are completed.\nAs we know, we are handling our async tasks completion\nin the `def handle_info({ref, result}, %{assigns: %{task_ref: ref}} = socket)` function.\nLet's change it like so.\n\n```elixir\n def handle_info({ref, result}, %{assigns: assigns} = socket) do\n # Flush async call\n Process.demonitor(ref, [:flush])\n\n # You need to change how you destructure the output of the model depending\n # on the model you've chosen for `prod` and `test` envs on `models.ex`.)\n label =\n case Application.get_env(:app, :use_test_models, false) do\n true -\u003e\n App.Models.extract_test_label(result)\n\n # coveralls-ignore-start\n false -\u003e\n App.Models.extract_prod_label(result)\n # coveralls-ignore-stop\n end\n\n cond do\n\n # If the upload task has finished executing, we update the socket assigns.\n Map.get(assigns, :task_ref) == ref -\u003e\n {:noreply, assign(socket, label: label, upload_running?: false)}\n\n # If the example task has finished executing, we upload the socket assigns.\n img = Map.get(assigns, :example_list_tasks) |\u003e Enum.find(\u0026(\u00261.ref == ref)) -\u003e\n\n # Update the element in the `example_list` enum to turn \"predicting?\" to `false`\n updated_example_list = Map.get(assigns, :example_list)\n |\u003e Enum.map(fn obj -\u003e\n if obj.ref == img.ref do\n Map.put(obj, :base64_encoded_url, img.base64_encoded_url)\n |\u003e Map.put(:label, label)\n |\u003e Map.put(:predicting?, false)\n\n else\n obj\n end end)\n\n {:noreply,\n assign(socket,\n example_list: updated_example_list,\n upload_running?: false,\n display_list?: true\n )}\n end\n end\n```\n\nThe only change we've made is that\nwe've added a [`cond`](https://hexdocs.pm/elixir/1.16/case-cond-and-if.html#cond)\nflow structure.\nWe are essentially checking\nif the task reference that has been completed\nis **from the uploaded image from the person** (`:task_ref` socket assign)\nor **from an example image** (inside the `:example_list_tasks` socket assign list).\n\nIf it's the latter,\nwe retrieve we are updating\nthe `example_list` socket assign list\nwith the **prediction** (`:label`),\nthe **base64-encoded image from the task list** (`:base64_encoded_url`)\nand setting the `:predicting` property to `false`.\n\nAnd that's it!\nGreat job! πŸ₯³\n\n#### 8.3 Updating the view\n\nNow that we've made all the necessary changes to our LiveView,\nwe need to update our view so it reflects them!\n\nHead over to `lib/app_web/live/page_live.html.heex`\nand change it to the following piece of code.\n\n```html\n\u003cdiv class=\"hidden\" id=\"tracker_el\" phx-hook=\"ActivityTracker\" /\u003e\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-24 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv class=\"flex flex-col justify-start\"\u003e\n \u003cdiv class=\"flex justify-center items-center w-full\"\u003e\n \u003cdiv class=\"2xl:space-y-12\"\u003e\n \u003cdiv class=\"mx-auto max-w-2xl lg:text-center\"\u003e\n \u003cp\u003e\n \u003cspan\n class=\"rounded-full w-fit bg-brand/5 px-2 py-1 text-[0.8125rem] font-medium text-center leading-6 text-brand\"\n \u003e\n \u003ca\n href=\"https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n \u003e\n πŸ”₯ LiveView\n \u003c/a\u003e\n +\n \u003ca\n href=\"https://github.com/elixir-nx/bumblebee\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n \u003e\n 🐝 Bumblebee\n \u003c/a\u003e\n \u003c/span\u003e\n \u003c/p\u003e\n \u003cp\n class=\"mt-2 text-3xl font-bold tracking-tight text-gray-900 sm:text-4xl\"\n \u003e\n Caption your image!\n \u003c/p\u003e\n \u003ch3 class=\"mt-6 text-lg leading-8 text-gray-600\"\u003e\n Upload your own image (up to 5MB) and perform image captioning with\n \u003ca\n href=\"https://elixir-lang.org/\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n class=\"font-mono font-medium text-sky-500\"\n \u003e\n Elixir\n \u003c/a\u003e\n !\n \u003c/h3\u003e\n \u003cp class=\"text-lg leading-8 text-gray-400\"\u003e\n Powered with\n \u003ca\n href=\"https://elixir-lang.org/\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n class=\"font-mono font-medium text-sky-500\"\n \u003e\n HuggingFaceπŸ€—\n \u003c/a\u003e\n transformer models, you can run this project locally and perform\n machine learning tasks with a handful lines of code.\n \u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv class=\"border-gray-900/10\"\u003e\n \u003c!-- File upload section --\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n phx-drop-target=\"{@uploads.image_list.ref}\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003c!-- Show image preview --\u003e\n \u003c%= if @image_preview_base64 do %\u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e\n \u003cimg src=\"{@image_preview_base64}\" /\u003e\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c% else %\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c!-- Show errors --\u003e\n \u003c%= for entry \u003c- @uploads.image_list.entries do %\u003e\n \u003cdiv class=\"mt-2\"\u003e\n \u003c%= for err \u003c- upload_errors(@uploads.image_list, entry) do %\u003e\n \u003cdiv class=\"rounded-md bg-red-50 p-4 mb-2\"\u003e\n \u003cdiv class=\"flex\"\u003e\n \u003cdiv class=\"flex-shrink-0\"\u003e\n \u003csvg\n class=\"h-5 w-5 text-red-400\"\n viewBox=\"0 0 20 20\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M10 18a8 8 0 100-16 8 8 0 000 16zM8.28 7.22a.75.75 0 00-1.06 1.06L8.94 10l-1.72 1.72a.75.75 0 101.06 1.06L10 11.06l1.72 1.72a.75.75 0 101.06-1.06L11.06 10l1.72-1.72a.75.75 0 00-1.06-1.06L10 8.94 8.28 7.22z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003c/div\u003e\n \u003cdiv class=\"ml-3\"\u003e\n \u003ch3 class=\"text-sm font-medium text-red-800\"\u003e\n \u003c%= error_to_string(err) %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c!-- Prediction text --\u003e\n \u003cdiv\n class=\"flex mt-2 space-x-1.5 items-center font-bold text-gray-900 text-xl\"\n \u003e\n \u003cspan\u003eDescription: \u003c/span\u003e\n \u003c!-- Spinner --\u003e\n \u003c%= if @upload_running? do %\u003e\n \u003cdiv role=\"status\"\u003e\n \u003cdiv\n class=\"relative w-6 h-6 animate-spin rounded-full bg-gradient-to-r from-purple-400 via-blue-500 to-red-400 \"\n \u003e\n \u003cdiv\n class=\"absolute top-1/2 left-1/2 transform -translate-x-1/2 -translate-y-1/2 w-3 h-3 bg-gray-200 rounded-full border-2 border-white\"\n \u003e\u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% else %\u003e \u003c%= if @label do %\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @label %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light\"\u003eWaiting for image input.\u003c/span\u003e\n \u003c% end %\u003e \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c!-- Examples --\u003e\n \u003c%= if @display_list? do %\u003e\n \u003cdiv class=\"flex flex-col\"\u003e\n \u003ch3\n class=\"mt-10 text-xl lg:text-center font-light tracking-tight text-gray-900 lg:text-2xl\"\n \u003e\n Examples\n \u003c/h3\u003e\n \u003cdiv class=\"flex flex-row justify-center my-8\"\u003e\n \u003cdiv\n class=\"mx-auto grid max-w-2xl grid-cols-1 gap-x-6 gap-y-20 sm:grid-cols-2\"\n \u003e\n \u003c%= for example_img \u003c- @example_list do %\u003e\n\n \u003c!-- Loading skeleton if it is predicting --\u003e\n \u003c%= if example_img.predicting? == true do %\u003e\n \u003cdiv\n role=\"status\"\n class=\"flex items-center justify-center w-full h-full max-w-sm bg-gray-300 rounded-lg animate-pulse\"\n \u003e\n \u003csvg\n class=\"w-10 h-10 text-gray-200 dark:text-gray-600\"\n aria-hidden=\"true\"\n xmlns=\"http://www.w3.org/2000/svg\"\n fill=\"currentColor\"\n viewBox=\"0 0 20 18\"\n \u003e\n \u003cpath\n d=\"M18 0H2a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V2a2 2 0 0 0-2-2Zm-5.5 4a1.5 1.5 0 1 1 0 3 1.5 1.5 0 0 1 0-3Zm4.376 10.481A1 1 0 0 1 16 15H4a1 1 0 0 1-.895-1.447l3.5-7A1 1 0 0 1 7.468 6a.965.965 0 0 1 .9.5l2.775 4.757 1.546-1.887a1 1 0 0 1 1.618.1l2.541 4a1 1 0 0 1 .028 1.011Z\"\n /\u003e\n \u003c/svg\u003e\n \u003cspan class=\"sr-only\"\u003eLoading...\u003c/span\u003e\n \u003c/div\u003e\n\n \u003c% else %\u003e\n \u003cdiv\u003e\n \u003cimg\n id=\"{example_img.base64_encoded_url}\"\n src=\"{example_img.base64_encoded_url}\"\n class=\"rounded-2xl object-cover\"\n /\u003e\n \u003ch3 class=\"mt-1 text-lg leading-8 text-gray-900 text-center\"\u003e\n \u003c%= example_img.label %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c% end %\u003e \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nWe've made two changes.\n\n- we've added some text to better introduce our application\n at the top of the page.\n- added a section to show the example images list.\n This section is only rendered if the\n **`display_list?`** socket assign is set to `true`.\n If so, we iterate over the\n **`example_list`** socket assign list\n and show a [loading skeleton](https://www.freecodecamp.org/news/how-to-build-skeleton-screens-using-css-for-better-user-experience/)\n if the image is `:predicting`.\n If not, it means the image has already been predicted,\n and we show the base64-encoded image\n like we do with the image uploaded by the person.\n\nAnd that's it! πŸŽ‰\n\n#### 8.4 Using URL of image instead of base64-encoded\n\nWhile our example list is being correctly rendered,\nwe are using additional CPU\nto base64 encode our images\nso they can be shown to the person.\n\nInitially, we did this because\n`https://picsum.photos`\nresolves into a different URL every time it is called.\nThis means that the image that was fed into the model\nwould be different from the one shown in the example list\nif we were to use this URL in our view.\n\nTo fix this,\n**we need to follow the redirection when the URL is resolved**.\n\n\u003e [!NOTE]\n\u003e We can do this with\n\u003e [`Finch`](https://github.com/sneako/finch),\n\u003e if we wanted to.\n\u003e\n\u003e We could do something like\n\u003e def rand_splash do\n\u003e\n\u003e ```elixir\n\u003e %{scheme: scheme, host: host, path: path} =\n\u003e Finch.build(:get, \"https://picsum.photos\")\n\u003e |\u003e Finch.request!(MyFinch)\n\u003e |\u003e Map.get(:headers)\n\u003e |\u003e Enum.filter(fn {a, _b} -\u003e a == \"location\" end)\n\u003e |\u003e List.first()\n\u003e |\u003e elem(1)\n\u003e |\u003e URI.parse()\n\u003e\n\u003e scheme \u003c\u003e \"://\" \u003c\u003e host \u003c\u003e path\n\u003e end\n\u003e ```\n\u003e\n\u003e And then call it, like so.\n\u003e\n\u003e ```elixir\n\u003e App.rand_splash()\n\u003e # https://images.unsplash.com/photo-1694813646634-9558dc7960e3\n\u003e ```\n\nBecause we are already using\n[`req`](https://github.com/wojtekmach/req),\nlet's make use of it\ninstead of adding additional dependencies.\n\nLet's first add a function that will do this\nin `lib/app_web/live/page_live.ex`.\nAdd the following piece of code\nat the end of the file.\n\n```elixir\n defp track_redirected(url) do\n # Create request\n req = Req.new(url: url)\n\n # Add tracking properties to req object\n req = req\n |\u003e Req.Request.register_options([:track_redirected])\n |\u003e Req.Request.prepend_response_steps(track_redirected: \u0026track_redirected_uri/1)\n\n # Make request\n {:ok, response} = Req.request(req)\n\n # Return the final URI\n %{url: URI.to_string(response.private.final_uri), body: response.body}\n end\n\n defp track_redirected_uri({request, response}) do\n {request, %{response | private: Map.put(response.private, :final_uri, request.url)}}\n end\n```\n\nThis function adds properties to the request object\nand tracks the redirection.\nIt will add a [`URI`](https://hexdocs.pm/elixir/1.12/URI.html#summary)\nobject inside `private.final_uri`.\nThis function returns\nthe `body` of the image\nand the final `url`\nit is resolved to\n(the URL of the image).\n\nNow all we need to do is use this function!\nHead over to the `handle_event(\"show_examples\"...` function\nand change the loop to the following.\n\n```elixir\n tasks = for _ \u003c- 1..2 do\n %{url: url, body: body} = track_redirected(random_image)\n predict_example_image(body, url)\n end\n```\n\nWe are making use of `track_redirected/1`,\nthe function we've just created.\nWe pass both `body` and `url` to `predict_example_image/1`,\nwhich we will now change.\n\n```elixir\n def predict_example_image(body, url) do\n with {:vix, {:ok, img_thumb}} \u003c-\n {:vix, Vix.Vips.Operation.thumbnail_buffer(body, @image_width)},\n {:pre_process, {:ok, t_img}} \u003c- {:pre_process, pre_process_image(img_thumb)} do\n\n # Create an async task to classify the image from Picsum\n Task.Supervisor.async(App.TaskSupervisor, fn -\u003e\n Nx.Serving.batched_run(ImageClassifier, t_img)\n end)\n |\u003e Map.merge(%{url: url})\n\n else\n {:vix, {:error, msg}} -\u003e {:error, msg}\n {:pre_process, {:error, msg}} -\u003e {:error, msg}\n end\n end\n```\n\nInstead of using `base64_encoded_url`,\nwe are now using the `url` we've acquired.\n\nThe last step we need to do in our LiveView\nis to finally use this `url`\nin `handle_info/3`.\n\n```elixir\n def handle_info({ref, result}, %{assigns: assigns} = socket) do\n Process.demonitor(ref, [:flush])\n\n label =\n case Application.get_env(:app, :use_test_models, false) do\n true -\u003e\n App.Models.extract_test_label(result)\n\n false -\u003e\n App.Models.extract_prod_label(result)\n end\n\n cond do\n\n Map.get(assigns, :task_ref) == ref -\u003e\n {:noreply, assign(socket, label: label, upload_running?: false)}\n\n img = Map.get(assigns, :example_list_tasks) |\u003e Enum.find(\u0026(\u00261.ref == ref)) -\u003e\n\n updated_example_list = Map.get(assigns, :example_list)\n |\u003e Enum.map(fn obj -\u003e\n if obj.ref == img.ref do\n obj\n |\u003e Map.put(:url, img.url) # change here\n |\u003e Map.put(:label, label)\n |\u003e Map.put(:predicting?, false)\n\n else\n obj\n end end)\n\n {:noreply,\n assign(socket,\n example_list: updated_example_list,\n upload_running?: false,\n display_list?: true\n )}\n end\n end\n```\n\nAnd that's it!\n\nThe last thing we need to do is change our view\nso it uses the `:url` parameter\ninstead of the obsolete `:base64_encoded_url`.\n\nHead over to `lib/app_web/live/page_live.html.heex`\nand change the `\u003cimg\u003e` being shown in the example list\nso it uses the `:url` parameter.\n\n```html\n\u003cimg\n id=\"{example_img.url}\"\n src=\"{example_img.url}\"\n class=\"rounded-2xl object-cover\"\n/\u003e\n```\n\nAnd we're done! πŸŽ‰\n\nWe are now rendering the image on the client\nthrough the URL the Picsum API resolves into\ninstead of having the LiveView server\nencoding the image.\nTherefore, we're saving some CPU\nto the thing that matters the most:\n_running our model_.\n\n#### 8.5 See it running\n\nNow let's see our application in action!\nWe are expecting the examples to be shown after\n**8 seconds** of inactivity.\nIf the person is inactive for this time duration,\nwe fetch a random image from Picsum API\nand feed it to our model!\n\nYou should see different images every time you use the app.\nIsn't that cool? 😎\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=800 src=\"https://github.com/dwyl/image-classifier/assets/17494745/1f8d08d1-f6ca-46aa-8c89-4bab45ad1e54\"\u003e\n\u003c/p\u003e\n\n### 9. Store metadata and classification info\n\nOur app is shaping up quite nicely!\nAs it stands, it's an application that does inference on images.\nHowever, it doesn't save them.\n\nLet's expand our application so it has a **database**\nwhere image classification is saved/persisted!\n\nWe'll use **`Postgres`** for this.\nTypically, when you create a new `Phoenix` project\nwith `mix phx.new`,\n[a `Postgres` database will be automatically created](https://hexdocs.pm/phoenix/installation.html#postgresql).\nBecause we didn't do this,\nwe'll have to configure this ourselves.\n\nLet's do it!\n\n#### 9.1 Installing dependencies\n\nWe'll install all the needed dependencies first.\nIn `mix.exs`, add the following snippet\nto the `deps` section.\n\n```elixir\n # HTTP Request\n {:httpoison, \"~\u003e 2.2\"},\n {:mime, \"~\u003e 2.0.5\"},\n {:ex_image_info, \"~\u003e 0.2.4\"},\n\n # DB\n {:phoenix_ecto, \"~\u003e 4.4\"},\n {:ecto_sql, \"~\u003e 3.10\"},\n {:postgrex, \"\u003e= 0.0.0\"},\n```\n\n- [**`httpoison`**](https://github.com/edgurgel/httpoison),\n [**`mime`**](https://hex.pm/packages/mime) and\n [**`ex_image_info`**](https://hex.pm/packages/ex_image_info)\n are used to make HTTP requests,\n get the content type and information from an image file,\n respectively.\n These will be needed to upload a given image to\n [`imgup`](https://github.com/dwyl/imgup),\n by making multipart requests.\n\n- [**`phoenix_ecto`**](https://github.com/phoenixframework/phoenix_ecto),\n [**`ecto_sql`**](https://github.com/elixir-ecto/ecto_sql) and\n [**`postgrex`**](https://github.com/elixir-ecto/postgrex)\n are needed to properly configure our driver in Elixir\n that will connect to a Postgres database,\n in which we will persist data.\n\nRun `mix deps.get` to install these dependencies.\n\n#### 9.2 Adding `Postgres` configuration files\n\nNow let's create the needed files\nto properly connect to a Postgres relational database.\nStart by going to `lib/app`\nand create `repo.ex`.\n\n```elixir\ndefmodule App.Repo do\n use Ecto.Repo,\n otp_app: :app,\n adapter: Ecto.Adapters.Postgres\nend\n```\n\nThis module will be needed in our configuration files\nso our app knows where the database is.\n\nNext, in `lib/app/application.ex`,\nadd the following line to the `children` array\nin the supervision tree.\n\n```elixir\n children = [\n AppWeb.Telemetry,\n App.Repo, # add this line\n {Phoenix.PubSub, name: App.PubSub},\n ...\n ]\n```\n\nAwesome! πŸŽ‰\n\nNow let's head over to the files inside the `config` folder.\n\nIn `config/config.exs`,\nadd these lines.\n\n```elixir\nconfig :app,\n ecto_repos: [App.Repo],\n generators: [timestamp_type: :utc_datetime]\n```\n\nWe are referring the module we've previously created (`App.Repo`)\nto `ecto_repos` so `ecto` knows where the configuration\nfor the database is located.\n\nIn `config/dev.exs`,\nadd:\n\n```elixir\nconfig :app, App.Repo,\n username: \"postgres\",\n password: \"postgres\",\n hostname: \"localhost\",\n database: \"app_dev\",\n stacktrace: true,\n show_sensitive_data_on_connection_error: true,\n pool_size: 10\n```\n\nWe are defining the parameters of the database\nthat is used during development.\n\nIn `config/runtime.exs`,\nadd:\n\n```elixir\nif config_env() == :prod do\n database_url =\n System.get_env(\"DATABASE_URL\") ||\n raise \"\"\"\n environment variable DATABASE_URL is missing.\n For example: ecto://USER:PASS@HOST/DATABASE\n \"\"\"\n\n maybe_ipv6 = if System.get_env(\"ECTO_IPV6\") in ~w(true 1), do: [:inet6], else: []\n\n config :app, App.Repo,\n # ssl: true,\n url: database_url,\n pool_size: String.to_integer(System.get_env(\"POOL_SIZE\") || \"10\"),\n socket_options: maybe_ipv6\n\n # ...\n```\n\nWe are configuring the runtime database configuration..\n\nIn `config/test.exs`,\nadd:\n\n```elixir\nconfig :app, App.Repo,\n username: \"postgres\",\n password: \"postgres\",\n hostname: \"localhost\",\n database: \"app_test#{System.get_env(\"MIX_TEST_PARTITION\")}\",\n pool: Ecto.Adapters.SQL.Sandbox,\n pool_size: 10\n```\n\nHere we're defining the database used during testing.\n\nNow let's create a **migration file** to create our database table.\nIn `priv/repo/migrations/`, create a file\ncalled `20231204092441_create_images.exs`\n(or any other timestamp string)\nwith the following piece of code.\n\n```elixir\ndefmodule App.Repo.Migrations.CreateImages do\n use Ecto.Migration\n\n def change do\n create table(:images) do\n add :url, :string\n add :description, :string\n add :width, :integer\n add :height, :integer\n\n timestamps(type: :utc_datetime)\n end\n end\nend\n```\n\nAnd that's it!\nThose are the needed files that our application needs\nto properly connect and persist data into the Postgres database.\n\nYou can now run `mix ecto.create` and `mix ecto.migrate`\nto create the database\nand the `\"images\"` table.\n\n#### 9.3 Creating `Image` schema\n\nFor now, let's create a simple table `\"images\"`\nin our database\nthat has the following properties:\n\n- **`description`**: the description of the image\n from the model.\n- **`width`**: width of the image.\n- **`height`**: height of the image.\n- **`url`**: public URL where the image is publicly hosted.\n\nWith this in mind, let's create a new file!\nIn `lib/app/`, create a file called `image.ex`.\n\n```elixir\ndefmodule App.Image do\n use Ecto.Schema\n alias App.{Image, Repo}\n\n @primary_key {:id, :id, autogenerate: true}\n schema \"images\" do\n field(:description, :string)\n field(:width, :integer)\n field(:url, :string)\n field(:height, :integer)\n\n timestamps(type: :utc_datetime)\n end\n\n def changeset(image, params \\\\ %{}) do\n image\n |\u003e Ecto.Changeset.cast(params, [:url, :description, :width, :height])\n |\u003e Ecto.Changeset.validate_required([:url, :description, :width, :height])\n end\n\n @doc \"\"\"\n Uploads the given image to S3\n and adds the image information to the database.\n \"\"\"\n def insert(image) do\n %Image{}\n |\u003e changeset(image)\n |\u003e Repo.insert!()\n end\nend\n```\n\nWe've just created the `App.Image` schema\nwith the aforementioned fields.\n\nWe've created `changeset/1`, which is used to cast\nand validate the properties of a given object\nbefore interacting with the database.\n\n`insert/1` receives an object,\nruns it through the changeset\nand inserts it in the database.\n\n#### 9.4 Changing our LiveView to persist data\n\nNow that we have our database set up,\nlet's change some of our code so we persist data into it!\nIn this section, we'll be working in the `lib/app_web/live/page_live.ex` file.\n\nFirst, let's import `App.Image`\nand create an `ImageInfo` struct to hold the information\nof the image throughout the process of uploading\nand classifying the image.\n\n```elixir\ndefmodule AppWeb.PageLive do\n use AppWeb, :live_view\n alias App.Image # add this import\n alias Vix.Vips.Image, as: Vimage\n\n\n defmodule ImageInfo do\n @doc \"\"\"\n General information for the image that is being analysed.\n This information is useful when persisting the image to the database.\n \"\"\"\n defstruct [:mimetype, :width, :height, :url, :file_binary]\n end\n\n # ...\n```\n\nWe are going to be using `ImageInfo`\nin our socket assigns.\nLet's add to it when the LiveView is mounting!\n\n```elixir\n |\u003e assign(\n label: nil,\n upload_running?: false,\n task_ref: nil,\n image_info: nil, # add this line\n image_preview_base64: nil,\n example_list_tasks: [],\n example_list: [],\n display_list?: false\n )\n```\n\nWhen the person uploads an image,\nwe want to retrieve its info (namely its _height_, _width_)\nand upload the image to an `S3` bucket (we're doing this through `imgup`)\nso we can populate the `:url` field of the schema in the database.\n\nWe can retrieve this information _while consuming the entry_\n/uploading the image file.\nFor this, go to `handle_progress(:image_list, entry, socket)`\nand change the function to the following.\n\n```elixir\n def handle_progress(:image_list, entry, socket) when entry.done? do\n # We've changed the object that is returned from `consume_uploaded_entry/3` to return an `image_info` object.\n %{tensor: tensor, image_info: image_info} =\n consume_uploaded_entry(socket, entry, fn %{} = meta -\u003e\n file_binary = File.read!(meta.path)\n\n # Add this line. It uses `ExImageInfo` to retrieve the info from the file binary.\n {mimetype, width, height, _variant} = ExImageInfo.info(file_binary)\n\n {:ok, thumbnail_vimage} =\n Vix.Vips.Operation.thumbnail(meta.path, @image_width, size: :VIPS_SIZE_DOWN)\n\n {:ok, tensor} = pre_process_image(thumbnail_vimage)\n\n # Add this line. Uploads the image to the S3, which returns the `url` and `compressed url`.\n # (we'll implement this function next)\n url = Image.upload_image_to_s3(meta.path, mimetype) |\u003e Map.get(\"url\")\n\n # Add this line. We are constructing the image_info object to be returned.\n image_info = %ImageInfo{mimetype: mimetype, width: width, height: height, file_binary: file_binary, url: url}\n\n # Return it\n {:ok, %{tensor: tensor, image_info: image_info}}\n end)\n\n task =\n Task.Supervisor.async(App.TaskSupervisor, fn -\u003e\n Nx.Serving.batched_run(ImageClassifier, tensor)\n end)\n\n base64 = \"data:image/png;base64, \" \u003c\u003e Base.encode64(image_info.file_binary)\n\n # Change this line so `image_info` is defined when the image is uploaded\n {:noreply, assign(socket, upload_running?: true, task_ref: task.ref, image_preview_base64: base64, image_info: image_info)}\n #else\n # {:noreply, socket}\n #end\n end\n```\n\nCheck the comment lines for more explanation on the changes that have bee nmade.\nWe are using `ExImageInfo` to fetch the information from the image\nand assigning it to the `image_info` socket we defined earlier.\n\nWe are also using `Image.upload_image_to_s3/2` to upload our image to `imgup`.\nLet's define this function in `lib/app/image.ex`.\n\n```elixir\n def upload_image_to_s3(file_path, mimetype) do\n extension = MIME.extensions(mimetype) |\u003e Enum.at(0)\n\n # Upload to Imgup - https://github.com/dwyl/imgup\n upload_response =\n HTTPoison.post!(\n \"https://imgup.fly.dev/api/images\",\n {:multipart,\n [\n {\n :file,\n file_path,\n {\"form-data\", [name: \"image\", filename: \"#{Path.basename(file_path)}.#{extension}\"]},\n [{\"Content-Type\", mimetype}]\n }\n ]},\n []\n )\n\n # Return URL\n Jason.decode!(upload_response.body)\n end\n```\n\nWe're using `HTTPoison` to make a multipart request to the `imgup` server,\neffectively uploading the image to the server.\nIf the upload is successful, it returns the `url` of the uploaded image.\n\nLet's go back to `lib/app_web/live/page_live.ex`.\nNow that we have `image_info` in the socket assigns,\nwe can use it to **insert a row in the `\"images\"` table in the database**.\nWe only want to do this after the model is done running,\nso simply change `handle_info/2` function\n(which is called after the model is done with classifying the image).\n\n```elixir\n cond do\n\n # If the upload task has finished executing, we update the socket assigns.\n Map.get(assigns, :task_ref) == ref -\u003e\n\n # Insert image to database\n image = %{\n url: assigns.image_info.url,\n width: assigns.image_info.width,\n height: assigns.image_info.height,\n description: label\n }\n Image.insert(image)\n\n # Update socket assigns\n {:noreply, assign(socket, label: label, upload_running?: false)}\n\n\n # ...\n```\n\nIn the `cond do` statement,\nwe want to change the one pertaining to the image that is uploaded,\n_not the example list_ that is defined below.\nWe simply create an `image` variable with information\nthat is passed down to `Image.insert/1`,\neffectively adding the row to the database.\n\nAnd that's it!\n\nNow every time a person uploads an image\nand the model is executed,\nwe are saving its location (`:url`),\ninformation (`:width` and `:height`)\nand the result of the classifying model\n(`:description`).\n\nπŸ₯³\n\n\u003e [!NOTE]\n\u003e\n\u003e If you're curious and want to see the data in your database,\n\u003e we recommend using [`DBeaver`](https://dbeaver.io/),\n\u003e an open-source database manager.\n\u003e\n\u003e You can learn more about it at https://github.com/dwyl/learn-postgresql.\n\n### 10. Adding double MIME type check and showing feedback to the person in case of failure\n\nCurrently, we are not handling any errors\nin case the upload of the image to `imgup` fails.\nAlthough this is not critical,\nit'd be better if we could show feedback to the person\nin case the upload to `imgup` fails.\nThis is good for us as well,\nbecause we _can monitor and locate the error faster_\nif we log the errors.\n\nFor this, let's head over to `lib/app/image.ex`\nand update the `upload_image_to_s3/2` function we've implemented.\n\n```elixir\n def upload_image_to_s3(file_path, mimetype) do\n extension = MIME.extensions(mimetype) |\u003e Enum.at(0)\n\n # Upload to Imgup - https://github.com/dwyl/imgup\n upload_response =\n HTTPoison.post(\n \"https://imgup.fly.dev/api/images\",\n {:multipart,\n [\n {\n :file,\n file_path,\n {\"form-data\", [name: \"image\", filename: \"#{Path.basename(file_path)}.#{extension}\"]},\n [{\"Content-Type\", mimetype}]\n }\n ]},\n []\n )\n\n # Process the response and return error if there was a problem uploading the image\n case upload_response do\n # In case it's successful\n {:ok, %HTTPoison.Response{status_code: 200, body: body}} -\u003e\n %{\"url\" =\u003e url, \"compressed_url\" =\u003e _} = Jason.decode!(body)\n {:ok, url}\n\n # In case it returns HTTP 400 with specific reason it failed\n {:ok, %HTTPoison.Response{status_code: 400, body: body}} -","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdwyl%2Fimage-classifier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdwyl%2Fimage-classifier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdwyl%2Fimage-classifier/lists"}