{"id":46714075,"url":"https://github.com/dwyl/image-classifier","last_synced_at":"2026-03-09T10:02:46.492Z","repository":{"id":204789760,"uuid":"709711262","full_name":"dwyl/image-classifier","owner":"dwyl","description":"πŸ–ΌοΈ Classify images and extract data from or describe their contents using machine learning","archived":false,"fork":false,"pushed_at":"2026-01-02T20:12:14.000Z","size":24486,"stargazers_count":31,"open_issues_count":5,"forks_count":5,"subscribers_count":7,"default_branch":"main","last_synced_at":"2026-01-06T21:34:12.228Z","etag":null,"topics":["ai","artificial-intelligence","classification","hugging-face","huggingface","image","image-classification","image-recognition","learn","machine-learning","machinelearning","tutorial"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dwyl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-10-25T08:42:36.000Z","updated_at":"2026-01-04T06:13:09.000Z","dependencies_parsed_at":"2025-09-01T17:35:55.463Z","dependency_job_id":null,"html_url":"https://github.com/dwyl/image-classifier","commit_stats":null,"previous_names":["dwyl/image-classifier"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dwyl/image-classifier","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dwyl","download_url":"https://codeload.github.com/dwyl/image-classifier/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2Fimage-classifier/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30290933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T02:57:19.223Z","status":"ssl_error","status_checked_at":"2026-03-09T02:56:26.373Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","classification","hugging-face","huggingface","image","image-classification","image-recognition","learn","machine-learning","machinelearning","tutorial"],"created_at":"2026-03-09T10:02:42.037Z","updated_at":"2026-03-09T10:02:46.478Z","avatar_url":"https://github.com/dwyl.png","language":"Jupyter Notebook","readme":"\u003cdiv align=\"center\"\u003e\n\n# Image Captioning \u0026 Semantic Search in `Elixir`\n\n![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/dwyl/image-classifier/ci.yml?label=build\u0026style=flat-square\u0026branch=main)\n[![codecov.io](https://img.shields.io/codecov/c/github/dwyl/image-classifier/main.svg?style=flat-square)](https://codecov.io/github/dwyl/image-classifier?branch=main)\n[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat-square)](https://github.com/dwyl/image-classifier/issues)\n[![HitCount](https://hits.dwyl.com/dwyl/image-classifier.svg?style=flat-square\u0026show=unique)](https://hits.dwyl.com/dwyl/image-classifier)\n\nLet's use `Elixir` machine learning capabilities\nto build an application\nthat performs **image captioning**\nand **semantic search**\nto look for uploaded images\nwith your voice! πŸŽ™οΈ\n\n\u003c/div\u003e\n\n\u003cbr /\u003e\n\n- [Image Captioning \\\u0026 Semantic Search in `Elixir`](#image-captioning--semantic-search-in-elixir)\n - [Why? 🀷](#why-)\n - [What? πŸ’­](#what-)\n - [Who? πŸ‘€](#who-)\n - [How? πŸ’»](#how-)\n - [Prerequisites](#prerequisites)\n - [πŸŒ„ Image Captioning in `Elixir`](#-image-captioning-in-elixir)\n - [0. Creating a fresh `Phoenix` project](#0-creating-a-fresh-phoenix-project)\n - [1. Installing initial dependencies](#1-installing-initial-dependencies)\n - [2. Adding `LiveView` capabilities to our project](#2-adding-liveview-capabilities-to-our-project)\n - [3. Receiving image files](#3-receiving-image-files)\n - [4. Integrating `Bumblebee`](#4-integrating-bumblebee)\n - [4.1 `Nx` configuration](#41-nx-configuration)\n - [4.2 `Async` processing the image for classification](#42-async-processing-the-image-for-classification)\n - [4.2.1 Considerations regarding `async` processes](#421-considerations-regarding-async-processes)\n - [4.2.2 Alternative for better testing](#422-alternative-for-better-testing)\n - [4.3 Image pre-processing](#43-image-pre-processing)\n - [4.4 Updating the view](#44-updating-the-view)\n - [4.5 Check it out!](#45-check-it-out)\n - [4.6 Considerations on user images](#46-considerations-on-user-images)\n - [5. Final Touches](#5-final-touches)\n - [5.1 Setting max file size](#51-setting-max-file-size)\n - [5.2 Show errors](#52-show-errors)\n - [5.3 Show image preview](#53-show-image-preview)\n - [6. What about other models?](#6-what-about-other-models)\n - [7. How do I deploy this thing?](#7-how-do-i-deploy-this-thing)\n - [8. Showing example images](#8-showing-example-images)\n - [8.1 Creating a hook in client](#81-creating-a-hook-in-client)\n - [8.2 Handling the example images list event inside our LiveView](#82-handling-the-example-images-list-event-inside-our-liveview)\n - [8.3 Updating the view](#83-updating-the-view)\n - [8.4 Using URL of image instead of base64-encoded](#84-using-url-of-image-instead-of-base64-encoded)\n - [8.5 See it running](#85-see-it-running)\n - [9. Store metadata and classification info](#9-store-metadata-and-classification-info)\n - [9.1 Installing dependencies](#91-installing-dependencies)\n - [9.2 Adding `Postgres` configuration files](#92-adding-postgres-configuration-files)\n - [9.3 Creating `Image` schema](#93-creating-image-schema)\n - [9.4 Changing our LiveView to persist data](#94-changing-our-liveview-to-persist-data)\n - [10. Adding double MIME type check and showing feedback to the person in case of failure](#10-adding-double-mime-type-check-and-showing-feedback-to-the-person-in-case-of-failure)\n - [10.1 Showing a toast component with error](#101-showing-a-toast-component-with-error)\n - [11. Benchmarking image captioning models](#11-benchmarking-image-captioning-models)\n - [πŸ” Semantic search](#-semantic-search)\n - [0. Overview of the process](#0-overview-of-the-process)\n - [0.1 Audio transcription](#01-audio-transcription)\n - [0.2 Creating embeddings](#02-creating-embeddings)\n - [0.3 Semantical search](#03-semantical-search)\n - [1. Pre-requisites](#1-pre-requisites)\n - [2. Transcribe an audio recording](#2-transcribe-an-audio-recording)\n - [1.1 Adding a loading spinner](#11-adding-a-loading-spinner)\n - [2.2 Defining `Javascript` hook](#22-defining-javascript-hook)\n - [2.3 Handling audio upload in `LiveView`](#23-handling-audio-upload-in-liveview)\n - [2.4 Serving the `Whisper` model](#24-serving-the-whisper-model)\n - [2.5 Handling the model's response and updating elements in the view](#25-handling-the-models-response-and-updating-elements-in-the-view)\n - [3. Embeddings and semantic search](#3-embeddings-and-semantic-search)\n - [3.1 The `HNSWLib` Index (GenServer)](#31-the-hnswlib-index-genserver)\n - [3.2 Saving the `HNSWLib` Index in the database](#32-saving-the-hnswlib-index-in-the-database)\n - [3.2 The embeding model](#32-the-embeding-model)\n - [4. Using the Index and embeddings](#4-using-the-index-and-embeddings)\n - [4.0 Check the folder \"hnswlib\"](#40-check-the-folder-hnswlib)\n - [4.1 Computing the embeddings in our app](#41-computing-the-embeddings-in-our-app)\n - [4.1.1 Changing the `Image` schema so it's embeddable](#411-changing-the-image-schema-so-its-embeddable)\n - [4.1.2 Using embeddings in semantic search](#412-using-embeddings-in-semantic-search)\n - [4.1.2.1 Mount socket assigns](#4121-mount-socket-assigns)\n - [4.1.2.2 Consuming image uploads](#4122-consuming-image-uploads)\n - [4.1.2.3 Using the embeddings to semantically search images](#4123-using-the-embeddings-to-semantically-search-images)\n - [4.1.2.4 Creating embeddings when uploading images](#4124-creating-embeddings-when-uploading-images)\n - [4.1.2.5 Update the LiveView view](#4125-update-the-liveview-view)\n - [5. Tweaking our UI](#5-tweaking-our-ui)\n - [_Please_ star the repo! ⭐️](#please-star-the-repo-️)\n\n\u003cbr /\u003e\n\n## Why? 🀷\n\nWhilst building our [app](https://github.com/dwyl/app),\nwe consider `images` an _essential_ medium of communication.\n\nWe needed a fully-offline capable (no 3rd party APIs/Services) image captioning service\nusing state-of-the-art pre-trained image and embedding models to describe images uploaded in our\n[`App`](https://github.com/dwyl/app).\n\nBy adding a way of captioning images, we make it _easy_ for people to suggest meta tags that describe images so they become **searchable**.\n\n## What? πŸ’­\n\nA step-by-step tutorial building a fully functional\n`Phoenix LiveView` web application that allows anyone\nto upload an image and have it described\nand searchable.\n\nIn addition to this,\nthe app will allow the person to record an audio\nwhich describes the image they want to find.\n\nThe audio will be transcribed into text\nand be semantically queryable.\nWe do this by encoding the image captions\nas vectors and running `knn search` on them.\n\nWe'll be using three different models:\n\n- Salesforce's BLIP model [`blip-image-captioning-large`](https://huggingface.co/Salesforce/blip-image-captioning-large)\n for image captioning.\n- OpenAI's speech recognition model\n [`whisper-small`](https://huggingface.co/openai/whisper-small).\n- [`sentence-transformers/paraphrase-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2)\n embedding model.\n\n## Who? πŸ‘€\n\nThis tutorial is aimed at `Phoenix` beginners\nwho want to start exploring the machine-learning capabilities\nof the Elixir language within a `Phoenix` application.\nWe propose to use pre-trained models from Hugging Face via `Bumblebee`\nand grasp how to:\n\n- run a model, in particular image captioning.\n- how to use embeddings.\n- how to run a semantic search using an\n [Approximate Nearest Neighbour](https://towardsdatascience.com/comprehensive-guide-to-approximate-nearest-neighbors-algorithms-8b94f057d6b6)\n algorithm.\n\nIf you are completely new to `Phoenix` and `LiveView`,\nwe recommend you follow the **`LiveView` _Counter_ Tutorial**:\n\n[dwyl/phoenix-liveview-counter-tutorial](https://github.com/dwyl/phoenix-liveview-counter-tutorial)\n\n## How? πŸ’»\n\nIn these chapters, we'll go over the development process of this small application.\nYou'll learn how to do this _yourself_, so grab some coffee and let's get cracking!\n\nThis section will be divided into two sections.\nOne will go over **image captioning**\nwhile the second one will expand the application\nby adding **semantic search**.\n\n## Prerequisites\n\nThis tutorial requires you to have `Elixir` and `Phoenix` installed.\n\nIf you don't, please see [how to install Elixir](https://github.com/dwyl/learn-elixir#installation) and [Phoenix](https://hexdocs.pm/phoenix/installation.html#phoenix).\n\nThis guide assumes you know the basics of `Phoenix`\nand have _some_ knowledge of how it works.\nIf you don't, we _highly suggest_ you follow our other tutorials first, e.g: [github.com/dwyl/**phoenix-chat-example**](https://github.com/dwyl/phoenix-chat-example)\n\nIn addition to this, **_some_ knowledge of `AWS`** - what it is, what an `S3` bucket is/does - **is assumed**.\n\n\u003e [!NOTE]\n\u003e If you have questions or get stuck,\n\u003e please open an issue!\n\u003e [/dwyl/image-classifier/issues](https://github.com/dwyl/image-classifier/issues)\n\n\u003cdiv align=\"center\"\u003e\n\n## πŸŒ„ Image Captioning in `Elixir`\n\n\u003e In this section, we'll start building our application\n\u003e with `Bumblebee` that supports Transformer models.\n\u003e At the end of this section,\n\u003e you'll have a fully functional application\n\u003e that receives an image,\n\u003e processes it accordingly\n\u003e and captions it.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/image-classifier/assets/17494745/05d0b510-ef9a-4a51-8425-d27902b0f7ad\"\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n### 0. Creating a fresh `Phoenix` project\n\nLet's create a fresh `Phoenix` project.\nRun the following command in a given folder:\n\n```sh\nmix phx.new . --app app --no-dashboard --no-ecto --no-gettext --no-mailer\n```\n\nWe're running [`mix phx.new`](https://hexdocs.pm/phoenix/Mix.Tasks.Phx.New.html) to generate a new project without a dashboard and mailer (email) service,\nsince we don't need those features in our project.\n\nAfter this, if you run `mix phx.server` to run your server, you should be able to see the following page.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/fields/assets/194400/891e890e-c94a-402e-baee-ee47fd3725a7\"\u003e\n\u003c/p\u003e\n\nWe're ready to start building.\n\n### 1. Installing initial dependencies\n\nNow that we're ready to go, let's start by adding some dependencies.\n\nHead over to `mix.exs`and add the following dependencies\nto the `deps` section.\n\n```elixir\n{:bumblebee, \"~\u003e 0.5.0\"},\n{:exla, \"~\u003e 0.7.0\"},\n{:nx, \"~\u003e 0.7.0 \"},\n{:hnswlib, \"~\u003e 0.1.5\"},\n```\n\n- [**`bumblebee`**](https://github.com/elixir-nx/bumblebee) is a framework that will allows us to integrate\n [`Transformer Models`](https://huggingface.co/docs/transformers/index) in `Phoenix`.\n The `Transformers` (from [Hugging Face](https://huggingface.co/))\n are APIs that allow us to easily download and use\n [pre-trained models](https://blogs.nvidia.com/blog/2022/12/08/what-is-a-pretrained-ai-model).\n The `Bumblebee` package aims to support all Transformer Models, even if some are still lacking.\n You may check which ones are supported by visiting\n `Bumblebee`'s repository or by visiting https://jonatanklosko-bumblebee-tools.hf.space/apps/repository-inspector\n and checking if the model is currently supported.\n\n- [**`Nx`**](https://hexdocs.pm/nx/Nx.html) is a library that allows us to work with\n [`Numerical Elixir`](https://github.com/elixir-nx/), the Elixir's way of doing [numerical computing](https://www.hilarispublisher.com/open-access/introduction-to-numerical-computing-2168-9679-1000423.pdf). It supports tensors and numericla computations.\n\n- [**`EXLA`**](https://hexdocs.pm/exla/EXLA.html) is the Elixir implementation of [Google's XLA](https://www.tensorflow.org/xla/),\n a compiler that provides faster linear algebra calculations\n with `TensorFlow` models.\n This backend compiler is needed for `Nx`.\n We are installing `EXLA` because it allows us to compile models _just-in-time_ and run them on CPU and/or GPU.\n\n- [**`Vix`**](https://hexdocs.pm/vix/readme.html) is an Elixir extension for [libvips](https://www.libvips.org/), an image processing library.\n\nIn `config/config.exs`, let's add our `:nx` configuration\nto use `EXLA`.\n\n```elixir\nconfig :nx, default_backend: EXLA.Backend\n```\n\n### 2. Adding `LiveView` capabilities to our project\n\nAs it stands, our project is not using `LiveView`.\nLet's fix this.\n\nThis will launch a super-powered process that establishes a WebSocket connection\nbetween the server and the browser.\n\nIn `lib/app_web/router.ex`, change the `scope \"/\"` to the following.\n\n```elixir\n scope \"/\", AppWeb do\n pipe_through :browser\n\n live \"/\", PageLive\n end\n```\n\nInstead of using the `PageController`,\nwe are going to be creating `PageLive`,\na `LiveView` file.\n\nLet's create our `LiveView` files.\nInside `lib/app_web`,\ncreate a folder called `live`\nand create the following file\n`page_live.ex`.\n\n```elixir\n#/lib/app_web/live/page_live.ex\ndefmodule AppWeb.PageLive do\n use AppWeb, :live_view\n\n @impl true\n def mount(_params, _session, socket) do\n {:ok, socket}\n end\nend\n\n```\n\nThis is a simple `LiveView` controller.\n\nIn the same `live` folder,\ncreate a file called `page_live.html.heex`\nand use the following code.\n\n```html\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-28 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv\n class=\"flex justify-center items-center mx-auto max-w-xl w-[50vw] lg:mx-0\"\n \u003e\n \u003cform\u003e\n \u003cdiv class=\"space-y-12\"\u003e\n \u003cdiv\u003e\n \u003ch2 class=\"text-base font-semibold leading-7 text-gray-900\"\u003e\n Image Classifier\n \u003c/h2\u003e\n \u003cp class=\"mt-1 text-sm leading-6 text-gray-600\"\u003e\n Drag your images and we'll run an AI model to caption it!\n \u003c/p\u003e\n\n \u003cdiv class=\"mt-10 grid grid-cols-1 gap-x-6 gap-y-8 sm:grid-cols-6\"\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cspan\u003eUpload a file\u003c/span\u003e\n \u003cinput\n id=\"file-upload\"\n name=\"file-upload\"\n type=\"file\"\n class=\"sr-only\"\n /\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/form\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nThis is a simple HTML form that uses\n[`Tailwind CSS`](https://github.com/dwyl/learn-tailwind)\nto enhance the presentation of the upload form.\nWe'll also remove the unused header of the page layout,\nwhile we're at it.\n\nLocate the file `lib/app_web/components/layouts/app.html.heex`\nand remove the `\u003cheader\u003e` class.\nThe file should only have the following code:\n\n```html\n\u003cmain class=\"px-4 py-20 sm:px-6 lg:px-8\"\u003e\n \u003cdiv class=\"mx-auto max-w-2xl\"\u003e\n \u003c.flash_group flash={@flash} /\u003e \u003c%= @inner_content %\u003e\n \u003c/div\u003e\n\u003c/main\u003e\n```\n\nNow you can safely delete the `lib/app_web/controllers` folder,\nwhich is no longer used.\n\nIf you run `mix phx.server`,\nyou should see the following screen:\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/imgup/assets/17494745/5a3438fe-fa45-47f9-8cb2-9d6d405f55a0\"\u003e\n\u003c/p\u003e\n\nThis means we've successfully added `LiveView`\nand changed our view!\n\n### 3. Receiving image files\n\nNow, let's start by receiving some image files.\n\nWith `LiveView`,\nwe can easily do this by using\n[`allow_upload/3`](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#allow_upload/3)\nwhen mounting our `LiveView`.\nWith this function, we can easily accept\nfile uploads with progress.\nWe can define file types, max number of entries,\nmax file size,\nvalidate the uploaded file and much more!\n\nFirstly,\nlet's make some changes to\n`lib/app_web/live/page_live.html.heex`.\n\n```html\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-28 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv\n class=\"flex justify-center items-center mx-auto max-w-xl w-[50vw] lg:mx-0\"\n \u003e\n \u003cdiv class=\"space-y-12\"\u003e\n \u003cdiv class=\"border-gray-900/10 pb-12\"\u003e\n \u003ch2 class=\"text-base font-semibold leading-7 text-gray-900\"\u003e\n Image Classification\n \u003c/h2\u003e\n \u003cp class=\"mt-1 text-sm leading-6 text-gray-600\"\u003e\n Do simple captioning with this\n \u003ca\n href=\"https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html\"\n class=\"font-mono font-medium text-sky-500\"\n \u003eLiveView\u003c/a\n \u003e\n demo, powered by\n \u003ca\n href=\"https://github.com/elixir-nx/bumblebee\"\n class=\"font-mono font-medium text-sky-500\"\n \u003eBumblebee\u003c/a\n \u003e.\n \u003c/p\u003e\n\n \u003c!-- File upload section --\u003e\n \u003cdiv class=\"mt-10 grid grid-cols-1 gap-x-6 gap-y-8 sm:grid-cols-6\"\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n phx-drop-target=\"{@uploads.image_list.ref}\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform phx-change=\"validate\" phx-submit=\"save\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nWe've added a few features:\n\n- we used\n [`\u003c.live_file_input/\u003e`](https://hexdocs.pm/phoenix_live_view/Phoenix.Component.html#live_file_input/1)\n for `LiveView` file upload.\n We've wrapped this component\n with an element that is annotated with the `phx-drop-target` attribute\n pointing to the DOM `id` of the file input.\n- because we used `\u003c.live_file_input/\u003e`,\n we need to annotate its wrapping element\n with `phx-submit` and `phx-change`,\n as per\n [hexdocs.pm/phoenix_live_view/uploads.html#render-reactive-elements](https://hexdocs.pm/phoenix_live_view/uploads.html#render-reactive-elements)\n\nBecause we've added these bindings,\nwe need to add the event handlers in\n`lib/app_web/live/page_live.ex`.\nOpen it and update it to:\n\n```elixir\ndefmodule AppWeb.PageLive do\n use AppWeb, :live_view\n\n @impl true\n def mount(_params, _session, socket) do\n {:ok,\n socket\n |\u003e assign(label: nil, upload_running?: false, task_ref: nil)\n |\u003e allow_upload(:image_list,\n accept: ~w(image/*),\n auto_upload: true,\n progress: \u0026handle_progress/3,\n max_entries: 1,\n chunk_size: 64_000\n )}\n end\n\n @impl true\n def handle_event(\"validate\", _params, socket) do\n {:noreply, socket}\n end\n\n @impl true\n def handle_event(\"remove-selected\", %{\"ref\" =\u003e ref}, socket) do\n {:noreply, cancel_upload(socket, :image_list, ref)}\n end\n\n @impl true\n def handle_event(\"save\", _params, socket) do\n {:noreply, socket}\n end\n\n defp handle_progress(:image_list, entry, socket) when entry.done? do\n uploaded_file =\n consume_uploaded_entry(socket, entry, fn %{path: _path} = _meta -\u003e\n {:ok, entry}\n end)\n {:noreply, socket}\n end\n\n defp handle_progress(:image_list, _, socket), do: {:noreply, socket}\nend\n```\n\n- when `mount/3`ing the LiveView,\n we are creating three socket assigns:\n `label` pertains to the model prediction;\n `upload_running?` is a boolean referring to whether the model is running or not;\n `task_ref` refers to the reference of the task that was created for image classification\n (we'll delve into this further later down the line).\n Additionally, we are using the `allow_upload/3` function to define our upload configuration.\n The most important settings here are `auto_upload` set to `true`\n and the `progress` fields.\n By configuring these two properties,\n we are telling `LiveView` that _whenever the person uploads a file_,\n **it is processed immediately and consumed**.\n\n- the `progress` field is handled by the `handle_progress/3` function.\n It receives chunks from the client with a build-in `UploadWriter` function\n (as explained in the [docs](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.UploadWriter.html)).\n When the chunks are all consumed, we get the boolean `entry.done? == true`.\n We consume the file in this function by using\n [`consume_uploaded_entry/3`](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#consume_uploaded_entry/3).\n The anonymous function returns `{:ok, data}` or `{:postpone, message}`.\n Whilst consuming the entry/file,\n we can access its path and then use its content.\n _For now_, we don't need to use it.\n But we will in the future to feed our image classifier with it!\n After the callback function is executed,\n this function \"consumes the entry\",\n essentially deleting the image from the temporary folder\n and removing it from the uploaded files list.()\n\n- the `\"validate\"`, `\"remove-selected\"`, `\"save\"` event handlers\n are called whenever the person uploads the image,\n wants to remove it from the list of uploaded images\n and when wants to submit the form,\n respectively.\n You may see that we're not doing much with these handlers;\n we're simply replying with a `:noreply`\n because we don't need to do anything with them.\n\nAnd that's it!\nIf you run `mix phx.server`, nothing will change.\n\n### 4. Integrating `Bumblebee`\n\nNow here comes the fun part!\nIt's time to do some image captioning! πŸŽ‰\n\n#### 4.1 `Nx` configuration\n\nWe first need to add some initial setup in the\n`lib/app/application.ex` file.\nHead over there and change\nthe `start` function like so:\n\n```elixir\n@impl true\ndef start(_type, _args) do\n children = [\n # Start the Telemetry supervisor\n AppWeb.Telemetry,\n # Start the PubSub system\n {Phoenix.PubSub, name: App.PubSub},\n {Nx.Serving, serving: serving(), name: ImageClassifier},\n # Start the Endpoint (http/https)\n AppWeb.Endpoint\n ]\n\n opts = [strategy: :one_for_one, name: App.Supervisor]\n Supervisor.start_link(children, opts)\nend\n\ndef serving do\n {:ok, model_info} = Bumblebee.load_model({:hf, \"microsoft/resnet-50\"})\n {:ok, featurizer} = Bumblebee.load_featurizer({:hf, \"microsoft/resnet-50\"})\n\n Bumblebee.Vision.image_classification(model_info, featurizer,\n top_k: 1,\n compile: [batch_size: 10],\n defn_options: [compiler: EXLA]\n )\nend\n```\n\nWe are using [`Nx.Serving`](https://hexdocs.pm/nx/Nx.Serving.html), which simply allows us to encapsulate tasks; it can be networking, machine learning, data processing or any other task.\n\nIn this specific case, we are using it to **batch requests**.\nThis is extremely useful and important because we are using models that typically run on\n[GPU](https://en.wikipedia.org/wiki/Graphics_processing_unit).\nThe GPU is _really good_ at **parallelizing tasks**.\nTherefore, instead of sending an image classification request one by one, we can _batch them_/bundle them together as much as we can and then send it over.\n\nWe can define the `batch_size` and `batch_timeout` with `Nx.Serving`.\nWe're going to use the default values, hence why we're not explicitly defining them.\n\nWith `Nx.Serving`, we define a `serving/0` function\nthat is then used by it, which in turn is executed in the supervision tree since we declare it as a child in the Application module.\n\nIn the `serving/0` function, we are loading the[`ResNet-50`](https://huggingface.co/microsoft/resnet-50) model and its featurizer.\n\n\u003e [!NOTE]\n\u003e A `featurizer` can be seen as a\n\u003e [`Feature Extractor`](https://huggingface.co/docs/transformers/main_classes/feature_extractor).\n\u003e It is essentially a component that is responsible for converting input data\n\u003e into a format that can be processed by a pre-trained language model.\n\u003e\n\u003e It takes raw information and performs various transformations,\n\u003e such as\n\u003e [tokenization](https://neptune.ai/blog/tokenization-in-nlp),\n\u003e [padding](https://www.baeldung.com/cs/deep-neural-networks-padding),\n\u003e and encoding to prepare the data for model training or inference.\n\nLastly, this function returns a serving for image classification by calling [`image_classification/3`](https://hexdocs.pm/bumblebee/Bumblebee.Vision.html#image_classification/3), where we can define our compiler and task batch size.\nWe gave our serving function the name `ImageClassifier` as declared in the Application module.\n\n#### 4.2 `Async` processing the image for classification\n\nNow we're ready to send the image to the model\nand get a prediction of it!\n\nEvery time we upload an image,\nwe are going to run **async processing**.\nThis means that the task responsible for image classification will be run in another process, thus asynchronously, meaning that the LiveView _won't have to wait_ for this task to finish\nto continue working.\n\nFor this scenario, we are going to be using the\n[`Task` module](https://hexdocs.pm/elixir/1.14/Task.html) to spawn processes to complete this task.\n\nGo to `lib/app_web/live/page_live.ex`\nand change the following code.\n\n```elixir\ndef handle_progress(:image_list, entry, socket) when entry.done? do\n # Consume the entry and get the tensor to feed to classifier\n tensor = consume_uploaded_entry(socket, entry, fn %{} = meta -\u003e\n {:ok, vimage} = Vix.Vips.Image.new_from_file(meta.path)\n pre_process_image(vimage)\n end)\n\n # Create an async task to classify the image\n task = Task.async(fn -\u003e Nx.Serving.batched_run(ImageClassifier, tensor) end)\n\n # Update socket assigns to show spinner whilst task is running\n {:noreply, assign(socket, upload_running?: true, task_ref: task.ref)}\nend\n\n@impl true\ndef handle_info({ref, result}, %{assigns: %{task_ref: ref}} = socket) do\n # This is called everytime an Async Task is created.\n # We flush it here.\n Process.demonitor(ref, [:flush])\n\n # And then destructure the result from the classifier.\n %{predictions: [%{label: label}]} = result\n\n # Update the socket assigns with result and stopping spinner.\n {:noreply, assign(socket, label: label, upload_running?: false)}\nend\n```\n\n\u003e [!NOTE]\n\u003e The `pre_process_image/1` function is yet to be defined.\n\u003e We'll do that in the following section.\n\nIn the `handle_progress/3` function,\nwhilst we are consuming the image,\nwe are first converting it to a\n[`Vix.Vips.Image`](https://hexdocs.pm/vix/Vix.Vips.Image.html) `Struct`\nusing the file path.\nWe then feed this image to the `pre_process_image/1` function that we'll implement later.\n\nWhat's important is to notice this line:\n\n```elixir\ntask = Task.async(fn -\u003e Nx.Serving.batched_run(ImageClassifier, tensor) end)\n```\n\nWe are using\n[`Task.async/1`](https://hexdocs.pm/elixir/1.12/Task.html#async/1)\nto call our `Nx.Serving` build function `ImageClassifier` we've defined earlier,\nthus initiating a batched run with the image tensor.\nWhile the task is spawned,\nwe update the socket assigns with the reference to the task (`:task_ref`)\nand update the `:upload_running?` assign to `true`,\nso we can show a spinner or a loading animation.\n\nWhen the task is spawned using `Task.async/1`,\na couple of things happen in the background.\nThe new process is monitored by the caller (our `LiveView`),\nwhich means that the caller will receive a\n`{:DOWN, ref, :process, object, reason}`\nmessage once the process it is monitoring dies.\nAnd, a link is created between both processes.\n\nTherefore,\nwe **don't need to use**\n[**`Task.await/2`**](https://hexdocs.pm/elixir/1.12/Task.html#await/2).\nInstead, we create a new handler to receive the aforementioned.\nThat's what we're doing in the\n`handle_info({ref, result}, %{assigns: %{task_ref: ref}} = socket)` function.\nThe received message contains a `{ref, result}` tuple,\nwhere `ref` is the monitor’s reference.\nWe use this reference to stop monitoring the task,\nsince we received the result we needed from our task\nand we can discard an exit message.\n\nIn this same function, we destructure the prediction\nfrom the model and assign it to the socket assign `:label`\nand set `:upload_running?` to `false`.\n\nQuite beautiful, isn't it?\nWith this, we don't have to worry if the person closes the browser tab.\nThe process dies (as does our `LiveView`),\nand the work is automatically cancelled,\nmeaning no resources are spent\non a process for which nobody expects a result anymore.\n\n##### 4.2.1 Considerations regarding `async` processes\n\nWhen a task is spawned using `Task.async/2`,\n**it is linked to the caller**.\nWhich means that they're related:\nif one dies, the other does too.\n\nWe ought to take this into account when developing our application.\nIf we don't have control over the result of the task,\nand we don't want our `LiveView` to crash if the task crashes,\nwe must use a different alternative to spawn our task -\n[`Task.Supervisor.async_nolink/3`](https://hexdocs.pm/elixir/1.14/Task.Supervisor.html#async_nolink/3)\ncan be used for this effect,\nmeaning we can use it if we want to make sure\nour `LiveView` won't die and the error is reported,\neven if the task crashes.\n\nWe've chosen `Task.async/2` for this very reason.\nWe are doing something **that takes time/is expensive**\nand we **want to stop the task if `LiveView` is closed/crashes**.\nHowever, if you are building something\nlike a report that has to be generated even if the person closes the browser tab,\nthis is not the right solution.\n\n##### 4.2.2 Alternative for better testing\n\nWe are spawning async tasks by calling `Task.async/1`.\nThis is creating an **_unsupervised_ task**.\nAlthough it's plausible for this simple app,\nit's best for us to create a\n[**`Supervisor`**](https://hexdocs.pm/elixir/1.15.7/Supervisor.html)\nthat manages their child tasks.\nThis gives more control over the execution\nand lifetime of the child tasks.\n\nAdditionally, it's better to have these tasks supervised\nbecause it makes it possible to create tests for our `LiveView`.\nFor this, we need to make a couple of changes.\n\nFirst, head over to `lib/app/application.ex`\nand add a supervisor to the `start/2` function children array.\n\n```elixir\ndef start(_type, _args) do\n children = [\n AppWeb.Telemetry,\n {Phoenix.PubSub, name: App.PubSub},\n {Nx.Serving, serving: serving(), name: ImageClassifier},\n {Task.Supervisor, name: App.TaskSupervisor}, # add this line\n AppWeb.Endpoint\n ]\n\n opts = [strategy: :one_for_one, name: App.Supervisor]\n Supervisor.start_link(children, opts)\nend\n```\n\nWe are creating a [`Task.Supervisor`](https://hexdocs.pm/elixir/Supervisor.html)\nwith the name `App.TaskSupervisor`.\n\nNow, in `lib/app_web/live/page_live.ex`,\nwe create the async task like so:\n\n```elixir\ntask = Task.Supervisor.async(App.TaskSupervisor, fn -\u003e Nx.Serving.batched_run(ImageClassifier, tensor) end)\n```\n\nWe are now using\n[`Task.Supervisor.async`](https://hexdocs.pm/elixir/1.15.7/Task.Supervisor.html#async/3),\npassing the name of the supervisor defined earlier.\n\nAnd that's it!\nWe are creating async tasks like before,\nthe only difference is that they're now **supervised**.\n\nIn tests, you can create a small module that waits for the tasks to be completed.\n\n```elixir\ndefmodule AppWeb.SupervisorSupport do\n\n @moduledoc \"\"\"\n This is a support module helper that is meant to wait for all the children of a supervisor to complete.\n If you go to `lib/app/application.ex`, you'll see that we created a `TaskSupervisor`, where async tasks are spawned.\n This module helps us to wait for all the children to finish during tests.\n \"\"\"\n\n @doc \"\"\"\n Find all children spawned by this supervisor and wait until they finish.\n \"\"\"\n def wait_for_completion() do\n pids = Task.Supervisor.children(App.TaskSupervisor)\n Enum.each(pids, \u0026Process.monitor/1)\n wait_for_pids(pids)\n end\n\n defp wait_for_pids([]), do: nil\n defp wait_for_pids(pids) do\n receive do\n {:DOWN, _ref, :process, pid, _reason} -\u003e wait_for_pids(List.delete(pids, pid))\n end\n end\nend\n```\n\nYou can call `AppWeb.SupervisorSupport.wait_for_completion()`\nin unit tests so they wait for the tasks to complete.\nIn our case,\nwe do that until the _prediction is made_.\n\n#### 4.3 Image pre-processing\n\nAs we've noted before,\nwe need to **pre-process the image before passing it to the model**.\nFor this, we have three main steps:\n\n- removing the [`alpha` ](https://en.wikipedia.org/wiki/Alpha_compositing)\n out of the image, flattening it out.\n- convert the image to `sRGB` [colourspace](https://en.wikipedia.org/wiki/Color_space).\n This is needed to ensure that the image is consistent\n and aligns with the model's training data images.\n- set the representation of the image as a `Tensor`\n to `height, width, bands`.\n The image tensor will then be organized as a three-dimensional array,\n where the first dimension represents the height of the image,\n the second refers to the width of the image,\n and the third pertains to the different\n [spectral bands/channels of the image](https://en.wikipedia.org/wiki/Multispectral_imaging).\n\nOur `pre_process_image/1` function will implement these three steps.\nLet's implement it now! \u003cbr /\u003e\nIn `lib/app_web/live/page_live.ex`,\nadd the following:\n\n```elixir\n defp pre_process_image(%Vimage{} = image) do\n\n # If the image has an alpha channel, flatten it:\n {:ok, flattened_image} = case Vix.Vips.Image.has_alpha?(image) do\n true -\u003e Vix.Vips.Operation.flatten(image)\n false -\u003e {:ok, image}\n end\n\n # Convert the image to sRGB colourspace ----------------\n {:ok, srgb_image} = Vix.Vips.Operation.colourspace(flattened_image, :VIPS_INTERPRETATION_sRGB)\n\n # Converting image to tensor ----------------\n {:ok, tensor} = Vix.Vips.Image.write_to_tensor(srgb_image)\n\n # We reshape the tensor given a specific format.\n # In this case, we are using {height, width, channels/bands}.\n %Vix.Tensor{data: binary, type: type, shape: {x, y, bands}} = tensor\n format = [:height, :width, :bands]\n shape = {x, y, bands}\n\n final_tensor =\n binary\n |\u003e Nx.from_binary(type)\n |\u003e Nx.reshape(shape, names: format)\n\n {:ok, final_tensor}\n end\n```\n\nThe function receives a `Vix` image,\nas detailed earlier.\nWe use [`flatten/1`](https://hexdocs.pm/vix/Vix.Vips.Operation.html#flatten/2)\nto flatten the alpha out of the image.\n\nThe resulting image has its colourspace changed\nby calling [`colourspace/3`](https://hexdocs.pm/vix/Vix.Vips.Operation.html#colourspace/3),\nwhere we change the to `sRGB`.\n\nThe colourspace-altered image is then converted to a\n[tensor](https://hexdocs.pm/vix/Vix.Tensor.html),\nby calling\n[`write_to_tensor/1`](https://hexdocs.pm/vix/Vix.Vips.Image.html#write_to_tensor/1).\n\nWe then\n[reshape](https://hexdocs.pm/nx/Nx.html#reshape/3)\nthe tensor according to the format that was previously mentioned.\n\nThis function returns the processed tensor,\nthat is then used as input to the model.\n\n### 4.4 Updating the view\n\nAll that's left is updating the view\nto reflect these changes we've made to the `LiveView`.\nHead over to `lib/app_web/live/page_live.html.heex`\nand change it to this.\n\n```html\n\u003c.flash_group flash={@flash} /\u003e\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-28 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv\n class=\"flex justify-center items-center mx-auto max-w-xl w-[50vw] lg:mx-0\"\n \u003e\n \u003cdiv class=\"space-y-12\"\u003e\n \u003cdiv class=\"border-gray-900/10 pb-12\"\u003e\n \u003ch2 class=\"text-base font-semibold leading-7 text-gray-900\"\u003e\n Image Classification\n \u003c/h2\u003e\n \u003cp class=\"mt-1 text-sm leading-6 text-gray-600\"\u003e\n Do simple classification with this\n \u003ca\n href=\"https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html\"\n class=\"font-mono font-medium text-sky-500\"\n \u003eLiveView\u003c/a\n \u003e\n demo, powered by\n \u003ca\n href=\"https://github.com/elixir-nx/bumblebee\"\n class=\"font-mono font-medium text-sky-500\"\n \u003eBumblebee\u003c/a\n \u003e.\n \u003c/p\u003e\n\n \u003c!-- File upload section --\u003e\n \u003cdiv class=\"mt-10 grid grid-cols-1 gap-x-6 gap-y-8 sm:grid-cols-6\"\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n phx-drop-target=\"{@uploads.image_list.ref}\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\n \u003c!-- Prediction text --\u003e\n \u003cdiv\n class=\"mt-6 flex space-x-1.5 items-center font-bold text-gray-900 text-xl\"\n \u003e\n \u003cspan\u003eDescription: \u003c/span\u003e\n \u003c!-- Spinner --\u003e\n \u003c%= if @upload_running? do %\u003e\n \u003cdiv role=\"status\"\u003e\n \u003cdiv\n class=\"relative w-6 h-6 animate-spin rounded-full bg-gradient-to-r from-purple-400 via-blue-500 to-red-400 \"\n \u003e\n \u003cdiv\n class=\"absolute top-1/2 left-1/2 transform -translate-x-1/2 -translate-y-1/2 w-3 h-3 bg-gray-200 rounded-full border-2 border-white\"\n \u003e\u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% else %\u003e \u003c%= if @label do %\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @label %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light\"\u003eWaiting for image input.\u003c/span\u003e\n \u003c% end %\u003e \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nIn these changes,\nwe've added the output of the model in the form of text.\nWe are rendering a spinner\nif the `:upload_running?` socket assign is set to true.\nOtherwise,\nwe add the `:label`, which holds the prediction made by the model.\n\nYou may have also noticed that\nwe've changed the `phx` event handlers\nto `noop`.\nThis is simply to simplify the `LiveView`.\n\nHead over to `lib/app_web/live/page_live.ex`.\nYou can now remove the `\"validate\"`, `\"save\"`\nand `\"remove-selected\"` handlers,\nbecause we're not going to be needing them.\nReplace them with this handler:\n\n```elixir\n @impl true\n def handle_event(\"noop\", _params, socket) do\n {:noreply, socket}\n end\n```\n\n#### 4.5 Check it out!\n\nAnd that's it!\nOur app is now _functional_ πŸŽ‰.\n\nIf you run the app,\nyou can drag and drop or select an image.\nAfter this, a task will be spawned that will run the model\nagainst the image that was submitted.\n\nOnce a prediction is made, display it!\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/aws-sdk-mock/assets/17494745/894b988e-4f60-4781-8838-c7fd95e571f0\" /\u003e\n\u003c/p\u003e\n\nYou can and **should** try other models.\n`ResNet-50` is just one of the many that are supported by `Bumblebee`.\nYou can see the supported models at https://github.com/elixir-nx/bumblebee#model-support.\n\n#### 4.6 Considerations on user images\n\nTo keep the app as simple as possible,\nwe are receiving the image from the person as is.\nAlthough we are processing the image,\nwe are doing it so **it is processable by the model**.\n\nWe have to understand that:\n\n- in most cases, **full-resolution images are not necessary**,\n because neural networks work on much smaller inputs\n (e.g. `ResNet-50` works with `224px x 224px` images).\n This means that a lot of data is unnecessarily uploaded over the network,\n increasing workload on the server to potentially downsize a large image.\n- decoding an image requires an additional package,\n meaning more work on the server.\n\nWe can avoid both of these downsides by moving this work to the client.\nWe can leverage the\n[`Canvas API` ](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API)\nto decode and downsize this image on the client-side,\nreducing server workload.\n\nYou can see an example implementation of this technique\nin `Bumblebee`'s repository\nat https://github.com/elixir-nx/bumblebee/blob/main/examples/phoenix/image_classification.exs\n\nHowever, since we are not using `JavaScript` for anything,\nwe can (and _should_!) properly downsize our images\nso they better fit the training dataset of the model we use.\nThis will allow the model to process faster\nsince larger images carry over more data that is ultimately unnecessary\nfor models to make predictions.\n\nOpen\n`lib/app_web/live/page_live.ex`,\nfind the `handle_progress/3` function\nand change resize the image _before processing it_.\n\n```elixir\n file_binary = File.read!(meta.path)\n\n # Get image and resize\n # This is dependant on the resolution of the model's dataset.\n # In our case, we want the width to be closer to 640, whilst maintaining aspect ratio.\n width = 640\n {:ok, thumbnail_vimage} = Vix.Vips.Operation.thumbnail(meta.path, width, size: :VIPS_SIZE_DOWN)\n\n # Pre-process it\n {:ok, tensor} = pre_process_image(thumbnail_vimage)\n\n #...\n```\n\nWe are using\n[`Vix.Vips.Operation.thumbnail/3`](https://hexdocs.pm/vix/Vix.Vips.Operation.html#thumbnail/3)\nto resize our image to a fixed width\nwhilst maintaining aspect ratio.\nThe `width` variable can be dependent on the model that you use.\nFor example, `ResNet-50` is trained on `224px224` pictures,\nso you may want to resize the image to this width.\n\n\u003e **Note**: We are using the `thumbnail/3` function\n\u003e instead of `resize/3` because it's _much_ faster. \u003cbr /\u003e\n\u003e Check\n\u003e https://github.com/libvips/libvips/wiki/HOWTO----Image-shrinking\n\u003e to know why.\n\n### 5. Final Touches\n\nAlthough our app is functional,\nwe can make it **better**. 🎨\n\n#### 5.1 Setting max file size\n\nIn order to better control user input,\nwe should add a limit to the size of the image that is being uploaded.\nIt will be easier on our server and ultimately save costs.\n\nLet's add a cap of `5MB` to our app!\nFortunately for you, this is super simple!\nYou just need to add the `max_file_size`\nto the `allow_uploads/2` function\nwhen mounting the `LiveView`!\n\n```elixir\n def mount(_params, _session, socket) do\n {:ok,\n socket\n |\u003e assign(label: nil, upload_running?: false, task_ref: nil)\n |\u003e allow_upload(:image_list,\n accept: ~w(image/*),\n auto_upload: true,\n progress: \u0026handle_progress/3,\n max_entries: 1,\n chunk_size: 64_000,\n max_file_size: 5_000_000 # add this\n )}\n end\n```\n\nAnd that's it!\nThe number is in `bytes`,\nhence why we set it as `5_000_000`.\n\n#### 5.2 Show errors\n\nIn case a person uploads an image that is too large,\nwe should show this feedback to the person!\n\nFor this, we can leverage the\n[`upload_errors/2`](https://hexdocs.pm/phoenix_live_view/Phoenix.Component.html#upload_errors/2)\nfunction.\nThis function will return the entry errors for an upload.\nWe need to add a handler for one of these errors to show it first.\n\nHead over `lib/app_web/live/page_live.ex`\nand add the following line.\n\n```elixir\n def error_to_string(:too_large), do: \"Image too large. Upload a smaller image up to 5MB.\"\n```\n\nNow, add the following section below the upload form\ninside `lib/app_web/live/page_live.html.heex`.\n\n```html\n\u003c!-- Show errors --\u003e\n\u003c%= for entry \u003c- @uploads.image_list.entries do %\u003e\n\u003cdiv class=\"mt-2\"\u003e\n \u003c%= for err \u003c- upload_errors(@uploads.image_list, entry) do %\u003e\n \u003cdiv class=\"rounded-md bg-red-50 p-4 mb-2\"\u003e\n \u003cdiv class=\"flex\"\u003e\n \u003cdiv class=\"flex-shrink-0\"\u003e\n \u003csvg\n class=\"h-5 w-5 text-red-400\"\n viewBox=\"0 0 20 20\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M10 18a8 8 0 100-16 8 8 0 000 16zM8.28 7.22a.75.75 0 00-1.06 1.06L8.94 10l-1.72 1.72a.75.75 0 101.06 1.06L10 11.06l1.72 1.72a.75.75 0 101.06-1.06L11.06 10l1.72-1.72a.75.75 0 00-1.06-1.06L10 8.94 8.28 7.22z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003c/div\u003e\n \u003cdiv class=\"ml-3\"\u003e\n \u003ch3 class=\"text-sm font-medium text-red-800\"\u003e\n \u003c%= error_to_string(err) %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n\u003c/div\u003e\n\u003c% end %\u003e\n```\n\nWe are iterating over the errors returned by `upload_errors/2`\nand invoking `error_to_string/1`,\nwhich we've just defined in our `LiveView`.\n\nNow, if you run the app\nand try to upload an image that is too large,\nan error will show up.\n\nAwesome! πŸŽ‰\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=800 src=\"https://github.com/dwyl/aws-sdk-mock/assets/17494745/1bf903eb-31d5-48a4-9da9-1f5f64932b6e\" /\u003e\n\u003c/p\u003e\n\n#### 5.3 Show image preview\n\nAs of now, even though our app predicts the given images,\nit is not showing a preview of the image the person submitted.\nLet's fix this πŸ› οΈ.\n\nLet's add a new socket assign variable\npertaining to the [base64](https://en.wikipedia.org/wiki/Base64) representation\nof the image in `lib/app_web/live_page/live.ex`\n\n```elixir\n |\u003e assign(label: nil, upload_running?: false, task_ref: nil, image_preview_base64: nil)\n```\n\nWe've added `image_preview_base64`\nas a new socket assign,\ninitializing it as `nil`.\n\nNext, we need to _read the file while consuming it_,\nand properly update the socket assign\nso we can show it to the person.\n\nIn the same file,\nchange the `handle_progress/3` function to the following.\n\n```elixir\n def handle_progress(:image_list, entry, socket) when entry.done? do\n # Consume the entry and get the tensor to feed to classifier\n %{tensor: tensor, file_binary: file_binary} = consume_uploaded_entry(socket, entry, fn %{} = meta -\u003e\n file_binary = File.read!(meta.path)\n\n {:ok, vimage} = Vix.Vips.Image.new_from_file(meta.path)\n {:ok, tensor} = pre_process_image(vimage)\n {:ok, %{tensor: tensor, file_binary: file_binary}}\n end)\n\n # Create an async task to classify the image\n task = Task.Supervisor.async(App.TaskSupervisor, fn -\u003e Nx.Serving.batched_run(ImageClassifier, tensor) end)\n\n # Encode the image to base64\n base64 = \"data:image/png;base64, \" \u003c\u003e Base.encode64(file_binary)\n\n # Update socket assigns to show spinner whilst task is running\n {:noreply, assign(socket, upload_running?: true, task_ref: task.ref, image_preview_base64: base64)}\n end\n```\n\nWe're using [`File.read!/1`](https://hexdocs.pm/elixir/1.13/File.html#read/1)\nto retrieve the binary representation of the image that was uploaded.\nWe use [`Base.encode64/2`](https://hexdocs.pm/elixir/1.12/Base.html#encode64/2)\nto encode this file binary\nand assign the newly created `image_preview_base64` socket assign\nwith this base64 representation of the image.\n\nNow, all that's left to do\nis to _render the image on our view_.\nIn `lib/app_web/live/page_live.html.heex`,\nlocate the line:\n\n```html\n\u003cdiv class=\"text-center\"\u003e\u003c/div\u003e\n```\n\nWe are going to update this `\u003cdiv\u003e`\nto show the image with the `image_preview_base64` socket assign.\n\n```html\n\u003cdiv class=\"text-center\"\u003e\n \u003c!-- Show image preview --\u003e\n \u003c%= if @image_preview_base64 do %\u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list} class=\"hidden\" /\u003e\n \u003cimg src=\"{@image_preview_base64}\" /\u003e\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c% else %\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list} class=\"hidden\" /\u003e\n Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003ePNG, JPG, GIF up to 5MB\u003c/p\u003e\n \u003c% end %\u003e\n\u003c/div\u003e\n```\n\nAs you can see,\nwe are checking if `@image_preview_base64` is defined.\nIf so, we simply show the image with it as `src` 😊.\n\nNow, if you run the application,\nyou'll see that after dragging the image,\nit is previewed and shown to the person!\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=800 src=\"https://github.com/dwyl/image-classifier/assets/17494745/2835c24f-f4ba-48bc-aab0-6b39830156ce\" /\u003e\n\u003c/p\u003e\n\n### 6. What about other models?\n\nMaybe you weren't happy with the results from this model.\n\nThat's fair. `ResNet-50` is a smaller, \"older\" model compared to other\nimage captioning/classification models.\n\nWhat if you wanted to use others?\nWell, as we've mentioned before,\n`Bumblebee` uses\n[**Transformer models from `HuggingFace`**](https://huggingface.co/docs/transformers/index).\nTo know if one is supported\n(as shown in [`Bumblebee`'s docs](https://github.com/elixir-nx/bumblebee#model-support)),\nwe need to check the `config.json` file\nin the model repository\nand copy the class name under `\"architectures\"`\nand search it on `Bumblebee`'s codebase.\n\nFor example,\nhere's one of the more popular image captioning models -\nSalesforce's `BLIP` -\nhttps://huggingface.co/Salesforce/blip-image-captioning-large/blob/main/config.json.\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"48%\" src=\"https://github.com/elixir-nx/bumblebee/assets/17494745/33dc869f-37f7-4d18-b126-3a0bd0d578d3\" /\u003e\n \u003cimg width=\"48%\" src=\"https://github.com/elixir-nx/bumblebee/assets/17494745/8f1d115c-171b-42bf-b974-08172c957a09\" /\u003e\n\u003c/p\u003e\n\nIf you visit `Bumblebee`'s codebase\nand search for the class name,\nyou'll find it is supported.\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"800\" src=\"https://github.com/elixir-nx/bumblebee/assets/17494745/500eb97b-c20a-4c9a-846e-327cdcd1c37c\" /\u003e\n\u003c/p\u003e\n\nAwesome!\nNow we can use it!\n\nIf you dig around `Bumblebee`'s docs as well\n(https://hexdocs.pm/bumblebee/Bumblebee.Vision.html#image_to_text/5),\nyou'll see that we've got to use `image_to_text/5` with this model.\nIt needs a `tokenizer`, `featurizer` and a `generation-config`\nso we can use it.\n\nLet's do it!\nHead over to `lib/app/application.ex`,\nand change the `serving/0` function.\n\n```elixir\n def serving do\n {:ok, model_info} = Bumblebee.load_model({:hf, \"Salesforce/blip-image-captioning-base\"})\n {:ok, featurizer} = Bumblebee.load_featurizer({:hf, \"Salesforce/blip-image-captioning-base\"})\n {:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, \"Salesforce/blip-image-captioning-base\"})\n {:ok, generation_config} = Bumblebee.load_generation_config({:hf, \"Salesforce/blip-image-captioning-base\"})\n\n Bumblebee.Vision.image_to_text(model_info, featurizer, tokenizer, generation_config,\n compile: [batch_size: 10],\n defn_options: [compiler: EXLA]\n )\n end\n```\n\nAs you can see, we're using the repository name of `BLIP`'s model\nfrom the HuggingFace website.\n\nIf you run `mix phx.server`,\nyou'll see that it will download the new models,\ntokenizers, featurizer and configs to run the model.\n\n```sh\n|======================================================================| 100% (989.82 MB)\n[info] TfrtCpuClient created.\n|======================================================================| 100% (711.39 KB)\n[info] Running AppWeb.Endpoint with cowboy 2.10.0 at 127.0.0.1:4000 (http)\n[info] Access AppWeb.Endpoint at http://localhost:4000\n[watch] build finished, watching for changes...\n```\n\nYou may think we're done here.\nBut we are not! βœ‹\n\nThe **destructuring of the output of the model may not be the same**. \u003cbr /\u003e\nIf you try to submit a photo,\nyou'll get this error:\n\n```sh\nno match of right hand side value:\n%{results: [%{text: \"a person holding a large blue ball on a beach\"}]}\n```\n\nThis means that we need to make some changes\nwhen parsing the output of the model πŸ˜€.\n\nHead over to `lib/app_web/live/page_live.ex`\nand change the `handle_info/3` function\nthat is called after the async task is completed.\n\n```elixir\n def handle_info({ref, result}, %{assigns: %{task_ref: ref}} = socket) do\n Process.demonitor(ref, [:flush])\n\n %{results: [%{text: label}]} = result # change this line\n\n {:noreply, assign(socket, label: label, upload_running?: false)}\n end\n```\n\nAs you can see, we are now correctly destructuring the result from the model.\nAnd that's it!\n\nIf you run `mix phx.server`,\nyou'll see that we got far more accurate results!\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"800\" src=\"https://github.com/elixir-nx/bumblebee/assets/17494745/0cae2db0-5ca4-4434-9c63-76aadb7d578b\" /\u003e\n\u003c/p\u003e\n\nAwesome! πŸŽ‰\n\n\u003e [!NOTE]\n\u003e Be aware that `BLIP`\n\u003e is a _much_ larger model than `ResNet-50`.\n\u003e There are more accurate and even larger models out there\n\u003e (e.g:\n\u003e [`blip-image-captioning-large`](https://huggingface.co/Salesforce/blip-image-captioning-large),\n\u003e the larger version of the model we've just used).\n\u003e This is a balancing act: the larger the model, the longer a prediction may take\n\u003e and more resources your server will need to have to handle this heavier workload.\n\n\u003e [!WARNING]\n\u003e\n\u003e We've created a small module that allows you to have multiple models\n\u003e cached and downloaded and keep this logic contained.\n\u003e\n\u003e For this, check the [`deployment guide`](./deployment.md#5-a-better-model-management).\n\n### 7. How do I deploy this thing?\n\nThere are a few considerations you may want to have\nbefore considering deploying this.\nLuckily for you,\nwe've created a small document\nthat will **guide you through deploying this app in `fly.io`**!\n\nCheck the [`deployment.md`](./deployment.md) file for more information.\n\n### 8. Showing example images\n\n\u003e [!WARNING]\n\u003e\n\u003e This section assumes you've made the changes made in the previous section.\n\u003e Therefore, you should follow the instructions in\n\u003e [`7. How do I deploy this thing?`](#7-how-do-i-deploy-this-thing)\n\u003e and come back after you're done.\n\nWe have a fully functioning application that predicts images.\nNow we can add some cool touches to show the person\nsome examples if they are inactive.\n\nFor this,\nwe are going to need to make **three changes**.\n\n- create a hook in the **client** (`Javascript`)\n to send an event when there's inactivity after\n a given number of seconds.\n- change `page_live.ex` **LiveView** to accommodate\n this new event.\n- change the **view** m `page_live.html.heex`\n to show these changes to the person.\n\nLet's go over each one!\n\n#### 8.1 Creating a hook in client\n\nWe are going to detect the inactivity of the person\nwith some `Javascript` code.\n\nHead over to `assets/js/app.js`\nand change it to the following.\n\n```js\n// If you want to use Phoenix channels, run `mix help phx.gen.channel`\n// to get started and then uncomment the line below.\n// import \"./user_socket.js\"\n\n// You can include dependencies in two ways.\n//\n// The simplest option is to put them in assets/vendor and\n// import them using relative paths:\n//\n// import \"../vendor/some-package.js\"\n//\n// Alternatively, you can `npm install some-package --prefix assets` and import\n// them using a path starting with the package name:\n//\n// import \"some-package\"\n//\n\n// Include phoenix_html to handle method=PUT/DELETE in forms and buttons.\nimport \"phoenix_html\";\n// Establish Phoenix Socket and LiveView configuration.\nimport { Socket } from \"phoenix\";\nimport { LiveSocket } from \"phoenix_live_view\";\nimport topbar from \"../vendor/topbar\";\n\n// Hooks to track inactivity\nlet Hooks = {};\nHooks.ActivityTracker = {\n mounted() {\n // Set the inactivity duration in milliseconds\n const inactivityDuration = 8000; // 8 seconds\n\n // Set a variable to keep track of the timer and if the process to predict example image has already been sent\n let inactivityTimer;\n let processHasBeenSent = false;\n\n let ctx = this;\n\n // Function to reset the timer\n function resetInactivityTimer() {\n // Clear the previous timer\n clearTimeout(inactivityTimer);\n\n // Start a new timer\n inactivityTimer = setTimeout(() =\u003e {\n // Perform the desired action after the inactivity duration\n // For example, send a message to the Elixir process using Phoenix Socket\n if (!processHasBeenSent) {\n processHasBeenSent = true;\n ctx.pushEvent(\"show_examples\", {});\n }\n }, inactivityDuration);\n }\n\n // Call the function to start the timer initially\n resetInactivityTimer();\n\n // Reset the timer whenever there is user activity\n document.addEventListener(\"mousemove\", resetInactivityTimer);\n document.addEventListener(\"keydown\", resetInactivityTimer);\n },\n};\n\nlet csrfToken = document\n .querySelector(\"meta[name='csrf-token']\")\n .getAttribute(\"content\");\nlet liveSocket = new LiveSocket(\"/live\", Socket, {\n hooks: Hooks,\n params: { _csrf_token: csrfToken },\n});\n\n// Show progress bar on live navigation and form submits\ntopbar.config({ barColors: { 0: \"#29d\" }, shadowColor: \"rgba(0, 0, 0, .3)\" });\nwindow.addEventListener(\"phx:page-loading-start\", (_info) =\u003e topbar.show(300));\nwindow.addEventListener(\"phx:page-loading-stop\", (_info) =\u003e topbar.hide());\n\n// connect if there are any LiveViews on the page\nliveSocket.connect();\n\n// expose liveSocket on window for web console debug logs and latency simulation:\n// \u003e\u003e liveSocket.enableDebug()\n// \u003e\u003e liveSocket.enableLatencySim(1000) // enabled for duration of browser session\n// \u003e\u003e liveSocket.disableLatencySim()\nwindow.liveSocket = liveSocket;\n```\n\n- we have added a `Hooks` variable,\n with a property **`ActivityTracker`**.\n This hook has the `mounted()` function\n that is executed when the component that is hooked with this hook is mounted.\n You can find more information at https://hexdocs.pm/phoenix_live_view/js-interop.html.\n- inside the `mounted()` function,\n we create a `resetInactivityTimer()` function\n that is executed every time the\n **mouse moves** (`mousemove` event)\n and a **key is pressed**(`keydown`).\n This function resets the timer\n that is run whilst there is a lack of inactivity.\n- if the person is inactive for _8 seconds_,\n we create an event `\"show_examples\"`.\n We will create a handler in the LiveView\n to handle this event later.\n- we add the `Hooks` variable to the `hooks`\n property when initializing the `livesocket`.\n\nAnd that's it!\n\nFor this hook to _actually be executed_,\nwe need to create a component that uses it\ninside our view file.\n\nFor this, we can simply create a hidden component\non top of the `lib/app_web/live/page_live.html.heex` file.\n\nAdd the following hidden component.\n\n```html\n\u003cdiv class=\"hidden\" id=\"tracker_el\" phx-hook=\"ActivityTracker\" /\u003e\n```\n\nWe use the `phx-hook` attribute\nto bind the hook we've created in our `app.js` file\nto the component so it's executed.\nWhen this component is mounted,\nthe `mounted()` function inside the hook\nis executed.\n\nAnd that's it! πŸ‘\n\nYour app won't work yet because\nwe haven't created a handler\nto handle the `\"show_examples\"` event.\n\nLet's do that right now!\n\n### 8.2 Handling the example images list event inside our LiveView\n\nNow that we have our client sorted,\nlet's head over to our LiveView\nat `lib/app_web/live/page_live.ex`\nand make the needed changes!\n\nBefore anything,\nlet's add socket assigns that we will need throughout our adventure!\nOn top of the file,\nchange the socket assigns to the following.\n\n```elixir\n socket\n |\u003e assign(\n # Related to the file uploaded by the user\n label: nil,\n upload_running?: false,\n task_ref: nil,\n image_preview_base64: nil,\n\n # Related to the list of image examples\n example_list_tasks: [],\n example_list: [],\n display_list?: false\n )\n```\n\nWe've added **three new assigns**:\n\n- **`example_list_tasks`** is a list of\n the async tasks that are created for each example image.\n- **`example_list`** is a list of the example images\n with their respective predictions.\n- **`display_list?`** is a boolean that\n tells us if the list is to be shown or not.\n\nAwesome! Let's continue.\n\nAs we've mentioned before,\nwe need to create a handler for our `\"show_examples\"` event.\n\nAdd the following function to the file.\n\n```elixir\n\n @image_width 640\n\n def handle_event(\"show_examples\", _data, socket) do\n\n # Only run if the user hasn't uploaded anything\n if(is_nil(socket.assigns.task_ref)) do\n # Retrieves a random image from Picsum with a given `image_width` dimension\n random_image = \"https://picsum.photos#{@image_width}/#{@image_width}\"\n\n # Spawns prediction tasks for example image from random Picsum image\n tasks = for _ \u003c- 1..2 do\n {:req, body} = {:req, Req.get!(random_image).body}\n predict_example_image(body)\n end\n\n\n # List to change `example_list` socket assign to show skeleton loading\n display_example_images = Enum.map(tasks, fn obj -\u003e %{predicting?: true, ref: obj.ref} end)\n\n # Updates the socket assigns\n {:noreply, assign(socket, example_list_tasks: tasks, example_list: display_example_images)}\n\n else\n {:noreply, socket}\n end\n end\n\n```\n\n\u003e [!WARNING]\n\u003e\n\u003e We are using the\n\u003e [`req`](https://github.com/wojtekmach/req) package\n\u003e to download the file binary from the URL.\n\u003e Make sure to install it in the `mix.exs` file.\n\n- we are using the [Picsum API](https://picsum.photos),\n an _awesome_ image API with lots of photos!\n They provide a `/random` URL that yields a random photo.\n In this URL we can inclusively define the dimensions we want!\n That's what we're doing in the first line of the function.\n We are using a module constant `@image_width 640` on top of the file,\n so add that to the top of the file.\n This function is relevant because we preferably want to deal\n with images that are in the same resolution as the dataset the model was trained in.\n- we are creating **two async tasks** that retrieve the binary of the image\n and pass it on to a `predict_example_image/1` function\n (we will create this function next).\n- the two tasks that we've created are in an array `tasks`.\n We create _another array_ `display_example_images` with the same number of elements as `tasks`,\n which will have two properties:\n **`predicting`**, meaning if the image is being predicted by the model;\n and **`ref`**, the reference of the task.\n- we assign the `tasks` array to the `example_list_tasks` socket assign\n and `display_example_images` array to the `example_list` array.\n So `example_list` will temporarily hold\n objects with `:predicting` and `:ref` properties\n whilst the model is being executed.\n\nAs we've just mentioned,\nwe are making use of a function called\n`predict_example_image/1` to make predictions\nof a given file binary.\n\nLet's implement it now!\nIn the same file, add:\n\n```elixir\n def predict_example_image(body) do\n with {:vix, {:ok, img_thumb}} \u003c-\n {:vix, Vix.Vips.Operation.thumbnail_buffer(body, @image_width)},\n {:pre_process, {:ok, t_img}} \u003c-\n {:pre_process, pre_process_image(img_thumb)} do\n\n # Create an async task to classify the image from Picsum\n Task.Supervisor.async(App.TaskSupervisor, fn -\u003e\n Nx.Serving.batched_run(ImageClassifier, t_img)\n end)\n |\u003e Map.merge(%{base64_encoded_url: \"data:image/png;base64, \" \u003c\u003e Base.encode64(body)})\n\n else\n {stage, error} -\u003e {stage, error}\n end\n end\n```\n\nFor the body of the image to be executed by the model,\nit needs to go through some pre-processing.\n\n- we are using [`thumbnail_buffer/3`](https://hexdocs.pm/vix/Vix.Vips.Operation.html#thumbnail_buffer/3)\n to make sure it's properly resized\n and then feeding the result to our own implemented\n `pre_process_image/1` function\n so it can be converted to a parseable tensor by the model.\n- after these two operations are successfully completed,\n we spawn two async tasks (like we've done before)\n and feed it to the model.\n We add the base64-encoded image\n to the return value so it can later be shown to the person.\n- if these operations fail, we return an error.\n\nGreat job! πŸ‘\n\nOur example images async tasks have successfully been created\nand are on their way to the model!\n\nNow we need to handle these newly created async tasks\nonce they are completed.\nAs we know, we are handling our async tasks completion\nin the `def handle_info({ref, result}, %{assigns: %{task_ref: ref}} = socket)` function.\nLet's change it like so.\n\n```elixir\n def handle_info({ref, result}, %{assigns: assigns} = socket) do\n # Flush async call\n Process.demonitor(ref, [:flush])\n\n # You need to change how you destructure the output of the model depending\n # on the model you've chosen for `prod` and `test` envs on `models.ex`.)\n label =\n case Application.get_env(:app, :use_test_models, false) do\n true -\u003e\n App.Models.extract_test_label(result)\n\n # coveralls-ignore-start\n false -\u003e\n App.Models.extract_prod_label(result)\n # coveralls-ignore-stop\n end\n\n cond do\n\n # If the upload task has finished executing, we update the socket assigns.\n Map.get(assigns, :task_ref) == ref -\u003e\n {:noreply, assign(socket, label: label, upload_running?: false)}\n\n # If the example task has finished executing, we upload the socket assigns.\n img = Map.get(assigns, :example_list_tasks) |\u003e Enum.find(\u0026(\u00261.ref == ref)) -\u003e\n\n # Update the element in the `example_list` enum to turn \"predicting?\" to `false`\n updated_example_list = Map.get(assigns, :example_list)\n |\u003e Enum.map(fn obj -\u003e\n if obj.ref == img.ref do\n Map.put(obj, :base64_encoded_url, img.base64_encoded_url)\n |\u003e Map.put(:label, label)\n |\u003e Map.put(:predicting?, false)\n\n else\n obj\n end end)\n\n {:noreply,\n assign(socket,\n example_list: updated_example_list,\n upload_running?: false,\n display_list?: true\n )}\n end\n end\n```\n\nThe only change we've made is that\nwe've added a [`cond`](https://hexdocs.pm/elixir/1.16/case-cond-and-if.html#cond)\nflow structure.\nWe are essentially checking\nif the task reference that has been completed\nis **from the uploaded image from the person** (`:task_ref` socket assign)\nor **from an example image** (inside the `:example_list_tasks` socket assign list).\n\nIf it's the latter,\nwe retrieve we are updating\nthe `example_list` socket assign list\nwith the **prediction** (`:label`),\nthe **base64-encoded image from the task list** (`:base64_encoded_url`)\nand setting the `:predicting` property to `false`.\n\nAnd that's it!\nGreat job! πŸ₯³\n\n#### 8.3 Updating the view\n\nNow that we've made all the necessary changes to our LiveView,\nwe need to update our view so it reflects them!\n\nHead over to `lib/app_web/live/page_live.html.heex`\nand change it to the following piece of code.\n\n```html\n\u003cdiv class=\"hidden\" id=\"tracker_el\" phx-hook=\"ActivityTracker\" /\u003e\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-24 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv class=\"flex flex-col justify-start\"\u003e\n \u003cdiv class=\"flex justify-center items-center w-full\"\u003e\n \u003cdiv class=\"2xl:space-y-12\"\u003e\n \u003cdiv class=\"mx-auto max-w-2xl lg:text-center\"\u003e\n \u003cp\u003e\n \u003cspan\n class=\"rounded-full w-fit bg-brand/5 px-2 py-1 text-[0.8125rem] font-medium text-center leading-6 text-brand\"\n \u003e\n \u003ca\n href=\"https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n \u003e\n πŸ”₯ LiveView\n \u003c/a\u003e\n +\n \u003ca\n href=\"https://github.com/elixir-nx/bumblebee\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n \u003e\n 🐝 Bumblebee\n \u003c/a\u003e\n \u003c/span\u003e\n \u003c/p\u003e\n \u003cp\n class=\"mt-2 text-3xl font-bold tracking-tight text-gray-900 sm:text-4xl\"\n \u003e\n Caption your image!\n \u003c/p\u003e\n \u003ch3 class=\"mt-6 text-lg leading-8 text-gray-600\"\u003e\n Upload your own image (up to 5MB) and perform image captioning with\n \u003ca\n href=\"https://elixir-lang.org/\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n class=\"font-mono font-medium text-sky-500\"\n \u003e\n Elixir\n \u003c/a\u003e\n !\n \u003c/h3\u003e\n \u003cp class=\"text-lg leading-8 text-gray-400\"\u003e\n Powered with\n \u003ca\n href=\"https://elixir-lang.org/\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n class=\"font-mono font-medium text-sky-500\"\n \u003e\n HuggingFaceπŸ€—\n \u003c/a\u003e\n transformer models, you can run this project locally and perform\n machine learning tasks with a handful lines of code.\n \u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv class=\"border-gray-900/10\"\u003e\n \u003c!-- File upload section --\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n phx-drop-target=\"{@uploads.image_list.ref}\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003c!-- Show image preview --\u003e\n \u003c%= if @image_preview_base64 do %\u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e\n \u003cimg src=\"{@image_preview_base64}\" /\u003e\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c% else %\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c!-- Show errors --\u003e\n \u003c%= for entry \u003c- @uploads.image_list.entries do %\u003e\n \u003cdiv class=\"mt-2\"\u003e\n \u003c%= for err \u003c- upload_errors(@uploads.image_list, entry) do %\u003e\n \u003cdiv class=\"rounded-md bg-red-50 p-4 mb-2\"\u003e\n \u003cdiv class=\"flex\"\u003e\n \u003cdiv class=\"flex-shrink-0\"\u003e\n \u003csvg\n class=\"h-5 w-5 text-red-400\"\n viewBox=\"0 0 20 20\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M10 18a8 8 0 100-16 8 8 0 000 16zM8.28 7.22a.75.75 0 00-1.06 1.06L8.94 10l-1.72 1.72a.75.75 0 101.06 1.06L10 11.06l1.72 1.72a.75.75 0 101.06-1.06L11.06 10l1.72-1.72a.75.75 0 00-1.06-1.06L10 8.94 8.28 7.22z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003c/div\u003e\n \u003cdiv class=\"ml-3\"\u003e\n \u003ch3 class=\"text-sm font-medium text-red-800\"\u003e\n \u003c%= error_to_string(err) %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c!-- Prediction text --\u003e\n \u003cdiv\n class=\"flex mt-2 space-x-1.5 items-center font-bold text-gray-900 text-xl\"\n \u003e\n \u003cspan\u003eDescription: \u003c/span\u003e\n \u003c!-- Spinner --\u003e\n \u003c%= if @upload_running? do %\u003e\n \u003cdiv role=\"status\"\u003e\n \u003cdiv\n class=\"relative w-6 h-6 animate-spin rounded-full bg-gradient-to-r from-purple-400 via-blue-500 to-red-400 \"\n \u003e\n \u003cdiv\n class=\"absolute top-1/2 left-1/2 transform -translate-x-1/2 -translate-y-1/2 w-3 h-3 bg-gray-200 rounded-full border-2 border-white\"\n \u003e\u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% else %\u003e \u003c%= if @label do %\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @label %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light\"\u003eWaiting for image input.\u003c/span\u003e\n \u003c% end %\u003e \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c!-- Examples --\u003e\n \u003c%= if @display_list? do %\u003e\n \u003cdiv class=\"flex flex-col\"\u003e\n \u003ch3\n class=\"mt-10 text-xl lg:text-center font-light tracking-tight text-gray-900 lg:text-2xl\"\n \u003e\n Examples\n \u003c/h3\u003e\n \u003cdiv class=\"flex flex-row justify-center my-8\"\u003e\n \u003cdiv\n class=\"mx-auto grid max-w-2xl grid-cols-1 gap-x-6 gap-y-20 sm:grid-cols-2\"\n \u003e\n \u003c%= for example_img \u003c- @example_list do %\u003e\n\n \u003c!-- Loading skeleton if it is predicting --\u003e\n \u003c%= if example_img.predicting? == true do %\u003e\n \u003cdiv\n role=\"status\"\n class=\"flex items-center justify-center w-full h-full max-w-sm bg-gray-300 rounded-lg animate-pulse\"\n \u003e\n \u003csvg\n class=\"w-10 h-10 text-gray-200 dark:text-gray-600\"\n aria-hidden=\"true\"\n xmlns=\"http://www.w3.org/2000/svg\"\n fill=\"currentColor\"\n viewBox=\"0 0 20 18\"\n \u003e\n \u003cpath\n d=\"M18 0H2a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V2a2 2 0 0 0-2-2Zm-5.5 4a1.5 1.5 0 1 1 0 3 1.5 1.5 0 0 1 0-3Zm4.376 10.481A1 1 0 0 1 16 15H4a1 1 0 0 1-.895-1.447l3.5-7A1 1 0 0 1 7.468 6a.965.965 0 0 1 .9.5l2.775 4.757 1.546-1.887a1 1 0 0 1 1.618.1l2.541 4a1 1 0 0 1 .028 1.011Z\"\n /\u003e\n \u003c/svg\u003e\n \u003cspan class=\"sr-only\"\u003eLoading...\u003c/span\u003e\n \u003c/div\u003e\n\n \u003c% else %\u003e\n \u003cdiv\u003e\n \u003cimg\n id=\"{example_img.base64_encoded_url}\"\n src=\"{example_img.base64_encoded_url}\"\n class=\"rounded-2xl object-cover\"\n /\u003e\n \u003ch3 class=\"mt-1 text-lg leading-8 text-gray-900 text-center\"\u003e\n \u003c%= example_img.label %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c% end %\u003e \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nWe've made two changes.\n\n- we've added some text to better introduce our application\n at the top of the page.\n- added a section to show the example images list.\n This section is only rendered if the\n **`display_list?`** socket assign is set to `true`.\n If so, we iterate over the\n **`example_list`** socket assign list\n and show a [loading skeleton](https://www.freecodecamp.org/news/how-to-build-skeleton-screens-using-css-for-better-user-experience/)\n if the image is `:predicting`.\n If not, it means the image has already been predicted,\n and we show the base64-encoded image\n like we do with the image uploaded by the person.\n\nAnd that's it! πŸŽ‰\n\n#### 8.4 Using URL of image instead of base64-encoded\n\nWhile our example list is being correctly rendered,\nwe are using additional CPU\nto base64 encode our images\nso they can be shown to the person.\n\nInitially, we did this because\n`https://picsum.photos`\nresolves into a different URL every time it is called.\nThis means that the image that was fed into the model\nwould be different from the one shown in the example list\nif we were to use this URL in our view.\n\nTo fix this,\n**we need to follow the redirection when the URL is resolved**.\n\n\u003e [!NOTE]\n\u003e We can do this with\n\u003e [`Finch`](https://github.com/sneako/finch),\n\u003e if we wanted to.\n\u003e\n\u003e We could do something like\n\u003e def rand_splash do\n\u003e\n\u003e ```elixir\n\u003e %{scheme: scheme, host: host, path: path} =\n\u003e Finch.build(:get, \"https://picsum.photos\")\n\u003e |\u003e Finch.request!(MyFinch)\n\u003e |\u003e Map.get(:headers)\n\u003e |\u003e Enum.filter(fn {a, _b} -\u003e a == \"location\" end)\n\u003e |\u003e List.first()\n\u003e |\u003e elem(1)\n\u003e |\u003e URI.parse()\n\u003e\n\u003e scheme \u003c\u003e \"://\" \u003c\u003e host \u003c\u003e path\n\u003e end\n\u003e ```\n\u003e\n\u003e And then call it, like so.\n\u003e\n\u003e ```elixir\n\u003e App.rand_splash()\n\u003e # https://images.unsplash.com/photo-1694813646634-9558dc7960e3\n\u003e ```\n\nBecause we are already using\n[`req`](https://github.com/wojtekmach/req),\nlet's make use of it\ninstead of adding additional dependencies.\n\nLet's first add a function that will do this\nin `lib/app_web/live/page_live.ex`.\nAdd the following piece of code\nat the end of the file.\n\n```elixir\n defp track_redirected(url) do\n # Create request\n req = Req.new(url: url)\n\n # Add tracking properties to req object\n req = req\n |\u003e Req.Request.register_options([:track_redirected])\n |\u003e Req.Request.prepend_response_steps(track_redirected: \u0026track_redirected_uri/1)\n\n # Make request\n {:ok, response} = Req.request(req)\n\n # Return the final URI\n %{url: URI.to_string(response.private.final_uri), body: response.body}\n end\n\n defp track_redirected_uri({request, response}) do\n {request, %{response | private: Map.put(response.private, :final_uri, request.url)}}\n end\n```\n\nThis function adds properties to the request object\nand tracks the redirection.\nIt will add a [`URI`](https://hexdocs.pm/elixir/1.12/URI.html#summary)\nobject inside `private.final_uri`.\nThis function returns\nthe `body` of the image\nand the final `url`\nit is resolved to\n(the URL of the image).\n\nNow all we need to do is use this function!\nHead over to the `handle_event(\"show_examples\"...` function\nand change the loop to the following.\n\n```elixir\n tasks = for _ \u003c- 1..2 do\n %{url: url, body: body} = track_redirected(random_image)\n predict_example_image(body, url)\n end\n```\n\nWe are making use of `track_redirected/1`,\nthe function we've just created.\nWe pass both `body` and `url` to `predict_example_image/1`,\nwhich we will now change.\n\n```elixir\n def predict_example_image(body, url) do\n with {:vix, {:ok, img_thumb}} \u003c-\n {:vix, Vix.Vips.Operation.thumbnail_buffer(body, @image_width)},\n {:pre_process, {:ok, t_img}} \u003c- {:pre_process, pre_process_image(img_thumb)} do\n\n # Create an async task to classify the image from Picsum\n Task.Supervisor.async(App.TaskSupervisor, fn -\u003e\n Nx.Serving.batched_run(ImageClassifier, t_img)\n end)\n |\u003e Map.merge(%{url: url})\n\n else\n {:vix, {:error, msg}} -\u003e {:error, msg}\n {:pre_process, {:error, msg}} -\u003e {:error, msg}\n end\n end\n```\n\nInstead of using `base64_encoded_url`,\nwe are now using the `url` we've acquired.\n\nThe last step we need to do in our LiveView\nis to finally use this `url`\nin `handle_info/3`.\n\n```elixir\n def handle_info({ref, result}, %{assigns: assigns} = socket) do\n Process.demonitor(ref, [:flush])\n\n label =\n case Application.get_env(:app, :use_test_models, false) do\n true -\u003e\n App.Models.extract_test_label(result)\n\n false -\u003e\n App.Models.extract_prod_label(result)\n end\n\n cond do\n\n Map.get(assigns, :task_ref) == ref -\u003e\n {:noreply, assign(socket, label: label, upload_running?: false)}\n\n img = Map.get(assigns, :example_list_tasks) |\u003e Enum.find(\u0026(\u00261.ref == ref)) -\u003e\n\n updated_example_list = Map.get(assigns, :example_list)\n |\u003e Enum.map(fn obj -\u003e\n if obj.ref == img.ref do\n obj\n |\u003e Map.put(:url, img.url) # change here\n |\u003e Map.put(:label, label)\n |\u003e Map.put(:predicting?, false)\n\n else\n obj\n end end)\n\n {:noreply,\n assign(socket,\n example_list: updated_example_list,\n upload_running?: false,\n display_list?: true\n )}\n end\n end\n```\n\nAnd that's it!\n\nThe last thing we need to do is change our view\nso it uses the `:url` parameter\ninstead of the obsolete `:base64_encoded_url`.\n\nHead over to `lib/app_web/live/page_live.html.heex`\nand change the `\u003cimg\u003e` being shown in the example list\nso it uses the `:url` parameter.\n\n```html\n\u003cimg\n id=\"{example_img.url}\"\n src=\"{example_img.url}\"\n class=\"rounded-2xl object-cover\"\n/\u003e\n```\n\nAnd we're done! πŸŽ‰\n\nWe are now rendering the image on the client\nthrough the URL the Picsum API resolves into\ninstead of having the LiveView server\nencoding the image.\nTherefore, we're saving some CPU\nto the thing that matters the most:\n_running our model_.\n\n#### 8.5 See it running\n\nNow let's see our application in action!\nWe are expecting the examples to be shown after\n**8 seconds** of inactivity.\nIf the person is inactive for this time duration,\nwe fetch a random image from Picsum API\nand feed it to our model!\n\nYou should see different images every time you use the app.\nIsn't that cool? 😎\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=800 src=\"https://github.com/dwyl/image-classifier/assets/17494745/1f8d08d1-f6ca-46aa-8c89-4bab45ad1e54\"\u003e\n\u003c/p\u003e\n\n### 9. Store metadata and classification info\n\nOur app is shaping up quite nicely!\nAs it stands, it's an application that does inference on images.\nHowever, it doesn't save them.\n\nLet's expand our application so it has a **database**\nwhere image classification is saved/persisted!\n\nWe'll use **`Postgres`** for this.\nTypically, when you create a new `Phoenix` project\nwith `mix phx.new`,\n[a `Postgres` database will be automatically created](https://hexdocs.pm/phoenix/installation.html#postgresql).\nBecause we didn't do this,\nwe'll have to configure this ourselves.\n\nLet's do it!\n\n#### 9.1 Installing dependencies\n\nWe'll install all the needed dependencies first.\nIn `mix.exs`, add the following snippet\nto the `deps` section.\n\n```elixir\n # HTTP Request\n {:httpoison, \"~\u003e 2.2\"},\n {:mime, \"~\u003e 2.0.5\"},\n {:ex_image_info, \"~\u003e 0.2.4\"},\n\n # DB\n {:phoenix_ecto, \"~\u003e 4.4\"},\n {:ecto_sql, \"~\u003e 3.10\"},\n {:postgrex, \"\u003e= 0.0.0\"},\n```\n\n- [**`httpoison`**](https://github.com/edgurgel/httpoison),\n [**`mime`**](https://hex.pm/packages/mime) and\n [**`ex_image_info`**](https://hex.pm/packages/ex_image_info)\n are used to make HTTP requests,\n get the content type and information from an image file,\n respectively.\n These will be needed to upload a given image to\n [`imgup`](https://github.com/dwyl/imgup),\n by making multipart requests.\n\n- [**`phoenix_ecto`**](https://github.com/phoenixframework/phoenix_ecto),\n [**`ecto_sql`**](https://github.com/elixir-ecto/ecto_sql) and\n [**`postgrex`**](https://github.com/elixir-ecto/postgrex)\n are needed to properly configure our driver in Elixir\n that will connect to a Postgres database,\n in which we will persist data.\n\nRun `mix deps.get` to install these dependencies.\n\n#### 9.2 Adding `Postgres` configuration files\n\nNow let's create the needed files\nto properly connect to a Postgres relational database.\nStart by going to `lib/app`\nand create `repo.ex`.\n\n```elixir\ndefmodule App.Repo do\n use Ecto.Repo,\n otp_app: :app,\n adapter: Ecto.Adapters.Postgres\nend\n```\n\nThis module will be needed in our configuration files\nso our app knows where the database is.\n\nNext, in `lib/app/application.ex`,\nadd the following line to the `children` array\nin the supervision tree.\n\n```elixir\n children = [\n AppWeb.Telemetry,\n App.Repo, # add this line\n {Phoenix.PubSub, name: App.PubSub},\n ...\n ]\n```\n\nAwesome! πŸŽ‰\n\nNow let's head over to the files inside the `config` folder.\n\nIn `config/config.exs`,\nadd these lines.\n\n```elixir\nconfig :app,\n ecto_repos: [App.Repo],\n generators: [timestamp_type: :utc_datetime]\n```\n\nWe are referring the module we've previously created (`App.Repo`)\nto `ecto_repos` so `ecto` knows where the configuration\nfor the database is located.\n\nIn `config/dev.exs`,\nadd:\n\n```elixir\nconfig :app, App.Repo,\n username: \"postgres\",\n password: \"postgres\",\n hostname: \"localhost\",\n database: \"app_dev\",\n stacktrace: true,\n show_sensitive_data_on_connection_error: true,\n pool_size: 10\n```\n\nWe are defining the parameters of the database\nthat is used during development.\n\nIn `config/runtime.exs`,\nadd:\n\n```elixir\nif config_env() == :prod do\n database_url =\n System.get_env(\"DATABASE_URL\") ||\n raise \"\"\"\n environment variable DATABASE_URL is missing.\n For example: ecto://USER:PASS@HOST/DATABASE\n \"\"\"\n\n maybe_ipv6 = if System.get_env(\"ECTO_IPV6\") in ~w(true 1), do: [:inet6], else: []\n\n config :app, App.Repo,\n # ssl: true,\n url: database_url,\n pool_size: String.to_integer(System.get_env(\"POOL_SIZE\") || \"10\"),\n socket_options: maybe_ipv6\n\n # ...\n```\n\nWe are configuring the runtime database configuration..\n\nIn `config/test.exs`,\nadd:\n\n```elixir\nconfig :app, App.Repo,\n username: \"postgres\",\n password: \"postgres\",\n hostname: \"localhost\",\n database: \"app_test#{System.get_env(\"MIX_TEST_PARTITION\")}\",\n pool: Ecto.Adapters.SQL.Sandbox,\n pool_size: 10\n```\n\nHere we're defining the database used during testing.\n\nNow let's create a **migration file** to create our database table.\nIn `priv/repo/migrations/`, create a file\ncalled `20231204092441_create_images.exs`\n(or any other timestamp string)\nwith the following piece of code.\n\n```elixir\ndefmodule App.Repo.Migrations.CreateImages do\n use Ecto.Migration\n\n def change do\n create table(:images) do\n add :url, :string\n add :description, :string\n add :width, :integer\n add :height, :integer\n\n timestamps(type: :utc_datetime)\n end\n end\nend\n```\n\nAnd that's it!\nThose are the needed files that our application needs\nto properly connect and persist data into the Postgres database.\n\nYou can now run `mix ecto.create` and `mix ecto.migrate`\nto create the database\nand the `\"images\"` table.\n\n#### 9.3 Creating `Image` schema\n\nFor now, let's create a simple table `\"images\"`\nin our database\nthat has the following properties:\n\n- **`description`**: the description of the image\n from the model.\n- **`width`**: width of the image.\n- **`height`**: height of the image.\n- **`url`**: public URL where the image is publicly hosted.\n\nWith this in mind, let's create a new file!\nIn `lib/app/`, create a file called `image.ex`.\n\n```elixir\ndefmodule App.Image do\n use Ecto.Schema\n alias App.{Image, Repo}\n\n @primary_key {:id, :id, autogenerate: true}\n schema \"images\" do\n field(:description, :string)\n field(:width, :integer)\n field(:url, :string)\n field(:height, :integer)\n\n timestamps(type: :utc_datetime)\n end\n\n def changeset(image, params \\\\ %{}) do\n image\n |\u003e Ecto.Changeset.cast(params, [:url, :description, :width, :height])\n |\u003e Ecto.Changeset.validate_required([:url, :description, :width, :height])\n end\n\n @doc \"\"\"\n Uploads the given image to S3\n and adds the image information to the database.\n \"\"\"\n def insert(image) do\n %Image{}\n |\u003e changeset(image)\n |\u003e Repo.insert!()\n end\nend\n```\n\nWe've just created the `App.Image` schema\nwith the aforementioned fields.\n\nWe've created `changeset/1`, which is used to cast\nand validate the properties of a given object\nbefore interacting with the database.\n\n`insert/1` receives an object,\nruns it through the changeset\nand inserts it in the database.\n\n#### 9.4 Changing our LiveView to persist data\n\nNow that we have our database set up,\nlet's change some of our code so we persist data into it!\nIn this section, we'll be working in the `lib/app_web/live/page_live.ex` file.\n\nFirst, let's import `App.Image`\nand create an `ImageInfo` struct to hold the information\nof the image throughout the process of uploading\nand classifying the image.\n\n```elixir\ndefmodule AppWeb.PageLive do\n use AppWeb, :live_view\n alias App.Image # add this import\n alias Vix.Vips.Image, as: Vimage\n\n\n defmodule ImageInfo do\n @doc \"\"\"\n General information for the image that is being analysed.\n This information is useful when persisting the image to the database.\n \"\"\"\n defstruct [:mimetype, :width, :height, :url, :file_binary]\n end\n\n # ...\n```\n\nWe are going to be using `ImageInfo`\nin our socket assigns.\nLet's add to it when the LiveView is mounting!\n\n```elixir\n |\u003e assign(\n label: nil,\n upload_running?: false,\n task_ref: nil,\n image_info: nil, # add this line\n image_preview_base64: nil,\n example_list_tasks: [],\n example_list: [],\n display_list?: false\n )\n```\n\nWhen the person uploads an image,\nwe want to retrieve its info (namely its _height_, _width_)\nand upload the image to an `S3` bucket (we're doing this through `imgup`)\nso we can populate the `:url` field of the schema in the database.\n\nWe can retrieve this information _while consuming the entry_\n/uploading the image file.\nFor this, go to `handle_progress(:image_list, entry, socket)`\nand change the function to the following.\n\n```elixir\n def handle_progress(:image_list, entry, socket) when entry.done? do\n # We've changed the object that is returned from `consume_uploaded_entry/3` to return an `image_info` object.\n %{tensor: tensor, image_info: image_info} =\n consume_uploaded_entry(socket, entry, fn %{} = meta -\u003e\n file_binary = File.read!(meta.path)\n\n # Add this line. It uses `ExImageInfo` to retrieve the info from the file binary.\n {mimetype, width, height, _variant} = ExImageInfo.info(file_binary)\n\n {:ok, thumbnail_vimage} =\n Vix.Vips.Operation.thumbnail(meta.path, @image_width, size: :VIPS_SIZE_DOWN)\n\n {:ok, tensor} = pre_process_image(thumbnail_vimage)\n\n # Add this line. Uploads the image to the S3, which returns the `url` and `compressed url`.\n # (we'll implement this function next)\n url = Image.upload_image_to_s3(meta.path, mimetype) |\u003e Map.get(\"url\")\n\n # Add this line. We are constructing the image_info object to be returned.\n image_info = %ImageInfo{mimetype: mimetype, width: width, height: height, file_binary: file_binary, url: url}\n\n # Return it\n {:ok, %{tensor: tensor, image_info: image_info}}\n end)\n\n task =\n Task.Supervisor.async(App.TaskSupervisor, fn -\u003e\n Nx.Serving.batched_run(ImageClassifier, tensor)\n end)\n\n base64 = \"data:image/png;base64, \" \u003c\u003e Base.encode64(image_info.file_binary)\n\n # Change this line so `image_info` is defined when the image is uploaded\n {:noreply, assign(socket, upload_running?: true, task_ref: task.ref, image_preview_base64: base64, image_info: image_info)}\n #else\n # {:noreply, socket}\n #end\n end\n```\n\nCheck the comment lines for more explanation on the changes that have bee nmade.\nWe are using `ExImageInfo` to fetch the information from the image\nand assigning it to the `image_info` socket we defined earlier.\n\nWe are also using `Image.upload_image_to_s3/2` to upload our image to `imgup`.\nLet's define this function in `lib/app/image.ex`.\n\n```elixir\n def upload_image_to_s3(file_path, mimetype) do\n extension = MIME.extensions(mimetype) |\u003e Enum.at(0)\n\n # Upload to Imgup - https://github.com/dwyl/imgup\n upload_response =\n HTTPoison.post!(\n \"https://imgup.fly.dev/api/images\",\n {:multipart,\n [\n {\n :file,\n file_path,\n {\"form-data\", [name: \"image\", filename: \"#{Path.basename(file_path)}.#{extension}\"]},\n [{\"Content-Type\", mimetype}]\n }\n ]},\n []\n )\n\n # Return URL\n Jason.decode!(upload_response.body)\n end\n```\n\nWe're using `HTTPoison` to make a multipart request to the `imgup` server,\neffectively uploading the image to the server.\nIf the upload is successful, it returns the `url` of the uploaded image.\n\nLet's go back to `lib/app_web/live/page_live.ex`.\nNow that we have `image_info` in the socket assigns,\nwe can use it to **insert a row in the `\"images\"` table in the database**.\nWe only want to do this after the model is done running,\nso simply change `handle_info/2` function\n(which is called after the model is done with classifying the image).\n\n```elixir\n cond do\n\n # If the upload task has finished executing, we update the socket assigns.\n Map.get(assigns, :task_ref) == ref -\u003e\n\n # Insert image to database\n image = %{\n url: assigns.image_info.url,\n width: assigns.image_info.width,\n height: assigns.image_info.height,\n description: label\n }\n Image.insert(image)\n\n # Update socket assigns\n {:noreply, assign(socket, label: label, upload_running?: false)}\n\n\n # ...\n```\n\nIn the `cond do` statement,\nwe want to change the one pertaining to the image that is uploaded,\n_not the example list_ that is defined below.\nWe simply create an `image` variable with information\nthat is passed down to `Image.insert/1`,\neffectively adding the row to the database.\n\nAnd that's it!\n\nNow every time a person uploads an image\nand the model is executed,\nwe are saving its location (`:url`),\ninformation (`:width` and `:height`)\nand the result of the classifying model\n(`:description`).\n\nπŸ₯³\n\n\u003e [!NOTE]\n\u003e\n\u003e If you're curious and want to see the data in your database,\n\u003e we recommend using [`DBeaver`](https://dbeaver.io/),\n\u003e an open-source database manager.\n\u003e\n\u003e You can learn more about it at https://github.com/dwyl/learn-postgresql.\n\n### 10. Adding double MIME type check and showing feedback to the person in case of failure\n\nCurrently, we are not handling any errors\nin case the upload of the image to `imgup` fails.\nAlthough this is not critical,\nit'd be better if we could show feedback to the person\nin case the upload to `imgup` fails.\nThis is good for us as well,\nbecause we _can monitor and locate the error faster_\nif we log the errors.\n\nFor this, let's head over to `lib/app/image.ex`\nand update the `upload_image_to_s3/2` function we've implemented.\n\n```elixir\n def upload_image_to_s3(file_path, mimetype) do\n extension = MIME.extensions(mimetype) |\u003e Enum.at(0)\n\n # Upload to Imgup - https://github.com/dwyl/imgup\n upload_response =\n HTTPoison.post(\n \"https://imgup.fly.dev/api/images\",\n {:multipart,\n [\n {\n :file,\n file_path,\n {\"form-data\", [name: \"image\", filename: \"#{Path.basename(file_path)}.#{extension}\"]},\n [{\"Content-Type\", mimetype}]\n }\n ]},\n []\n )\n\n # Process the response and return error if there was a problem uploading the image\n case upload_response do\n # In case it's successful\n {:ok, %HTTPoison.Response{status_code: 200, body: body}} -\u003e\n %{\"url\" =\u003e url, \"compressed_url\" =\u003e _} = Jason.decode!(body)\n {:ok, url}\n\n # In case it returns HTTP 400 with specific reason it failed\n {:ok, %HTTPoison.Response{status_code: 400, body: body}} -\u003e\n %{\"errors\" =\u003e %{\"detail\" =\u003e reason}} = Jason.decode!(body)\n {:error, reason}\n\n # In case the request fails for whatever other reason\n {:error, %HTTPoison.Error{reason: reason}} -\u003e\n {:error, reason}\n end\n end\n```\n\nAs you can see,\nwe are returning `{:error, reason}` if an error occurs,\nand providing feedback alongside it.\nIf it's successful, we return `{:ok, url}`.\n\nBecause we've just changed this function,\nwe need to also update `def handle_progress(:image_list...`\ninside `lib/app_web/live/page_live.ex`\nto properly handle this new function output.\n\nWe are also introducing a double MIME type check to ensure that only image files are uploaded and processed.\nWe use [GenMagic](https://hexdocs.pm/gen_magic/readme.html). It provides supervised and customisable access to `libmagic` using a supervised external process.\n[This gist](https://gist.github.com/leommoore/f9e57ba2aa4bf197ebc5) explains that Magic numbers are the first bits of a file\nwhich uniquely identifies the type of file.\n\nWe use the GenMagic server as a daemon; it is started in the Application module.\nIt is referenced by its name.\nWhen we run `perform`, we obtain a map and compare the mime type with the one read by `ExImageInfo`.\nIf they correspond with each other, we continue, or else we stop the process.\n\nOn your computer, for this to work locally, you should install the package `libmagic-dev`.\n\n\u003e [!NOTE]\n\u003e\n\u003e Depending on your OS, you may install `libmagic` in different ways.\n\u003e A quick Google search will suffice,\n\u003e but here are a few resources nonetheless:\n\u003e\n\u003e - Mac: https://gist.github.com/eparreno/1845561\n\u003e - Windows: https://github.com/nscaife/file-windows\n\u003e - Linux: https://zoomadmin.com/HowToInstall/UbuntuPackage/libmagic-dev\n\u003e\n\u003e **Definitely read `gen_magic`'s installation section in https://github.com/evadne/gen_magic#installation**.\n\u003e You may need to perform additional steps.\n\nYou'll need to add [`gen_magic`](https://github.com/evadne/gen_magic)\nto `mix.exs`.\nThis dependency will allow us to access `libmagic` through `Elixir`.\n\n```elixir\ndef deps do\n [\n {:gen_magic, \"~\u003e 1.1.1\"}\n ]\nend\n```\n\nIn the `Application` module, you should add the `GenMagic` daemon\n(the C lib is loaded once for all and referenced by its name).\n\n```elixir\n#application.ex\nchildren = [\n ...,\n {GenMagic.Server, name: :gen_magic},\n]\n```\n\nIn the Dockerfile (needed to deploy this app), we will install the `libmagic-dev` as well:\n\n```Dockerfile\nRUN apt-get update -y \u0026\u0026 \\\n apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates libmagic-dev\\\n \u0026\u0026 apt-get clean \u0026\u0026 rm -f /var/lib/apt/lists/*_*\n```\n\nAdd the following function in the module App.Image:\n\n```elixir\n @doc \"\"\"\n Check file type via magic number. It uses a GenServer running the `C` lib \"libmagic\".\n \"\"\"\n def gen_magic_eval(path, accepted_mime) do\n GenMagic.Server.perform(:gen_magic, path)\n |\u003e case do\n {:error, reason} -\u003e\n {:error, reason}\n\n {:ok,\n %GenMagic.Result{\n mime_type: mime,\n encoding: \"binary\",\n content: _content\n }} -\u003e\n if Enum.member?(accepted_mime, mime),\n do: {:ok, %{mime_type: mime}},\n else: {:error, \"Not accepted mime type.\"}\n\n {:ok, %GenMagic.Result{} = res} -\u003e\n require Logger\n Logger.warning(\"⚠️ MIME type error: #{inspect(res)}\")\n {:error, \"Not acceptable.\"}\n end\n end\nend\n```\n\nIn the `page_live.ex` module, add the functions:\n\n```elixir\n@doc\"\"\"\nUse the previous function and eturn the GenMagic reponse from the previous function\n\"\"\"\n\ndef magic_check(path) do\n App.Image.gen_magic_eval(path, @accepted_mime)\n |\u003e case do\n {:ok, %{mime_type: mime}} -\u003e\n {:ok, %{mime_type: mime}}\n\n {:error, msg} -\u003e\n {:error, msg}\n end\nend\n\n@doc \"\"\"\nDouble-checks the MIME type of uploaded file to ensure that the file\nis an image and is not corrupted.\n\"\"\"\ndef check_mime(magic_mime, info_mime) do\n if magic_mime == info_mime, do: :ok, else: :error\nend\n```\n\nWe are now ready to double-check the file input\nwith `ExImageInfo` and `GenMagic` to ensure the safety of the uploads.\n\n```elixir\ndef handle_progress(:image_list, entry, socket) when entry.done? do\n # We consume the entry only if the entry is done uploading from the image\n # and if consuming the entry was successful.\n\n with %{tensor: tensor, image_info: image_info} \u003c-\n consume_uploaded_entry(socket, entry, fn %{path: path} -\u003e\n with {:magic, {:ok, %{mime_type: mime}}} \u003c-\n {:magic, magic_check(path)},\n {:read, {:ok, file_binary}} \u003c-\n {:read, File.read(path)},\n {:image_info, {mimetype, width, height, _variant}} \u003c-\n {:image_info, ExImageInfo.info(file_binary)},\n {:check_mime, :ok} \u003c-\n {:check_mime, check_mime(mime, mimetype)},\n # Get image and resize\n {:ok, thumbnail_vimage} \u003c-\n Vix.Vips.Operation.thumbnail(path, @image_width, size: :VIPS_SIZE_DOWN),\n # Pre-process it\n {:ok, tensor} \u003c-\n pre_process_image(thumbnail_vimage) do\n # Upload image to S3\n Image.upload_image_to_s3(path, mimetype)\n |\u003e case do\n {:ok, url} -\u003e\n image_info = %ImageInfo{\n mimetype: mimetype,\n width: width,\n height: height,\n file_binary: file_binary,\n url: url\n }\n\n {:ok, %{tensor: tensor, image_info: image_info}}\n\n # If S3 upload fails, we return error\n {:error, reason} -\u003e\n {:ok, %{error: reason}}\n end\n else\n {:error, reason} -\u003e {:postpone, %{error: reason}}\n end\n end) do\n\n # If consuming the entry was successful, we spawn a task to classify the image\n # and update the socket assigns\n task =\n Task.Supervisor.async(App.TaskSupervisor, fn -\u003e\n Nx.Serving.batched_run(ImageClassifier, tensor)\n end)\n\n # Encode the image to base64\n base64 = \"data:image/png;base64, \" \u003c\u003e Base.encode64(image_info.file_binary)\n\n {:noreply,\n assign(socket,\n upload_running?: true,\n task_ref: task.ref,\n image_preview_base64: base64,\n image_info: image_info\n )}\n\n # Otherwise, if there was an error uploading the image, we log the error and show it to the person.\n else\n %{error: reason} -\u003e\n Logger.warning(\"⚠️ Error uploading image. #{inspect(reason)}\")\n {:noreply, push_event(socket, \"toast\", %{message: \"Image couldn't be uploaded to S3.\\n#{reason}\"})}\n\n _ -\u003e\n {:noreply, socket}\n end\nend\n```\n\nPhew! That's a lot!\nLet's go through the changes we've made.\n\n- we are using the [`with` statement](https://www.openmymind.net/Elixirs-With-Statement/)\n to only feed the image to the model for classification\n in case the upload to `imgup` succeeds.\n We've changed what `consume_uploaded_entry/3` returns\n in case the upload fails - we return `{:ok, %{error: reason}}`.\n- in case the upload fails,\n we pattern match the `{:ok, %{error: reason}}` object\n and push a `\"toast\"` event to the Javascript client\n (we'll implement these changes shortly).\n\nBecause we push an event in case the upload fails,\nwe are going to make some changes to the Javascript client.\nWe are going to **show a toast with the error when the upload fails**.\n\n#### 10.1 Showing a toast component with error\n\nTo show a [toast component](https://getbootstrap.com/docs/4.3/components/toasts/),\nwe are going to use\n[`toastify.js`](https://apvarun.github.io/toastify-js/).\n\nNavigate to `assets` folder\nand run:\n\n```sh\n pnpm install toastify-js\n```\n\nWith this installed, we need to import `toastify` styles\nin `assets/css/app.css`.\n\n```css\n@import \"../node_modules/toastify-js/src/toastify.css\";\n```\n\nAll that's left is **handle the `\"toast\"` event in `assets/js/app.js`**.\nAdd the following snippet of code to do so.\n\n```js\n// Hook to show message toast\nHooks.MessageToaster = {\n mounted() {\n this.handleEvent(\"toast\", (payload) =\u003e {\n Toastify({\n text: payload.message,\n gravity: \"bottom\",\n position: \"right\",\n style: {\n background: \"linear-gradient(to right, #f27474, #ed87b5)\",\n },\n duration: 4000,\n }).showToast();\n });\n },\n};\n```\n\nWith the `payload.message` we're receiving from the LiveView\n(remember when we executed `push_event/3` in our LiveView?),\nwe are using it to create a `Toastify` object\nthat is shown in case the upload fails.\n\nAnd that's it!\nQuite easy, isn't it? πŸ˜‰\n\nIf `imgup` is down or the image that was sent was for example, invalid, an error should be shown, like so.\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"800\" src=\"https://github.com/dwyl/image-classifier/assets/17494745/d730d10c-b45e-4dce-a37a-bb389c3cd548\" /\u003e\n\u003c/p\u003e\n\n### 11. Benchmarking image captioning models\n\nYou may be wondering: which model is most suitable for me?\nDepending on the use case,\n`Bumblebee` supports different models\nfor different scenarios.\n\nTo help you make up your mind,\nwe've created a guide\nthat benchmarks some of `Bumblebee`-supported models\nfor image captioning.\n\nAlthough few models are supported,\nas they add more models,\nthis comparison table will grow.\nSo any contribution is more than welcome! πŸŽ‰\n\nYou may check the guide\nand all of the code\ninside the\n[`_comparison`](./_comparison/) folder.\n\n\u003cdiv align=\"center\"\u003e\n\n## πŸ” Semantic search\n\n\u003e In this section, we will focus on implementing a\n\u003e **_full-text_ search query** through the captions of the images.\n\u003e At the end of this,\n\u003e you'll be able to transcribe audio,\n\u003e create embeddings from the audio transcription\n\u003e and search the closest related image.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/image-classifier/assets/17494745/b3568de8-2b0c-4413-8528-a3aee4135ea0\"\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n\u003e [!NOTE]\n\u003e\n\u003e This section was kindly implemented and documented by\n\u003e [@ndrean](https://github.com/ndrean). It is based on articles written by Sean Moriarty's published in the Dockyard's blog.\n\u003e Do check him out! πŸŽ‰\n\nWe can leverage machine learning to greatly improve this search process:\nwe'll look for images whose captions _are close in terms of meaning_\nto the search.\n\nIn this section, you'll learn how to perform\n[**semantic search**](https://www.elastic.co/what-is/semantic-search)\nwith machine learning.\nThese techniques are widely used in search engines,\nincluding in widespread tools like\n[Elastic Search](https://www.elastic.co/).\n\n### 0. Overview of the process\n\nLet's go over the process in detail so we know what to expect.\n\nAs it stands, when images are uploaded and captioned,\nthe URL is saved, as well as the caption,\nin our local database.\n\nHere's an overview of how semantic search usually works\n(which is what we'll exactly implement).\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"800\" src=\"https://github.com/dwyl/image-classifier/assets/17494745/90e3d370-b324-46fc-b1f6-83b6240f28a5\" /\u003e\n\u003c/p\u003e\n\n\u003e Source: https://www.elastic.co/what-is/semantic-search\n\nWe will use the following toolchain:\n\n\u003cp align=\"center\"\u003e\n \u003cimg width=\"800\" src=\"https://github.com/ndrean/image-classifier/assets/6793008/f5aad51b-2d49-4184-b5a4-07236449c821\" /\u003e\n\u003c/p\u003e\n\n#### 0.1 Audio transcription\n\nWe simply let the user start and stop the recording\nby using a submit button in a form.\nThis can of course be greatly refined by using **Voice Detection**. You may find an example [here](https://github.com/ricky0123/vad).\n\nFirstly, we will:\n\n- record an audio with [MediaRecorder](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder) API.\n- run a **Speech-To-Text** process to produce a text transcription\n from the audio.\n\nWe will use the _pre-trained_ model [openai/whisper-small](https://huggingface.co/openai/whisper-small)\nfrom \u003chttps://huggingface.co\u003e\nand use it with the help of the [Bumblebee.Audio.speech_to_text_whisper](https://hexdocs.pm/bumblebee/Bumblebee.Audio.html#speech_to_text_whisper/5) function.\nWe get an `Nx.Serving` that we will use to run this model with an input.\n\n#### 0.2 Creating embeddings\n\nWe then want to find images whose captions\napproximate this text in terms of meaning.\nThis transcription is the `\"target text\"`.\nThis is where **embeddings** come into play:\nthey are **vector representations** of certain inputs,\nwhich in our case, are the text transcription of the audio file recorded by the user.\nWe encode each transcription as an embedding\nand then use an approximation algorithm to find the closest neighbours.\n\nOur next steps will be to prepare the\n[symmetric semantic search](https://www.sbert.net/examples/applications/semantic-search/README.html#symmetric-vs-asymmetric-semantic-search).\nWe will use a\n[transformer](\u003chttps://en.wikipedia.org/wiki/Transformer_(machine_learning_model)\u003e) model,\nmore specifically the pre-trained [sBert](https://www.sbert.net/docs/pretrained_models.html#sentence-embedding-models)\nsystem available in\n[Huggingface](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2).\n\nWe transform a text into a vector with the sentence-transformer model [`sentence-transformers/paraphrase-MiniLM-L6-v2` ](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2).\n\n\u003e [!NOTE]\n\u003e You may find models in the [MTEB English leaderboard](https://huggingface.co/spaces/mteb/leaderboard). We looked for \"small\" models in terms of file size and dimensions. You may want to try and use [GTE small](https://huggingface.co/thenlper/gte-small).\n\nWe will run the model with the help of the\n[Bumblebee.Text.TextEmbedding.text_embedding](https://hexdocs.pm/bumblebee/Bumblebee.Text.html#text_embedding/3) function.\n\nThis encoding is done for each image caption.\n\n#### 0.3 Semantical search\n\nAt this point, we have:\n\n- the embedding of the text transcription of the recording made by the user\n (e.g `\"a dog\"`).\n- all the embeddings of all the images in our \"image bank\".\n\nTo search for the images that are related to `\"a dog\"`,\nwe need to apply an algorithm that compares these embeddings!\n\nFor this, we will run a [**knn_neighbour**](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) search.\nThere are several ways to do this.\n\n- we can use`pgvector` , a vector extension of Postgres. It is used to store vectors (the embeddings) and to run similarity searches.\n With `pgvector`, we can run:\n\n - a full exact search with the [cosine similarity](https://github.com/pgvector/pgvector#distances) operator `\u003c=\u003e`,\n - or use an Approximate Nearest Neighbour seach with indexing algorithms. The extension proposes the [`IVFFLAT`](https://github.com/pgvector/pgvector#ivfflat) and the [`HNSWLIB`](https://github.com/pgvector/pgvector#hnsw) algorithms. You can find some explanations on both algorithms in https://tembo.io/blog/vector-indexes-in-pgvector and https://neon.tech/blog/understanding-vector-search-and-hnsw-index-with-pgvector.\n\n\u003e [!NOTE]\n\u003e Note that [Supabase](https://supabase.com/docs/guides/database/extensions/pgvector) can use the `pgvector` extension, and you can use [Supabase with Fly.io](https://fly.io/docs/reference/supabase/).\n\n\u003e [!WARNING]\n\u003e Note that you need to save the embeddings (as vectors) into the database, so the database will be intensively used. This may lead to scaling problems and potential race conditions.\n\n- we can alternatively use the `hnswlib` library and its Elixir binding [HNSWLib](https://github.com/elixir-nx/hnswlib).\n This \"externalises\" the ANN search from the database as it uses an in-memory file.\n This file needs to be persisted on disk, thus at the expense of using the filesystem with again potential race conditions.\n It works with an **[index struct](https://www.datastax.com/guides/what-is-a-vector-index)**: this struct will allow us to efficiently retrieve vector data.\n\n**We will use this last option**,\nmostly because we use Fly.io\nand `pgvector` is hard to come by on this platform.\nWe will use a GenServer to wrap all the calls to `hnswlib` so every writes will be run synchronously.\nAdditionally, you don't rely on a framework that does the heavy lifting for you.\nWe're here to learn, aren't we? πŸ˜ƒ\n\nWe will append incrementally the computed embedding from the captions into the Index.\nWe will get an indice which simply is the order of this embedding in the Index.\nWe then run a \"knn_search\" algorithm; the input will be the embedding of the audio transcript.\nThis algorithm will return the most relevant position(s) - `indices` -\namong the `Index` indices that minimize the chosen distance between this input and the existing vectors.\n\nThis is where we'll need to save:\n\n- whether the index,\n- or the embedding\n\nto look up for the corresponding image(s), depending upon if you append items one by one or by batch.\n\nIn our case, you will append items one by one so we will use the index to uniquely recover the nearest image whose caption is close semantically to our audio.\n\nDo note that the measured distance is dependent on the [similarity metric](https://www.pinecone.io/learn/vector-similarity/)\nused by the embedding model.\nBecause the \"sentence-transformer\" model we've chosen was trained with **_cosine_similarity_**,\nthis is what we'll use.\nBumblebee may have options to correctly use this metric, but we used a normalisation process which fits our needs.\n\n### 1. Pre-requisites\n\nWe have already installed all the dependencies that we need.\n\n\u003e [!WARNING] \u003e **You will also need to install [`ffmpeg`](https://ffmpeg.org/)**.\n\u003e Indeed, `Bumblebee` uses `ffmpeg` under the hood to process audio files into tensors,\n\u003e but it uses it _as an external dependency_.\n\nAnd now we're ready to rock and roll! 🎸\n\n### 2. Transcribe an audio recording\n\n\u003e **Source:** \u003chttps://dockyard.com/blog/2023/03/07/audio-speech-recognition-in-elixir-with-whisper-bumblebee?utm_source=elixir-merge\u003e\n\nWe first need to capture the audio and upload it to the server.\n\nThe process is quite similar to the image upload, except that we\nuse a special Javascript hook to record the audio\nand upload it to the Phoenix LiveView.\n\nWe use a `live_file_input` in a form to capture the audio and use the Javascript `MediaRecorder API`.\nThe Javascript code is triggered by an attached hook `Audio` declared in the HTML.\nWe also let the user listen to his audio by adding an [embedded audio element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/audio) `\u003caudio\u003e` in the HTML.\nIts source is the audio blob as a URL object.\n\n#### 1.1 Adding a loading spinner\n\nWe also add a spinner to display that the transcription process is running,\nin the same way as we did for the captioning process.\nTo avoid code duplication, we introduce a Phoenix component \"Spinner\".\nCreate the file\n`spinner.ex` in `lib/app_web/components/`\nand create the `Spinner` component,\nlike so:\n\n```elixir\n# /lib/app_web/components/spinner.ex\ndefmodule AppWeb.Spinner do\n use Phoenix.Component\n\n attr :spin, :boolean, default: false\n\n def spin(assigns) do\n ~H\"\"\"\n \u003cdiv :if={@spin} role=\"status\"\u003e\n \u003cdiv class=\"relative w-6 h-6 animate-spin rounded-full bg-gradient-to-r from-purple-400 via-blue-500 to-red-400 \"\u003e\n \u003cdiv class=\"absolute top-1/2 left-1/2 transform -translate-x-1/2 -translate-y-1/2 w-3 h-3 bg-gray-200 rounded-full border-2 border-white\"\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \"\"\"\n end\nend\n```\n\nIn `page_live_html.heex`, add the following snippet of code.\n\n```html\n# page_live.html.heex\n\u003cform phx-change=\"noop\"\u003e\n \u003c.live_file_input upload={@uploads.speech} class=\"hidden\" /\u003e\n \u003cbutton\n id=\"record\"\n class=\"bg-blue-500 hover:bg-blue-700 text-white font-bold px-4 rounded\"\n type=\"button\"\n phx-hook=\"Audio\"\n disabled=\"{@mic_off?}\"\n \u003e\n \u003cHeroicons.microphone\n outline\n class=\"w-6 h-6 text-white font-bold group-active:animate-pulse\"\n /\u003e\n \u003cspan id=\"text\"\u003eRecord\u003c/span\u003e\n \u003c/button\u003e\n\u003c/form\u003e\n\u003cp class=\"flex flex-col items-center\"\u003e\n \u003caudio id=\"audio\" controls\u003e\u003c/audio\u003e\n \u003cAppWeb.Spinner.spin spin=\"{@audio_running?}\" /\u003e\n\u003c/p\u003e\n```\n\nYou can also use this component to display the spinner when the captioning task is running,\nso this part of your code will shrink to:\n\n```elixir\n\u003c!-- Spinner --\u003e\n\u003c%= if @upload_running? do %\u003e\n \u003cAppWeb.Spinner.spin spin={@upload_running?} /\u003e\n\u003c% else %\u003e\n \u003c%= if @label do %\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @label %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light text-justify\"\u003eWaiting for image input.\u003c/span\u003e\n \u003c% end %\u003e\n\u003c% end %\u003e\n```\n\n#### 2.2 Defining `Javascript` hook\n\nCurrently, we provide a basic user experience:\nwe let the user click on a button to start and stop the recording.\n\nWhen doing image captioning,\nwe carefully worked on the size of the image file\nused for the captioning model\nto optimize the app's latency.\n\nIn the same spirit,\nwe can downsize the original audio file\nso it's easier for the model to process it.\nThis will have the benefit of less overhead in our application.\n[Even though the `whisper` model does downsize the audio's sampling rate](https://github.com/openai/whisper/discussions/870),\nwe can do this on our client side to skip this step\nand marginally reduce the overhead of running the model in our application.\n\nThe main parameters we're dealing with are:\n\n- **lower sampling rate** (the higher the more accurate is the sound),\n- **use mono** instead of stereo,\n- and **the file type** (WAV, MP3)\n\nSince most microphones on PC have a single channel (mono) and sample at `48kHz`,\nwe will focus on resampling to `16kHz`.\nWe will *not* make the conversion to mp3 here.\n\nNext, we define the hook in a new JS file, located in the `assets/js` folder.\n\nTo resample to `16kHz`, we use an [AudioContext](https://developer.mozilla.org/en-US/docs/Web/API/AudioContext),\nand pass the desired `sampleRate`.\nWe then use the method [decodeAudioData](https://developer.mozilla.org/en-US/docs/Web/API/BaseAudioContext/decodeAudioData)\nwhich receives an [AudioBuffer](https://developer.mozilla.org/en-US/docs/Web/API/AudioBuffer).\nWe get one from the `Blob` method [arrayBuffer()](https://developer.mozilla.org/en-US/docs/Web/API/Blob/arrayBuffer).\n\nThe important part is the `Phoenix.js` function `upload`,\nto which we pass an identifier `\"speech\"`:\nthis sends the data as `Blob` via a channel to the server.\n\nWe use an action button in the HTML,\nand attach Javascript listeners to it on the `\"click\"`, `\"dataavailable\"` and `\"stop\"` events.\nWe also play with the CSS classes to modify the appearance of the action button when recording or not.\n\nNavigate to the `assets` folder and run the following command.\nWe will use this to lower the sampling rate\nand conver the recorded audio file.\n\n```bash\nnpm add \"audiobuffer-to-wav\"\n```\n\nCreate a file called `assets/js/micro.js`\nand use the code below.\n\n```js\n// /assets/js/micro.js\nimport toWav from \"audiobuffer-to-wav\";\n\nexport default {\n mounted() {\n let mediaRecorder,\n audioChunks = [];\n\n // Defining the elements and styles to be used during recording\n // and shown on the HTML.\n const recordButton = document.getElementById(\"record\"),\n audioElement = document.getElementById(\"audio\"),\n text = document.getElementById(\"text\"),\n blue = [\"bg-blue-500\", \"hover:bg-blue-700\"],\n pulseGreen = [\"bg-green-500\", \"hover:bg-green-700\", \"animate-pulse\"];\n\n _this = this;\n\n // Adding event listener for \"click\" event\n recordButton.addEventListener(\"click\", () =\u003e {\n // Check if it's recording.\n // If it is, we stop the record and update the elements.\n if (mediaRecorder \u0026\u0026 mediaRecorder.state === \"recording\") {\n mediaRecorder.stop();\n text.textContent = \"Record\";\n }\n\n // Otherwise, it means the user wants to start recording.\n else {\n navigator.mediaDevices.getUserMedia({ audio: true }).then((stream) =\u003e {\n // Instantiate MediaRecorder\n mediaRecorder = new MediaRecorder(stream);\n mediaRecorder.start();\n\n // And update the elements\n recordButton.classList.remove(...blue);\n recordButton.classList.add(...pulseGreen);\n text.textContent = \"Stop\";\n\n // Add \"dataavailable\" event handler\n mediaRecorder.addEventListener(\"dataavailable\", (event) =\u003e {\n audioChunks.push(event.data);\n });\n\n // Add \"stop\" event handler for when the recording stops.\n mediaRecorder.addEventListener(\"stop\", () =\u003e {\n const audioBlob = new Blob(audioChunks);\n // update the source of the Audio tag for the user to listen to his audio\n audioElement.src = URL.createObjectURL(audioBlob);\n\n // create an AudioContext with a sampleRate of 16000\n const audioContext = new AudioContext({ sampleRate: 16000 });\n\n // async read the Blob as ArrayBuffer to feed the \"decodeAudioData\"\n const arrayBuffer = await audioBlob.arrayBuffer();\n // decodes the ArrayBuffer into the AudioContext format\n const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);\n // converts the AudioBuffer into a WAV format\n const wavBuffer = toWav(audioBuffer);\n // builds a Blob to pass to the Phoenix.JS.upload\n const wavBlob = new Blob([wavBuffer], { type: \"audio/wav\" });\n // upload to the server via a chanel with the built-in Phoenix.JS.upload\n _this.upload(\"speech\", [wavBlob]);\n // close the MediaRecorder instance\n mediaRecorder.stop();\n // cleanups\n audioChunks = [];\n recordButton.classList.remove(...pulseGreen);\n recordButton.classList.add(...blue);\n });\n });\n }\n });\n },\n};\n```\n\nNow let's import this file and declare our `hook` object in our `livesocket` object.\nIn our `assets/js/app.js` file, let's do:\n\n```js\n// /assets/js/app.js\n...\nimport Audio from \"./micro.js\";\n...\nlet liveSocket = new LiveSocket(\"/live\", Socket, {\n params: { _csrf_token: csrfToken },\n hooks: { Audio },\n});\n```\n\n#### 2.3 Handling audio upload in `LiveView`\n\nWe now need to add some server-side code.\n\nThe uploaded audio file will be saved on disk\nas a temporary file in the `/priv/static/uploads` folder.\nWe will also make this file _unique_ every time a user records an audio.\nWe use an `Ecto.UUID` string to the file name and pass it into the Liveview socket.\n\nThe Liveview `mount/3` function returns a socket.\nLet's update it\nand pass extra arguments -\ntypically booleans for the UI such as the button disabling and the spinner -\nas well as another `allow_upload/3` to handle the upload process of the audio file.\n\nIn `lib/app_web/live/page_live.ex`,\nwe change the code like so:\n\n```elixir\n#page_live.ex\n@upload_dir Application.app_dir(:app, [\"priv\", \"static\", \"uploads\"])\n@tmp_wav Path.expand(\"priv/static/uploads/tmp.wav\")\n\ndef mount(_,_,socket) do\n socket\n |\u003e assign(\n ...,\n transcription: nil,\n mic_off?: false,\n audio_running?: false,\n tmp_wav: @tmp_wav\n )\n |\u003e allow_upload(:speech,\n accept: :any,\n auto_upload: true,\n progress: \u0026handle_progress/3,\n max_entries: 1\n )\n |\u003e allow_upload(:image_list, ...)\nend\n```\n\nWe then create a specific `handle_progress` for the `:speech` event\nas we did with the `:image_list` event.\nIt will launch a task to run the **Automatic Speech Recognition model**\non this audio file.\nWe named the serving `\"Whisper\"`.\n\n```elixir\ndef handle_progress(:speech, entry, %{assigns: assigns} = socket) when entry.done? do\n tmp_wav =\n socket\n |\u003e consume_uploaded_entry(entry, fn %{path: path} -\u003e\n tmp_wav = assigns.tmp_wav \u003c\u003e Ecto.UUID.generate() \u003c\u003e \".wav\"\n :ok = File.cp!(path, tmp_wav)\n {:ok, tmp_wav}\n end)\n\n audio_task =\n Task.Supervisor.async(\n App.TaskSupervisor,\n fn -\u003e\n Nx.Serving.batched_run(Whisper, {:file, @tmp_wav})\n end\n )\n\n {:noreply, socket\n |\u003e assign(\n audio_ref: audio_task.ref,\n mic_off?: true,\n audio_running?: true,\n tmp_wav: tmp_wav,\n )}\nend\n```\n\nAnd that's it for the Liveview portion!\n\n#### 2.4 Serving the `Whisper` model\n\nNow that we are adding several models,\nlet's refactor our `models.ex` module\nthat manages the models.\nSince we're dealing with multiple models,\nwe want our app to shutdown if there's any problem loading them.\n\nWe now add the model `Whisper` in the\n`lib/app/application.ex`\nso it's available throughout the application on runtime.\n\n```elixir\n# lib/app/application.ex\n\ndef check_models_on_startup do\n App.Models.verify_and_download_models()\n |\u003e case do\n {:error, msg} -\u003e\n Logger.error(\"⚠️ #{msg}\")\n System.stop(0)\n\n :ok -\u003e\n :ok\n end\nend\n\ndef start(_type, _args) do\n end\n\n # model check-up\n :ok = check_models_on_startup()\n\n children = [\n ...,\n # Nx serving for Speech-to-Text\n {Nx.Serving,\n serving:\n if Application.get_env(:app, :use_test_models) == true do\n App.Models.audio_serving_test()\n else\n App.Models.audio_serving()\n end,\n name: Whisper},\n ,...\n ]\n ...\n```\n\nAs you can see, we're using a serving similar\nto the captioning model we've implemented earlier.\nFor this to work, we need to make some changes to the\n`models.ex` module.\nRecall that this module simply manages the models that are\ndownloaded locally and used in our application.\n\nTo implement the functions above,\nwe change the `lib/app/models.ex` module so it looks like so.\n\n```elixir\ndefmodule ModelInfo do\n @moduledoc \"\"\"\n Information regarding the model being loaded.\n It holds the name of the model repository and the directory it will be saved into.\n It also has booleans to load each model parameter at will - this is because some models (like BLIP) require featurizer, tokenizations and generation configuration.\n \"\"\"\n\n defstruct [:name, :cache_path, :load_featurizer, :load_tokenizer, :load_generation_config]\nend\n\ndefmodule App.Models do\n @moduledoc \"\"\"\n Manages loading the modules and their location according to env.\n \"\"\"\n require Logger\n\n # IMPORTANT: This should be the same directory as defined in the `Dockerfile`\n # where the models will be downloaded into.\n @models_folder_path Application.compile_env!(:app, :models_cache_dir)\n\n # Embedding-------\n @embedding_model %ModelInfo{\n name: \"sentence-transformers/paraphrase-MiniLM-L6-v2\",\n cache_path: Path.join(@models_folder_path, \"paraphrase-MiniLM-L6-v2\"),\n load_featurizer: false,\n load_tokenizer: true,\n load_generation_config: true\n }\n # Captioning --\n @captioning_test_model %ModelInfo{\n name: \"microsoft/resnet-50\",\n cache_path: Path.join(@models_folder_path, \"resnet-50\"),\n load_featurizer: true\n }\n\n @captioning_prod_model %ModelInfo{\n name: \"Salesforce/blip-image-captioning-base\",\n cache_path: Path.join(@models_folder_path, \"blip-image-captioning-base\"),\n load_featurizer: true,\n load_tokenizer: true,\n load_generation_config: true\n }\n\n # Audio transcription --\n @audio_test_model %ModelInfo{\n name: \"openai/whisper-small\",\n cache_path: Path.join(@models_folder_path, \"whisper-small\"),\n load_featurizer: true,\n load_tokenizer: true,\n load_generation_config: true\n }\n\n @audio_prod_model %ModelInfo{\n name: \"openai/whisper-small\",\n cache_path: Path.join(@models_folder_path, \"whisper-small\"),\n load_featurizer: true,\n load_tokenizer: true,\n load_generation_config: true\n }\n\n def extract_captioning_test_label(result) do\n %{predictions: [%{label: label}]} = result\n label\n end\n\n def extract_captioning_prod_label(result) do\n %{results: [%{text: label}]} = result\n label\n end\n\n @doc \"\"\"\n Verifies and downloads the models according to configuration\n and if they are already cached locally or not.\n\n The models that are downloaded are hardcoded in this function.\n \"\"\"\n def verify_and_download_models() do\n {\n Application.get_env(:app, :force_models_download, false),\n Application.get_env(:app, :use_test_models, false)\n }\n |\u003e case do\n {true, true} -\u003e\n # Delete any cached pre-existing models\n File.rm_rf!(@models_folder_path)\n\n with :ok \u003c- download_model(@captioning_test_model),\n :ok \u003c- download_model(@embedding_model),\n :ok \u003c- download_model(@audio_test_model) do\n :ok\n else\n {:error, msg} -\u003e {:error, msg}\n end\n\n {true, false} -\u003e\n # Delete any cached pre-existing models\n File.rm_rf!(@models_folder_path)\n\n with :ok \u003c- download_model(@captioning_prod_model),\n :ok \u003c- download_model(@audio_prod_model),\n :ok \u003c- download_model(@embedding_model) do\n :ok\n else\n {:error, msg} -\u003e {:error, msg}\n end\n\n {false, false} -\u003e\n # Check if the prod model cache directory exists or if it's not empty.\n # If so, we download the prod models.\n\n with :ok \u003c- check_folder_and_download(@captioning_prod_model),\n :ok \u003c- check_folder_and_download(@audio_prod_model),\n :ok \u003c- check_folder_and_download(@embedding_model) do\n :ok\n else\n {:error, msg} -\u003e {:error, msg}\n end\n\n {false, true} -\u003e\n # Check if the test model cache directory exists or if it's not empty.\n # If so, we download the test models.\n\n with :ok \u003c- check_folder_and_download(@captioning_test_model),\n :ok \u003c- check_folder_and_download(@audio_test_model),\n :ok \u003c- check_folder_and_download(@embedding_model) do\n :ok\n else\n {:error, msg} -\u003e {:error, msg}\n end\n end\n end\n\n @doc \"\"\"\n Serving function that serves the `Bumblebee` captioning model used throughout the app.\n This function is meant to be called and served by `Nx` in `lib/app/application.ex`.\n\n This assumes the models that are being used exist locally, in the @models_folder_path.\n \"\"\"\n def caption_serving do\n load_offline_model(@captioning_prod_model)\n |\u003e then(fn response -\u003e\n case response do\n {:ok, model} -\u003e\n %Nx.Serving{} =\n Bumblebee.Vision.image_to_text(\n model.model_info,\n model.featurizer,\n model.tokenizer,\n model.generation_config,\n compile: [batch_size: 1],\n defn_options: [compiler: EXLA],\n # needed to run on `Fly.io`\n preallocate_params: true\n )\n\n {:error, msg} -\u003e\n {:error, msg}\n end\n end)\n end\n\n @doc \"\"\"\n Serving function that serves the `Bumblebee` audio transcription model used throughout the app.\n \"\"\"\n def audio_serving do\n load_offline_model(@audio_prod_model)\n |\u003e then(fn response -\u003e\n case response do\n {:ok, model} -\u003e\n %Nx.Serving{} =\n Bumblebee.Audio.speech_to_text_whisper(\n model.model_info,\n model.featurizer,\n model.tokenizer,\n model.generation_config,\n chunk_num_seconds: 30,\n task: :transcribe,\n defn_options: [compiler: EXLA],\n preallocate_params: true\n )\n\n {:error, msg} -\u003e\n {:error, msg}\n end\n end)\n end\n\n @doc \"\"\"\n Serving function for tests only. It uses a test audio transcription model.\n \"\"\"\n def audio_serving_test do\n load_offline_model(@audio_test_model)\n |\u003e then(fn response -\u003e\n case response do\n {:ok, model} -\u003e\n %Nx.Serving{} =\n Bumblebee.Audio.speech_to_text_whisper(\n model.model_info,\n model.featurizer,\n model.tokenizer,\n model.generation_config,\n chunk_num_seconds: 30,\n task: :transcribe,\n defn_options: [compiler: EXLA],\n preallocate_params: true\n )\n\n {:error, msg} -\u003e\n {:error, msg}\n end\n end)\n end\n\n @doc \"\"\"\n Serving function for tests only. It uses a test captioning model.\n This function is meant to be called and served by `Nx` in `lib/app/application.ex`.\n\n This assumes the models that are being used exist locally, in the @models_folder_path.\n \"\"\"\n def caption_serving_test do\n load_offline_model(@captioning_test_model)\n |\u003e then(fn response -\u003e\n case response do\n {:ok, model} -\u003e\n %Nx.Serving{} =\n Bumblebee.Vision.image_classification(\n model.model_info,\n model.featurizer,\n top_k: 1,\n compile: [batch_size: 10],\n defn_options: [compiler: EXLA],\n # needed to run on `Fly.io`\n preallocate_params: true\n )\n\n {:error, msg} -\u003e\n {:error, msg}\n end\n end)\n end\n\n # Loads the models from the cache folder.\n # It will load the model and the respective the featurizer, tokenizer and generation config if needed,\n # and return a map with all of these at the end.\n @spec load_offline_model(map()) ::\n {:ok, map()} | {:error, String.t()}\n\n defp load_offline_model(model) do\n Logger.info(\"ℹ️ Loading #{model.name}...\")\n\n # Loading model\n loading_settings = {:hf, model.name, cache_dir: model.cache_path, offline: true}\n\n Bumblebee.load_model(loading_settings)\n |\u003e case do\n {:ok, model_info} -\u003e\n info = %{model_info: model_info}\n\n # Load featurizer, tokenizer and generation config if needed\n info =\n if Map.get(model, :load_featurizer) do\n {:ok, featurizer} = Bumblebee.load_featurizer(loading_settings)\n Map.put(info, :featurizer, featurizer)\n else\n info\n end\n\n info =\n if Map.get(model, :load_tokenizer) do\n {:ok, tokenizer} = Bumblebee.load_tokenizer(loading_settings)\n Map.put(info, :tokenizer, tokenizer)\n else\n info\n end\n\n info =\n if Map.get(model, :load_generation_config) do\n {:ok, generation_config} =\n Bumblebee.load_generation_config(loading_settings)\n\n Map.put(info, :generation_config, generation_config)\n else\n info\n end\n\n # Return a map with the model and respective parameters.\n {:ok, info}\n\n {:error, msg} -\u003e\n {:error, msg}\n end\n end\n\n # Downloads the pre-trained models according to a given %ModelInfo struct.\n # It will load the model and the respective the featurizer, tokenizer and generation config if needed.\n @spec download_model(map()) :: {:ok, map()} | {:error, binary()}\n defp download_model(model) do\n Logger.info(\"ℹ️ Downloading #{model.name}...\")\n\n # Download model\n downloading_settings = {:hf, model.name, cache_dir: model.cache_path}\n\n # Download featurizer, tokenizer and generation config if needed\n Bumblebee.load_model(downloading_settings)\n |\u003e case do\n {:ok, _} -\u003e\n if Map.get(model, :load_featurizer) do\n {:ok, _} = Bumblebee.load_featurizer(downloading_settings)\n end\n\n if Map.get(model, :load_tokenizer) do\n {:ok, _} = Bumblebee.load_tokenizer(downloading_settings)\n end\n\n if Map.get(model, :load_generation_config) do\n {:ok, _} = Bumblebee.load_generation_config(downloading_settings)\n end\n\n :ok\n\n {:error, msg} -\u003e\n {:error, msg}\n end\n end\n\n # Checks if the folder exists and downloads the model if it doesn't.\n def check_folder_and_download(model) do\n :ok = File.mkdir_p!(@models_folder_path)\n\n model_location =\n Path.join(model.cache_path, \"huggingface\")\n\n if File.ls(model_location) == {:error, :enoent} or File.ls(model_location) == {:ok, []} do\n download_model(model)\n |\u003e case do\n :ok -\u003e :ok\n {:error, msg} -\u003e {:error, msg}\n end\n else\n Logger.info(\"ℹ️ No download needed: #{model.name}\")\n :ok\n end\n end\nend\n```\n\nThat's a lot! But we just need to focus on some new parts we've added:\n\n- we've created **`audio_serving_test/1`** and\n **`audio_serving/1`**, our audio serving functions\n that are used in the `application.ex` file.\n- added `@audio_prod_model` and `@audio_test_model`,\n the `Whisper` model definitions to be used to download the models locally.\n- refactored the image captioning model definitions to be more clear.\n\nNow we're successfully serving audio-to-text capabilities\nin our application!\n\n#### 2.5 Handling the model's response and updating elements in the view\n\nWe expect the response of this task to be\nin the following form:\n\n```elixir\n%{\n chunks:\n [%{\n text: \"Hi there\",\n #^^^the text of our audio\n start_timestamp_seconds: nil,\n end_timestamp_seconds: nil\n }]\n}\n```\n\nWe capture this response in a `handle_info` callback\nwhere we simply prune the temporary audio file\nand update the socket state with the result,\nand update the booleans used for our UI\n(the spinner element, the button availability and reset of the task once done).\n\n```elixir\ndef handle_info({ref, %{chunks: [%{text: text}]} = _result}, %{assigns: assigns} = socket)\n when assigns.audio_ref == ref do\n Process.demonitor(ref, [:flush])\n File.rm!(assigns.tmp_wav)\n\n {:noreply,\n assign(socket,\n transcription: String.trim(text),\n mic_off?: false,\n audio_running?: false,\n audio_ref: nil,\n tmp_wav: @tmp_wav\n )}\nend\n```\n\nAnd that's it for this section!\nOur application is now able to **record audio**\nand **transcribe it**. πŸŽ‰\n\n### 3. Embeddings and semantic search\n\nWe want to encode every caption and the input text\ninto an embedding which is a vector of a specific vector space.\nIn other words, we encode a string into a list of numbers.\n\nWe chose the transformer `\"sentence-transformers/paraphrase-MiniLM-L6-v2\"` model.\n\nThis transformer uses a **`384`** dimensional vector space.\nSince this transformer is trained with a `cosine metric`,\nwe embed the vector space of embeddings with the same distance.\nYou can read more about [cosine_similarity here](https://en.wikipedia.org/wiki/Cosine_similarity).\n\nThis model is loaded and served by an `Nx.Serving` started in the Application module like all other models.\n\n#### 3.1 The `HNSWLib` Index (GenServer)\n\nThis library [`HNSWLib`](https://github.com/elixir-nx/hnswlib)\nworks with an **[index](https://www.datastax.com/guides/what-is-a-vector-index)**.\nWe instantiate the Index file in a `GenServer` which holds the index in the state.\n\nWe will use an Index file that is saved locally in our file system.\nThis file will be updated any time we append an embedding;\nall the client calls and writes to the HNSWLib index are handled by the GenServer.\nThey will happen synchronously. We want to minimize the race conditions in case several users interact with the app.\nThis app is only meant to run **on a single node**.\n\nIt is started in the Application module (`application.ex`).\nWhen the app starts, we either read or create this file. The file is saved in the \"/priv/static/uploads\" folder.\n\nBecause we are deploying with Fly.io, we need to persist the Index file in the database because the machine - thus its attached volume - is pruned when inactive.\n\nIt is crucial to save the correspondence between the `Image` table and the Index file to retrieve the correct images.\nIn simple terms, **the file in the `Index` table in the DB must correspond to the Index file in the system.**\n\nWe therefore disable a user from loading several times the same file as otherwise,\nwe would have several indexes for the same picture.\nThis is done through **SHA computation**.\n\nSince computations using models is a long-run process,\nand because several users may interact with the app,\nwe need several steps to ensure that the information is synchronized between the database and the index file.\n\nWe also endow the vector space with a `:cosine` pseudo-metric.\n\nAdd the following `GenServer` file:\nit will load the Index file,\nand also provide a client API to interact with the Index,\nwhich is held in the state of the GenServer.\n\nAgain, this solution works for a single node _only_.\n\n```elixir\ndefmodule App.KnnIndex do\n use GenServer\n\n @moduledoc \"\"\"\n A GenServer to load and handle the Index file for HNSWLib.\n It loads the index from the FileSystem if existing or from the table HnswlibIndex.\n It creates an new one if no Index file is found in the FileSystem\n and if the table HnswlibIndex is empty.\n It holds the index and the App.Image singleton table in the state.\n \"\"\"\n\n require Logger\n\n @dim 384\n @max_elements 200\n @upload_dir Application.app_dir(:app, [\"priv\", \"static\", \"uploads\"])\n @saved_index if Application.compile_env(:app, :knnindex_indices_test, false),\n do: Path.join(@upload_dir, \"indexes_test.bin\"),\n else: Path.join(@upload_dir, \"indexes.bin\")\n\n # Client API ------------------\n def start_link(args) do\n :ok = File.mkdir_p!(@upload_dir)\n GenServer.start_link(__MODULE__, args, name: __MODULE__)\n end\n\n def index_path do\n @saved_index\n end\n\n def save_index_to_db do\n GenServer.call(__MODULE__, :save_index_to_db)\n end\n\n def get_count do\n GenServer.call(__MODULE__, :get_count)\n end\n\n def add_item(embedding) do\n GenServer.call(__MODULE__, {:add_item, embedding})\n end\n\n def knn_search(input) do\n GenServer.call(__MODULE__, {:knn_search, input})\n end\n\n def not_empty_index do\n GenServer.call(__MODULE__, :not_empty)\n end\n\n # ---------------------------------------------------\n @impl true\n def init(args) do\n # Trying to load the index file\n index_path = Keyword.fetch!(args, :index)\n space = Keyword.fetch!(args, :space)\n\n case File.exists?(index_path) do\n # If the index file doesn't exist, we try to load from the database.\n false -\u003e\n {:ok, index, index_schema} =\n App.HnswlibIndex.maybe_load_index_from_db(space, @dim, @max_elements)\n\n {:ok, {index, index_schema, space}}\n\n # If the index file does exist, we compare the one with teh table and check for incoherences.\n true -\u003e\n Logger.info(\"ℹ️ Index file found on disk. Let's compare it with the database...\")\n\n App.Repo.get_by(App.HnswlibIndex, id: 1)\n |\u003e case do\n nil -\u003e\n {:stop,\n {:error,\n \"Error comparing the index file with the one on the database. Incoherence on table.\"}}\n\n schema -\u003e\n check_integrity(index_path, schema, space)\n end\n end\n end\n\n defp check_integrity(path, schema, space) do\n # We check the count of the images in the database and the one in the index.\n with db_count \u003c-\n App.Repo.all(App.Image) |\u003e length(),\n {:ok, index} \u003c-\n HNSWLib.Index.load_index(space, @dim, path),\n {:ok, index_count} \u003c-\n HNSWLib.Index.get_current_count(index),\n true \u003c-\n index_count == db_count do\n Logger.info(\"ℹ️ Integrity: βœ…\")\n {:ok, {index, schema, space}}\n\n # If it fails, we return an error.\n else\n false -\u003e\n {:stop,\n {:error, \"Integrity error. The count of images from index differs from the database.\"}}\n\n {:error, msg} -\u003e\n Logger.error(\"⚠️ #{msg}\")\n {:stop, {:error, msg}}\n end\n end\n\n @impl true\n def handle_call(:save_index_to_db, _, {index, index_schema, space} = state) do\n # We read the index file and try to update the index on the table as well.\n File.read(@saved_index)\n |\u003e case do\n {:ok, file} -\u003e\n {:ok, updated_schema} =\n index_schema\n |\u003e App.HnswlibIndex.changeset(%{file: file})\n |\u003e App.Repo.update()\n\n {:reply, {:ok, updated_schema}, {index, updated_schema, space}}\n\n {:error, msg} -\u003e\n {:reply, {:error, msg}, state}\n end\n end\n\n def handle_call(:get_count, _, {index, _, _} = state) do\n {:ok, count} = HNSWLib.Index.get_current_count(index)\n {:reply, count, state}\n end\n\n def handle_call({:add_item, embedding}, _, {index, _, _} = state) do\n # We add the new item to the index and update it.\n with :ok \u003c-\n HNSWLib.Index.add_items(index, embedding),\n {:ok, idx} \u003c-\n HNSWLib.Index.get_current_count(index),\n :ok \u003c-\n HNSWLib.Index.save_index(index, @saved_index) do\n\n {:reply, {:ok, idx}, state}\n else\n {:error, msg} -\u003e\n {:reply, {:error, msg}, state}\n end\n end\n\n def handle_call({:knn_search, nil}, _, state) do\n {:reply, {:error, \"No index found\"}, state}\n end\n\n def handle_call({:knn_search, input}, _, {index, _, _} = state) do\n # We search for the nearest neighbors of the input embedding.\n case HNSWLib.Index.knn_query(index, input, k: 1) do\n {:ok, labels, _distances} -\u003e\n\n response =\n labels[0]\n |\u003e Nx.to_flat_list()\n |\u003e hd()\n |\u003e then(fn idx -\u003e\n App.Repo.get_by(App.Image, %{idx: idx + 1})\n end)\n\n # TODO: add threshold on \"distances\"\n {:reply, response, state}\n\n {:error, msg} -\u003e\n {:reply, {:error, msg}, state}\n end\n end\n\n def handle_call(:not_empty, _, {index, _, _} = state) do\n case HNSWLib.Index.get_current_count(index) do\n {:ok, 0} -\u003e\n Logger.warning(\"⚠️ Empty index.\")\n {:reply, :error, state}\n\n {:ok, _} -\u003e\n {:reply, :ok, state}\n end\n end\nend\n\n```\n\nLet's unpack a bit of what we are doing here.\n\n- we first are **defining the module constants**.\n Here, we add the dimensions of the embedding vector space\n (these are dependent on the model you choose).\n Check with the model you've used to tweak this settings optimally.\n\n- define the upload directory where **the index file will be saved inside the filesystem**.\n- when the GenServer is initialized (`init/1` function),\n we perform several _integrity verifications_,\n checking if both the `Index` file in the filesystem\n and the file in the `Index` table\n (from now on, this table will be called `HnswlibIndex`,\n under the name of the same schema).\n These validations essentially make sure the content\n of both files are the same.\n\n- the other functions provide a basic API for\n callers to add items to the index file,\n so it is saved.\n\n#### 3.2 Saving the `HNSWLib` Index in the database\n\nAs you may have seen from the previous GenServer,\nwe are calling functions from a module called\n`App.HnswlibIndex` that we have not yet created.\n\nThis module pertains to the **schema** that will hold\ninformation of the `HNSWLib` table.\nThis table will only have a single row,\nwith the file contents.\nAs we've discussed earlier,\nwe will compare the Index file in this row\nwith the one in the filesystem\nto check for any inconsistencies that may arise.\n\nLet's implement this module now!\n\nInside `lib/app`, create a file called `hnswlib_index.ex`\nand use the following code.\n\n```elixir\ndefmodule App.HnswlibIndex do\n use Ecto.Schema\n alias App.HnswlibIndex\n\n require Logger\n\n @moduledoc \"\"\"\n Ecto schema to save the HNSWLib Index file into a singleton table\n with utility functions\n \"\"\"\n\n schema \"hnswlib_index\" do\n field(:file, :binary)\n field(:lock_version, :integer, default: 1)\n end\n\n def changeset(struct \\\\ %__MODULE__{}, params \\\\ %{}) do\n struct\n |\u003e Ecto.Changeset.cast(params, [:id, :file])\n |\u003e Ecto.Changeset.optimistic_lock(:lock_version)\n |\u003e Ecto.Changeset.validate_required([:id])\n end\n\n @doc \"\"\"\n Tries to load index from DB.\n If the table is empty, it creates a new one.\n If the table is not empty but there's no file, an index is created from scratch.\n If there's one, we use it and load it to be used throughout the application.\n \"\"\"\n def maybe_load_index_from_db(space, dim, max_elements) do\n # Check if the table has an entry\n App.Repo.get_by(HnswlibIndex, id: 1)\n |\u003e case do\n # If the table is empty\n nil -\u003e\n Logger.info(\"ℹ️ No index file found in DB. Creating new one...\")\n create(space, dim, max_elements)\n\n # If the table is not empty but has no file\n response when response.file == nil -\u003e\n Logger.info(\"ℹ️ Empty index file in DB. Recreating one...\")\n\n # Purge the table and create a new file row in it\n App.Repo.delete_all(App.HnswlibIndex)\n create(space, dim, max_elements)\n\n # If the table is not empty and has a file\n index_db -\u003e\n Logger.info(\"ℹ️ Index file found in DB. Loading it...\")\n\n # We get the path of the index\n with path \u003c- App.KnnIndex.index_path(),\n # Save the file on disk\n :ok \u003c- File.write(path, index_db.file),\n # And load it\n {:ok, index} \u003c- HNSWLib.Index.load_index(space, dim, path) do\n {:ok, index, index_db}\n end\n end\n end\n\n defp create(space, dim, max_elements) do\n # Inserting the row in the table\n {:ok, schema} =\n HnswlibIndex.changeset(%__MODULE__{}, %{id: 1})\n |\u003e App.Repo.insert()\n\n # Creates index\n {:ok, index} =\n HNSWLib.Index.new(space, dim, max_elements)\n\n # Builds index for testing only\n if Application.get_env(:app, :use_test_models, false) do\n empty_index =\n Application.app_dir(:app, [\"priv\", \"static\", \"uploads\"])\n |\u003e Path.join(\"indexes_empty.bin\")\n\n HNSWLib.Index.save_index(index, empty_index)\n end\n\n {:ok, index, schema}\n end\nend\n```\n\nIn this module:\n\n- we are creating **two fields**: `lock_version`,\n to simply check the version of the file;\n and `file`,\n the binary content of the index file.\n\n- `lock_version` will be extremely useful to\n perform [**optmistic locking**](https://stackoverflow.com/questions/129329/optimistic-vs-pessimistic-locking),\n which is what we do in the `changeset/2` function.\n This will allow us to prevent deadlocking\n when two different people upload the same image at the same time,\n and overcome any race condition that may occur.\n This will maintain the data consistency in the Index file.\n\n- `maybe_load_index_from_db/3` fetches the singleton row\n on this table and checks if the file exists in the row.\n If it doesn't, it creates a new one.\n Otherwise, it just loads the existing one inside the row.\n\n- `create/3` creates a new index file.\n It's a private function that encapsulates creating\n the Index file so it can be used in the singleton row\n inside the table.\n\nAnd that's it!\nWe've added additional code to conditionally create different indexes\naccording to the environment\n(useful for testing),\nbut you can safely ignore those conditional calls\nif you're not interested in testing\n(though you should πŸ˜›).\n\n#### 3.2 The embeding model\n\nWe provide a serving for the embedding model in the `App.Models` module.\nIt should look like this:\n\n```elixir\n#App.Models\n@embedding_model %ModelInfo{\n name: \"sentence-transformers/paraphrase-MiniLM-L6-v2\",\n cache_path: Path.join(@models_folder_path, \"paraphrase-MiniLM-L6-v2\"),\n load_featurizer: false,\n load_tokenizer: true,\n load_generation_config: true\n }\n\ndef embedding() do\n load_offline_model(@embedding_model)\n |\u003e then(fn response -\u003e\n case response do\n {:ok, model} -\u003e\n # return n %Nx.Serving{} struct\n %Nx.Serving{} =\n Bumblebee.Text.TextEmbedding.text_embedding(\n model.model_info,\n model.tokenizer,\n defn_options: [compiler: EXLA],\n preallocate_params: true\n )\n\n {:error, msg} -\u003e\n {:error, msg}\n end\n end)\n end\n\n\ndef verify_and_download_models() do\n force_models_download = Application.get_env(:app, :force_models_download, false)\n use_test_models = Application.get_env(:app, :use_test_models, false)\n\n case {force_models_download, use_test_models} do\n {true, true} -\u003e\n File.rm_rf!(@models_folder_path)\n download_model(@captioning_test_model)\n download_model(@audio_test_model)\n\n {true, false} -\u003e\n File.rm_rf!(@models_folder_path)\n download_model(@embedding_model)\n ^^^\n download_model(@captioning_prod_model)\n download_model(@audio_prod_model)\n\n {false, false} -\u003e\n check_folder_and_download(@embedding_model)\n ^^\n check_folder_and_download(@captioning_prod_model)\n check_folder_and_download(@audio_prod_model)\n\n {false, true} -\u003e\n check_folder_and_download(@captioning_test_model)\n check_folder_and_download(@audio_test_model)\n end\n end\n```\n\nYou then add the `Nx.Serving` for the embeddings:\n\n```elixir\n#application.ex\n\nchildren = [\n ...,\n {Nx.Serving,\n serving: App.Models.embedding(),\n name: Embedding,\n batch_size: 5\n },\n ...\n]\n```\n\nYour `application.ex` file should look like so:\n\n```elixir\ndefmodule App.Application do\n # See https://hexdocs.pm/elixir/Application.html\n # for more information on OTP Applications\n @moduledoc false\n require Logger\n use Application\n\n @upload_dir Application.app_dir(:app, [\"priv\", \"static\", \"uploads\"])\n\n @saved_index if Application.compile_env(:app, :knnindex_indices_test, false),\n do: Path.join(@upload_dir, \"indexes_test.bin\"),\n else: Path.join(@upload_dir, \"indexes.bin\")\n\n def check_models_on_startup do\n App.Models.verify_and_download_models()\n |\u003e case do\n {:error, msg} -\u003e\n Logger.error(\"⚠️ #{msg}\")\n System.stop(0)\n\n :ok -\u003e\n Logger.info(\"ℹ️ Models: βœ…\")\n :ok\n end\n end\n\n @impl true\n def start(_type, _args) do\n :ok = check_models_on_startup()\n\n children = [\n # Start the Telemetry supervisor\n AppWeb.Telemetry,\n # Setup DB\n App.Repo,\n # Start the PubSub system\n {Phoenix.PubSub, name: App.PubSub},\n # Nx serving for the embedding\n {Nx.Serving, serving: App.Models.embedding(), name: Embedding, batch_size: 1},\n # Nx serving for Speech-to-Text\n {Nx.Serving,\n serving:\n if Application.get_env(:app, :use_test_models) == true do\n App.Models.audio_serving_test()\n else\n App.Models.audio_serving()\n end,\n name: Whisper},\n # Nx serving for image classifier\n {Nx.Serving,\n serving:\n if Application.get_env(:app, :use_test_models) == true do\n App.Models.caption_serving_test()\n else\n App.Models.caption_serving()\n end,\n name: ImageClassifier},\n {GenMagic.Server, name: :gen_magic},\n\n # Adding a supervisor\n {Task.Supervisor, name: App.TaskSupervisor},\n # Start the Endpoint (http/https)\n AppWeb.Endpoint\n # Start a worker by calling: App.Worker.start_link(arg)\n # {App.Worker, arg}\n ]\n\n # We are starting the HNSWLib Index GenServer only during testing.\n # Because this GenServer needs the database to be seeded first,\n # we only add it when we're not testing.\n # When testing, you need to spawn this process manually (it is done in the test_helper.exs file).\n children =\n if Application.get_env(:app, :start_genserver, true) == true do\n Enum.concat(children, [{App.KnnIndex, [space: :cosine, index: @saved_index]}])\n else\n children\n end\n\n # See https://hexdocs.pm/elixir/Supervisor.html\n # for other strategies and supported options\n opts = [strategy: :one_for_one, name: App.Supervisor]\n Supervisor.start_link(children, opts)\n end\n\n # Tell Phoenix to update the endpoint configuration\n # whenever the application is updated.\n @impl true\n def config_change(changed, _new, removed) do\n AppWeb.Endpoint.config_change(changed, removed)\n :ok\n end\nend\n```\n\n\u003e [!NOTE]\n\u003e\n\u003e We have added a few alterations to how the supervision tree\n\u003e in `application.ex` is initialized.\n\u003e This is because we _test our code_,\n\u003e so that's why you see some of these changes above.\n\u003e\n\u003e If you don't want to change test the code,\n\u003e you can ignore the conditional changes that are made\n\u003e to the supervision tree according to the environment\n\u003e (which we do to check if the code is being tested or not).\n\n### 4. Using the Index and embeddings\n\nIn this section, we'll go over how to use the Index\nand the embeddings and tie everything together to\nhave a working application 😍.\n\nIf you want to better understand embeddings and\nhow to use `HNSWLib`,\nthe math behind it and see a working example\nof running an embedding model,\nyou can check the next section.\nHowever, _it is entirely optional_\nand not necessary for our app.\n\n#### 4.0 Check the folder \"hnswlib\"\n\nFor a working example on how to use the index in `hnswlib`,\nyou can run the \".exs\" file there.\n\n#### 4.1 Computing the embeddings in our app\n\n```elixir\n@tmp_wav Path.expand(\"priv/static/uploads/tmp.wav\")\n\ndef mount(_, _, socket) do\n {:ok,\n socket\n |\u003e assign(\n ...,\n # Related to the Audio\n transcription: nil,\n mic_off?: false,\n audio_running?: false,\n audio_search_result: nil,\n tmp_wav: @tmp_wav,\n )\n |\u003e allow_upload(:speech,...)\n [...]\n }\nend\n```\n\nRecall that every time you upload an image,\nyou get back a URL from our bucket\nand you compute a caption as a string.\nWe will now compute an embedding from this string\nand save it in the Index.\nThis is done in the `handle_info` callback.\n\nUpdate the Liveview `handle_info` callback where we handle the captioning results:\n\n```elixir\ndef handle_info({ref, result}, %{assigns: assigns} = socket) do\n # Flush async call\n Process.demonitor(ref, [:flush])\n\n cond do\n # If the upload task has finished executing,\n # we update the socket assigns.\n Map.get(assigns, :task_ref) == ref -\u003e\n image =\n %{\n url: assigns.image_info.url,\n width: assigns.image_info.width,\n height: assigns.image_info.height,\n description: label\n }\n\n with %{embedding: data} \u003c-\n Nx.Serving.batched_run(Embedding, label),\n # compute a normed embedding (cosine case only) on the text result\n normed_data \u003c-\n Nx.divide(data, Nx.LinAlg.norm(data)),\n {:check_used, {:ok, pending_image}} \u003c-\n {:check_used, App.Image.check_before_append_to_index(image.sha1)} do\n {:ok, idx} =\n App.KnnIndex.add_item(normed_data) do\n # save the App.Image to the DB\n Map.merge(image, %{idx: idx, caption: label})\n |\u003e App.Image.insert()\n\n {:noreply,\n socket\n |\u003e assign(\n upload_running?: false,\n task_ref: nil,\n label: label\n )\n }\n else\n {:error, msg} -\u003e\n {:noreply,\n socket\n |\u003e put_flash(:error, msg)\n |\u003e assign(\n upload_running?: false,\n task_ref: nil,\n label: nil\n )\n }\n end\n [...]\n end\nend\n```\n\nEvery time we produce an audio file, we transcribe it into a text.\nWe then compute the embedding of the audio input transcription and run an ANN search.\nThe last step should return a (possibly) populated `%App.Image{}` struct with a look-up in the database.\nWe then update the `\"audio_search_result\"` assign with it and display the transcription.\n\nModify the following handler:\n\n```elixir\ndef handle_info({ref, %{chunks: [%{text: text}]} = result}, %{assigns: assigns} = socket)\n when assigns.audio_ref == ref do\n Process.demonitor(ref, [:flush])\n File.rm!(@tmp_wav)\n\n # compute an normed embedding (cosine case only) on the text result\n # and returns an App.Image{} as the result of a \"knn_search\"\n with %{embedding: input_embedding} \u003c-\n Nx.Serving.batched_run(Embedding, text),\n normed_input_embedding \u003c-\n Nx.divide(input_embedding, Nx.LinAlg.norm(input_embedding)),\n {:not_empty_index, :ok} \u003c-\n {:not_empty_index, App.KnnIndex.not_empty_index()},\n # {:not_empty_index, App.HnswlibIndex.not_empty_index(index)},\n %App.Image{} = result \u003c-\n App.KnnIndex.knn_search(normed_input_embedding) do\n\n {:noreply,\n assign(socket,\n transcription: String.trim(text),\n mic_off?: false,\n audio_running?: false,\n audio_search_result: result,\n audio_ref: nil,\n tmp_wav: @tmp_wav\n )}\n # record without entries\n {:not_empty_index, :error} -\u003e\n {:noreply,\n assign(socket,\n mic_off?: false,\n audio_search_result: nil,\n audio_running?: false,\n audio_ref: nil,\n tmp_wav: @tmp_wav\n )}\n\n nil -\u003e\n {:noreply,\n assign(socket,\n transcription: String.trim(text),\n mic_off?: false,\n audio_search_result: nil,\n audio_running?: false,\n audio_ref: nil,\n tmp_wav: @tmp_wav\n )}\n end\nend\n```\n\nWe next come back to the `knn_search` function we defined in the \"KnnIndex\" GenServer.\nThe \"approximate nearest neighbour\" search function uses the function `HNSWLib.Index.knn_query/3`.\nIt returns a tuple `{:ok, indices, distances}` where \"indices\" and \"distances\" are lists.\nThe length is the number of neighbours you want to find parametrized by the `k` parameter.\nWith `k=1`, we ask for a single neighbour.\n\n\u003e [!NOTE]\n\u003e\n\u003e You may further use a cut-off distance to exclude responses that might not be meaningful.\n\nWe will now display the found image with the URL field of the `%App.Image{}` struct.\n\nAdd this to `\"page_live.html.heex\"`:\n\n```html\n\u003c!-- /lib/app_Web/live/page_live.html.heex --\u003e\n\n\u003cdiv :if=\"{@audio_search_result}\"\u003e\n \u003cimg src=\"{@audio_search_result.url}\" alt=\"found_image\" /\u003e\n\u003c/div\u003e\n```\n\n##### 4.1.1 Changing the `Image` schema so it's embeddable\n\nNow we'll save the index found.\nLet's add a column to the `Image` table.\nTo do this, run a `mix` task to generate a timestamped file.\n\n```bash\nmix ecto.gen.migration add_idx_to_images\n```\n\nIn the `\"/priv/repo\"` folder, open the newly created file and add:\n\n```elixir\ndefmodule App.Repo.Migrations.AddIdxToImages do\n use Ecto.Migration\n\n def change do\n alter table(:images) do\n add(:idx, :integer, default: 0)\n add(:sha1, :string)\n end\n end\nend\n```\n\nand run the migration\nby running `mix ecto.migrate`.\n\nModify the `App.Image` struct and the changeset:\n\n```elixir\n@primary_key {:id, :id, autogenerate: true}\nschema \"images\" do\n field(:description, :string)\n field(:width, :integer)\n field(:url, :string)\n field(:height, :integer)\n field(:idx, :integer)\n field(:sha1, :string)\n\n timestamps(type: :utc_datetime)\nend\n\ndef changeset(image, params \\\\ %{}) do\n image\n |\u003e Ecto.Changeset.cast(params, [:url, :description, :width, :height, :idx, :sha1])\n |\u003e Ecto.Changeset.validate_required([:width, :height])\n |\u003e Ecto.Changeset.unique_constraint(:sha1, name: :images_sha1_index)\n |\u003e Ecto.Changeset.unique_constraint(:idx, name: :images_idx_index)\nend\n```\n\nWe've added the fields `idx` and `sha1` to the image schema.\nThe former pertains to the index of the image\nwithin the `HNSWLIB` index file,\nso we can look for the image.\nThe latter pertains to the `sha1` representation of the image.\nThis will allow us to check if two images are the same,\nso we can avoid adding duplicate images\nand save some throughput in our application.\n\nIn our `changeset/2` function,\nwe've fundamentally added two `unique_constraint/3` functions\nto check for the uniqueness of the newly added\n`idx` and `sha1` function.\nThese are enforced at the database level so we don't have\nduplicated images.\n\nIn addition to these changes,\nwe are going to need functions to\n**calculate the `sha1` of the image**.\nAdd the following functions to the same file.\n\n```elixir\n def calc_sha1(file_binary) do\n :crypto.hash(:sha, file_binary)\n |\u003e Base.encode16()\n end\n\n def check_sha1(sha1) when is_binary(sha1) do\n App.Repo.get_by(App.Image, %{sha1: sha1})\n |\u003e case do\n nil -\u003e\n nil\n\n %App.Image{} = image -\u003e\n {:ok, image}\n end\n end\n```\n\n- `calc_sha1/1` uses the `:crypto` package to hash the file binary\n and encode it.\n- `check_sha1/1` fetches an image according to a given `sha1` code\n and returns the result.\n\nAnd that's all we need to deal with our images!\n\n##### 4.1.2 Using embeddings in semantic search\n\nNow we have\n\n- all the embedding models ready to be used,\n- our Index file correctly created and maintained through\n filesystem and in the database in the `hnswlib_index` schema,\n- the needed `sha1` functions to check for duplicated images.\n\nIt's time to bring everything together and use all of these tools\nto implement semantic search into our application.\n\nWe are going to be working inside `lib/app_web/live/page_live.ex` from now on.\n\n###### 4.1.2.1 Mount socket assigns\n\nFirst, we are going to update our socket assigns on `mount/3`.\n\n```elixir\n\n @image_width 640\n @accepted_mime ~w(image/jpeg image/jpg image/png image/webp)\n @tmp_wav Path.expand(\"priv/static/uploads/tmp.wav\")\n\n @impl true\n def mount(_params, _session, socket) do\n {:ok,\n socket\n |\u003e assign(\n # Related to the file uploaded by the user\n label: nil,\n upload_running?: false,\n task_ref: nil,\n image_info: nil,\n image_preview_base64: nil,\n\n # Related to the list of image examples\n example_list_tasks: [],\n example_list: [],\n display_list?: false,\n\n # Related to the Audio\n transcription: nil,\n mic_off?: false,\n audio_running?: false,\n audio_search_result: nil,\n tmp_wav: @tmp_wav\n )\n |\u003e allow_upload(:image_list,\n accept: ~w(image/*),\n auto_upload: true,\n progress: \u0026handle_progress/3,\n max_entries: 1,\n chunk_size: 64_000,\n max_file_size: 5_000_000\n )\n |\u003e allow_upload(:speech,\n accept: :any,\n auto_upload: true,\n progress: \u0026handle_progress/3,\n max_entries: 1\n )}\n end\n```\n\nTo reiterate:\n\n- we've added a few fields related to audio.\n\n - `transcription` will pertain to the result of the audio transcription\n that will occur after transcribing the audio from the person.\n - `mic_off?` is simply a toggle to visually show the person\n whether the microphone is recording or not.\n - `audio_running?` is a boolean to show the person\n if the audio transcription and semantic search are occurring (loading).\n - `audio_search_result` is the result of the image\n that is closest semantically to the image's label from the\n transcribed audio.\n - `tmp_wav` is the path of the temporary audio file\n that is saved in the filesystem while the audio is being transcribed.\n\n- additionally, we also have added\n `allow_upload/3` pertaining to the audio upload\n (it is tagged as `:speech` and is being handled\n in the same function as the upload `:image_list`).\n\nThese are the socket assigns\nthat will allow us to dynamically update the person using our app\nwith what the app is doing.\n\n###### 4.1.2.2 Consuming image uploads\n\nAs you can see, we are using `handle_progress/3`\nwith `allow_upload/3`.\nAs we know, `handle_progress/3` is called whenever an upload happens\n(whether an image or recording of the person's voice).\nWe define two different declarations for how we want to\nprocess `:image` uploads and `:speech` uploads.\n\nLet's start with the first one.\n\nWe have added `sha1` and `idx` as fields to our image schema.\nTherefore, we are going to need to make some changes\nto the `handle_progress/3` of the `:image_list`.\nChange it like so:\n\n```elixir\ndef handle_progress(:image_list, entry, socket) when entry.done? do\n # We consume the entry only if the entry is done uploading from the image\n # and if consuming the entry was successful.\n consume_uploaded_entry(socket, entry, fn %{path: path} -\u003e\n with {:magic, {:ok, %{mime_type: mime}}} \u003c- {:magic, magic_check(path)},\n # Check if file can be properly read\n {:read, {:ok, file_binary}} \u003c- {:read, File.read(path)},\n # Check the image info\n {:image_info, {mimetype, width, height, _variant}} \u003c-\n {:image_info, ExImageInfo.info(file_binary)},\n # Check mime type\n {:check_mime, :ok} \u003c- {:check_mime, check_mime(mime, mimetype)},\n # Get SHA1 code from the image and check it\n sha1 = App.Image.calc_sha1(file_binary),\n {:sha_check, nil} \u003c- {:sha_check, App.Image.check_sha1(sha1)},\n # Get image and resize\n {:ok, thumbnail_vimage} \u003c- Vops.thumbnail(path, @image_width, size: :VIPS_SIZE_DOWN),\n # Pre-process the image as tensor\n {:pre_process, {:ok, tensor}} \u003c- {:pre_process, pre_process_image(thumbnail_vimage)} do\n # Create image info to be saved as partial image\n image_info = %{\n mimetype: mimetype,\n width: width,\n height: height,\n sha1: sha1,\n description: nil,\n url: nil,\n # set a random big int to the \"idx\" field\n idx: :rand.uniform(1_000_000_000_000) * 1_000\n }\n\n # Save partial image\n App.Image.insert(image_info)\n |\u003e case do\n {:ok, _} -\u003e\n image_info =\n Map.merge(image_info, %{\n file_binary: file_binary\n })\n\n {:ok, %{tensor: tensor, image_info: image_info, path: path}}\n\n {:error, changeset} -\u003e\n {:error, changeset.errors}\n end\n |\u003e handle_upload()\n else\n {:magic, {:error, msg}} -\u003e {:postpone, %{error: msg}}\n {:read, msg} -\u003e {:postpone, %{error: inspect(msg)}}\n {:image_info, nil} -\u003e {:postpone, %{error: \"image_info error\"}}\n {:check_mime, :error} -\u003e {:postpone, %{error: \"Bad mime type\"}}\n {:sha_check, {:ok, %App.Image{}}} -\u003e {:postpone, %{error: \"Image already uploaded\"}}\n {:pre_process, {:error, _msg}} -\u003e {:postpone, %{error: \"pre_processing error\"}}\n {:error, reason} -\u003e {:postpone, %{error: inspect(reason)}}\n end\n end)\n |\u003e case do\n # If consuming the entry was successful, we spawn a task to classify the image\n # and update the socket assigns\n %{tensor: tensor, image_info: image_info} -\u003e\n task =\n Task.Supervisor.async(App.TaskSupervisor, fn -\u003e\n Nx.Serving.batched_run(ImageClassifier, tensor)\n end)\n\n # Encode the image to base64\n base64 = \"data:image/png;base64, \" \u003c\u003e Base.encode64(image_info.file_binary)\n\n {:noreply,\n assign(socket,\n upload_running?: true,\n task_ref: task.ref,\n image_preview_base64: base64,\n image_info: image_info\n )}\n\n # Otherwise, if there was an error uploading the image, we log the error and show it to the person.\n %{error: error} -\u003e\n Logger.warning(\"⚠️ Error uploading image. #{inspect(error)}\")\n {:noreply, push_event(socket, \"toast\", %{message: \"Image couldn't be uploaded to S3.\\n#{error}\"})}\n end\n end\n```\n\nLet's go over these changes.\nSome of these is code that has been written prior,\nbut for clarification, we'll go over them again.\n\n- we use `consume_uploaded_entry/3` to consume the image\n that the person uploads.\n To consume the image successfully,\n the image goes through an array of validations.\n\n - we use `magic_check/1` to check the MIME type of the image validity.\n - we read the contents of the image using `ExImageInfo.info/1`.\n - we check if the MIME type is valid using `check_mime/2`.\n - we calculate the `sha1` with the `App.Image.calc_sha1/1` function\n we've developed earlier.\n - we resize the image and scale it down to the same width\n as the images that are trained using the image captioning model we've chosen\n (to yield better results and to save memory bandwidth).\n We use `Vix.Operations.thumbnail/3` to resize the image.\n - finally, we convert the resized image to a tensor using\n `pre_process_image/1` so it can be consumed by our image captioning model.\n\n- after this series of validations,\n we use the image info we've obtained earlier to\n **create an \"early-save\" of the image**.\n With this, we are saving the image and associating it with\n the `sha1` that was retrieved from the image contents.\n We are doing this \"partial image saving\"\n in case two identical images are being uploaded at the same time.\n Because we are enforcing `sha1` to be unique at the database level,\n this race condition is solved by the database optimistically.\n\n- afterwards, we call `handle_upload/0`.\n This function will upload the image to the `S3` bucket.\n We are going to implement this function in just a second πŸ˜‰.\n\n- if the upload is successful,\n using the tensor and the image information from the previous steps,\n we spawn the async task to run the model.\n This step should be familiar to you\n since we've already implemented this.\n Finally, we update the socket assigns accordingly.\n\n- we handle all possible errors in the `else` statement of the\n `with` flow control statement before the image is uploaded.\n\nHopefully, this demystifies some of the code we've just implemented!\n\nBecause we are using `handle_upload/0` in this function\nto upload the image to our `S3` bucket,\nlet's do it right now!\n\n```elixir\ndef handle_upload({:ok, %{path: path, tensor: tensor, image_info: image_info} = map})\n when is_map(map) do\n # Upload the image to S3\n Image.upload_image_to_s3(path, image_info.mimetype)\n |\u003e case do\n # If the upload is successful, we update the socket assigns with the image info\n {:ok, url} -\u003e\n image_info =\n struct(\n %ImageInfo{},\n Map.merge(image_info, %{url: url})\n )\n\n {:ok, %{tensor: tensor, image_info: image_info}}\n\n # If S3 upload fails, we return error\n {:error, reason} -\u003e\n Logger.warning(\"⚠️ Error uploading image: #{inspect(reason)}\")\n {:postpone, %{error: \"Bucket error\"}}\n end\nend\n\ndef handle_upload({:error, error}) do\n Logger.warning(\"⚠️ Error creating partial image: #{inspect(error)}\")\n {:postpone, %{error: \"Error creating partial image\"}}\nend\n```\n\nThis function is fairly easy to understand.\nWe upload the image by calling `Image.upload_image_to_s3/2` and,\nif successful,\nwe add the returning URL to the image struct.\nOtherwise, we handle the error and return it.\n\nAfter this small detour,\nlet's implement the `handle_progress/3`\nfor the **`:speech` uploads**,\nthat is, the audio the person records.\n\n```elixir\n def handle_progress(:speech, entry, %{assigns: assigns} = socket) when entry.done? do\n # We consume the audio file\n tmp_wav =\n socket\n |\u003e consume_uploaded_entry(entry, fn %{path: path} -\u003e\n tmp_wav = assigns.tmp_wav \u003c\u003e Ecto.UUID.generate() \u003c\u003e \".wav\"\n :ok = File.cp!(path, tmp_wav)\n {:ok, tmp_wav}\n end)\n\n # After consuming the audio file, we spawn a task to transcribe the audio\n audio_task =\n Task.Supervisor.async(\n App.TaskSupervisor,\n fn -\u003e\n Nx.Serving.batched_run(Whisper, {:file, tmp_wav})\n end\n )\n\n # Update the socket assigns\n {:noreply,\n assign(socket,\n audio_ref: audio_task.ref,\n mic_off?: true,\n tmp_wav: tmp_wav,\n audio_running?: true,\n audio_search_result: nil,\n transcription: nil\n )}\n end\n```\n\nAs we know, this function is called after the upload is completed.\nIn the case of audio uploads,\nthe hook is called by the person recording their voice\nin `assets/js/app.js`.\nSimilarly to the `handle_progress/3` function of the `:image_list` uploads,\nwe also use `consume_uploaded_entry/3` to consume the audio file.\n\n- we consume the audio file and save it in our filesystem\n as a `.wav` file.\n- we spawn the async task and use the `whisper` audio transcription model\n with the audio file we've just saved.\n- we update the socket assigns accordingly.\n\nPretty simple, right?\n\n###### 4.1.2.3 Using the embeddings to semantically search images\n\nIn this section, we'll finally use\nour embedding model and semantically search for our images!\n\nAs you've seen in the previous section,\nwe've spawned the task to transcribe the audio into the `whipser` model.\nNow we need a handler!\nFor this scenario,\nadd the following function.\n\n```elixir\n @impl true\n def handle_info({ref, %{chunks: [%{text: text}]} = _result}, %{assigns: assigns} = socket)\n when assigns.audio_ref == ref do\n Process.demonitor(ref, [:flush])\n File.rm!(assigns.tmp_wav)\n\n # Compute an normed embedding (cosine case only) on the text result\n # and returns an App.Image{} as the result of a \"knn_search\"\n with {:not_empty_index, :ok} \u003c-\n {:not_empty_index, App.KnnIndex.not_empty_index()},\n %{embedding: input_embedding} \u003c-\n Nx.Serving.batched_run(Embedding, text),\n %Nx.Tensor{} = normed_input_embedding \u003c-\n Nx.divide(input_embedding, Nx.LinAlg.norm(input_embedding)),\n %App.Image{} = result \u003c-\n App.KnnIndex.knn_search(normed_input_embedding) do\n {:noreply,\n assign(socket,\n transcription: String.trim(text),\n mic_off?: false,\n audio_running?: false,\n audio_search_result: result,\n audio_ref: nil,\n tmp_wav: @tmp_wav\n )}\n else\n # Stop transcription if no entries in the Index\n {:not_empty_index, :error} -\u003e\n {:noreply,\n socket\n |\u003e push_event(\"toast\", %{message: \"No images yet\"})\n |\u003e assign(\n mic_off?: false,\n transcription: \"!! The image bank is empty. Please upload some !!\",\n audio_search_result: nil,\n audio_running?: false,\n audio_ref: nil,\n tmp_wav: @tmp_wav\n )}\n\n nil -\u003e\n {:noreply,\n assign(socket,\n transcription: String.trim(text),\n mic_off?: false,\n audio_search_result: nil,\n audio_running?: false,\n audio_ref: nil,\n tmp_wav: @tmp_wav\n )}\n end\n end\n```\n\nLet's break down this function:\n\n- given the **recording text transcription**:\n - we check if the Index file holding is _not empty_.\n - we use the text transcription and run it\n **through the embedding model** and get its result.\n - with the embedding we've received from the model,\n we **normalize it**.\n - with the normalized embedding,\n we \\*\\*run it through a `knn search`.\n For this, we call the `App.KnnIndex.knn_search/1` function\n we've defined in the `App.KnnIndex` GenServer\n we've implemented earlier on.\n - the `knn search` returns the closest semantical image\n (through the image caption)\n from the audio transcription.\n - upon the success of this process, we update the socket assigns.\n - otherwise, we handle each error case accordingly\n and update the socket assigns.\n\nAnd that's it!\nWe just add to sequentially call the functions\nthat we've implemented prior!\n\n###### 4.1.2.4 Creating embeddings when uploading images\n\nNow that we have _used_ the embeddings,\nthere's one thing we forgot:\n**we forgot to keep track of the embeddings of each image that is uploaded**.\nThese embeddings are saved in the Index file.\n\nTo fix this, we need to create an embedding of the image\nafter it is uploaded and captioned.\nHead over to the `handle_info/2` pertaining to the image captioning,\nand change it to the following piece of code:\n\n```elixir\n def handle_info({ref, result}, %{assigns: assigns} = socket) do\n # Flush async call\n Process.demonitor(ref, [:flush])\n\n # You need to change how you destructure the output of the model depending\n # on the model you've chosen for `prod` and `test` envs on `models.ex`.)\n label =\n case Application.get_env(:app, :use_test_models, false) do\n true -\u003e\n App.Models.extract_captioning_test_label(result)\n\n # coveralls-ignore-start\n false -\u003e\n App.Models.extract_captioning_prod_label(result)\n # coveralls-ignore-stop\n end\n\n %{image_info: image_info} = assigns\n\n cond do\n # If the upload task has finished executing, we run the embedding model on the image\n Map.get(assigns, :task_ref) == ref -\u003e\n image =\n %{\n url: image_info.url,\n width: image_info.width,\n height: image_info.height,\n description: label,\n sha1: image_info.sha1\n }\n\n # Create embedding task\n with %{embedding: data} \u003c- Nx.Serving.batched_run(Embedding, label),\n # Compute a normed embedding (cosine case only) on the text result\n normed_data \u003c- Nx.divide(data, Nx.LinAlg.norm(data)),\n # Check the SHA1 of the image\n {:check_used, {:ok, pending_image}} \u003c-\n {:check_used, App.Image.check_sha1(image.sha1)} do\n Ecto.Multi.new()\n # Save updated Image to DB\n |\u003e Ecto.Multi.run(:update_image, fn _, _ -\u003e\n idx = App.KnnIndex.get_count() + 1\n\n Ecto.Changeset.change(pending_image, %{\n idx: idx,\n description: image.description,\n url: image.url\n })\n |\u003e App.Repo.update()\n end)\n\n # Save Index file to DB\n |\u003e Ecto.Multi.run(:save_index, fn _, _ -\u003e\n {:ok, _idx} = App.KnnIndex.add_item(normed_data)\n App.KnnIndex.save_index_to_db()\n end)\n |\u003e App.Repo.transaction()\n |\u003e case do\n {:error, :update_image, _changeset, _} -\u003e\n {:noreply,\n socket\n |\u003e push_event(\"toast\", %{message: \"Invalid entry\"})\n |\u003e assign(\n upload_running?: false,\n task_ref: nil,\n label: nil\n )}\n\n {:error, :save_index, _, _} -\u003e\n {:noreply,\n socket\n |\u003e push_event(\"toast\", %{message: \"Please retry\"})\n |\u003e assign(\n upload_running?: false,\n task_ref: nil,\n label: nil\n )}\n\n {:ok, _} -\u003e\n {:noreply,\n socket\n |\u003e assign(\n upload_running?: false,\n task_ref: nil,\n label: label\n )}\n end\n else\n {:check_used, nil} -\u003e\n {:noreply,\n socket\n |\u003e push_event(\"toast\", %{message: \"Race condition\"})\n |\u003e assign(\n upload_running?: false,\n task_ref: nil,\n label: nil\n )}\n\n {:error, msg} -\u003e\n {:noreply,\n socket\n |\u003e push_event(\"toast\", %{message: msg})\n |\u003e assign(\n upload_running?: false,\n task_ref: nil,\n label: nil\n )}\n end\n\n # If the example task has finished executing, we upload the socket assigns.\n img = Map.get(assigns, :example_list_tasks) |\u003e Enum.find(\u0026(\u00261.ref == ref)) -\u003e\n # Update the element in the `example_list` enum to turn \"predicting?\" to `false`\n updated_example_list = update_example_list(assigns, img, label)\n\n {:noreply,\n assign(socket,\n example_list: updated_example_list,\n upload_running?: false,\n display_list?: true\n )}\n end\n end\n```\n\nLet's go over the flow of this function:\n\n- we extract the captioning label from the result of the image captioning model.\n This code is the same as it was before.\n- afterwards, we get the label\n and **feed it into the embedding model**.\n- the embedding model yields the embedding,\n _we normalize it_ and **check if the `sha1` code of the image is already being used**.\n- if these three processes occur successfuly,\n we perform a _database transaction_ where\n we **save the updated image to the database**,\n **update the Index file count (we increment it)**\n and **save the index file to the database**.\n- we finally update the socket assigns accordingly.\n- if any of the previous calls fail,\n we handle these error scenarios\n and update the socket assigns.\n\nAnd that's it!\nOur app is fully loaded with semantic search capabilities! πŸ”‹\n\n###### 4.1.2.5 Update the LiveView view\n\nAll that's left is updating our view.\nWe are going to add basic elements\nto make this transition as smooth as possible.\n\nHead over to `lib/app_web/live/page_live.html.heex`\nand update it as so:\n\n```html\n\u003cdiv class=\"hidden\" id=\"tracker_el\" phx-hook=\"ActivityTracker\" /\u003e\n\u003cdiv\n class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 sm:py-24 lg:px-8 xl:px-28 xl:py-32\"\n\u003e\n \u003cdiv class=\"flex flex-col justify-start\"\u003e\n \u003cdiv class=\"flex justify-center items-center w-full\"\u003e\n \u003cdiv class=\"2xl:space-y-12\"\u003e\n \u003cdiv class=\"mx-auto max-w-2xl lg:text-center\"\u003e\n \u003cp\u003e\n \u003cspan\n class=\"rounded-full w-fit bg-brand/5 px-2 py-1 text-[0.8125rem] font-medium text-center leading-6 text-brand\"\n \u003e\n \u003ca\n href=\"https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n \u003e\n πŸ”₯ LiveView\n \u003c/a\u003e\n +\n \u003ca\n href=\"https://github.com/elixir-nx/bumblebee\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n \u003e\n 🐝 Bumblebee\n \u003c/a\u003e\n \u003c/span\u003e\n \u003c/p\u003e\n \u003cp\n class=\"mt-2 text-3xl font-bold tracking-tight text-gray-900 sm:text-4xl\"\n \u003e\n Caption your image!\n \u003c/p\u003e\n \u003ch3 class=\"mt-6 text-lg leading-8 text-gray-600\"\u003e\n Upload your own image (up to 5MB) and perform image captioning with\n \u003ca\n href=\"https://elixir-lang.org/\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n class=\"font-mono font-medium text-sky-500\"\n \u003e\n Elixir\n \u003c/a\u003e\n !\n \u003c/h3\u003e\n \u003cp class=\"text-lg leading-8 text-gray-400\"\u003e\n Powered with\n \u003ca\n href=\"https://elixir-lang.org/\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n class=\"font-mono font-medium text-sky-500\"\n \u003e\n HuggingFaceπŸ€—\n \u003c/a\u003e\n transformer models, you can run this project locally and perform\n machine learning tasks with a handful lines of code.\n \u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv\u003e\u003c/div\u003e\n \u003cdiv class=\"border-gray-900/10\"\u003e\n \u003c!-- File upload section --\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n phx-drop-target=\"{@uploads.image_list.ref}\"\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003c!-- Show image preview --\u003e\n \u003c%= if @image_preview_base64 do %\u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c%= if not @upload_running? do %\u003e \n \u003c.live_file_input upload={@uploads.image_list} class=\"hidden\" /\u003e \n \u003c% end %\u003e\n \u003cimg src=\"{@image_preview_base64}\" /\u003e\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c% else %\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list}\n class=\"hidden\" /\u003e Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003e\n PNG, JPG, GIF up to 5MB\n \u003c/p\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c!-- Show errors --\u003e\n \u003c%= for entry \u003c- @uploads.image_list.entries do %\u003e\n \u003cdiv class=\"mt-2\"\u003e\n \u003c%= for err \u003c- upload_errors(@uploads.image_list, entry) do %\u003e\n \u003cdiv class=\"rounded-md bg-red-50 p-4 mb-2\"\u003e\n \u003cdiv class=\"flex\"\u003e\n \u003cdiv class=\"flex-shrink-0\"\u003e\n \u003csvg\n class=\"h-5 w-5 text-red-400\"\n viewBox=\"0 0 20 20\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M10 18a8 8 0 100-16 8 8 0 000 16zM8.28 7.22a.75.75 0 00-1.06 1.06L8.94 10l-1.72 1.72a.75.75 0 101.06 1.06L10 11.06l1.72 1.72a.75.75 0 101.06-1.06L11.06 10l1.72-1.72a.75.75 0 00-1.06-1.06L10 8.94 8.28 7.22z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003c/div\u003e\n \u003cdiv class=\"ml-3\"\u003e\n \u003ch3 class=\"text-sm font-medium text-red-800\"\u003e\n \u003c%= error_to_string(err) %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c!-- Prediction text --\u003e\n \u003cdiv\n class=\"flex mt-2 space-x-1.5 items-center font-bold text-gray-900 text-xl\"\n \u003e\n \u003cspan\u003eDescription: \u003c/span\u003e\n \u003c!-- conditional Spinner or display caption text or waiting text--\u003e\n \u003cAppWeb.Spinner.spin spin=\"{@upload_running?}\" /\u003e\n \u003c%= if @label do %\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @label %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light\"\u003eWaiting for image input.\u003c/span\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c!-- Audio --\u003e\n \u003cbr /\u003e\n \u003cdiv class=\"mx-auto max-w-2xl lg\"\u003e\n \u003ch2\n class=\"mt-2 text-3xl font-bold tracking-tight text-gray-900 sm:text-4xl text-center\"\n \u003e\n Semantic search using an audio\n \u003c/h2\u003e\n \u003cbr /\u003e\n \u003cp\u003e\n Please record a phrase. You can listen to your audio. It will be\n transcripted automatically into a text and appear below. The semantic\n search for matching images will then run automatically and the found\n image appear below.\n \u003c/p\u003e\n \u003cbr /\u003e\n \u003cform\n id=\"audio-upload-form\"\n phx-change=\"noop\"\n class=\"flex flex-col items-center\"\n \u003e\n \u003c.live_file_input upload={@uploads.speech} class=\"hidden\" /\u003e\n \u003cbutton\n id=\"record\"\n class=\"bg-blue-500 hover:bg-blue-700 text-white font-bold px-4 rounded flex\"\n type=\"button\"\n phx-hook=\"Audio\"\n disabled=\"{@mic_off?}\"\n \u003e\n \u003cHeroicons.microphone\n outline\n class=\"w-6 h-6 text-white font-bold group-active:animate-pulse\"\n /\u003e\n \u003cspan id=\"text\"\u003eRecord\u003c/span\u003e\n \u003c/button\u003e\n \u003c/form\u003e\n \u003cbr /\u003e\n \u003cp class=\"flex flex-col items-center\"\u003e\n \u003caudio id=\"audio\" controls\u003e\u003c/audio\u003e\n \u003c/p\u003e\n \u003cbr /\u003e\n \u003cdiv\n class=\"flex mt-2 space-x-1.5 items-center font-bold text-gray-900 text-xl\"\n \u003e\n \u003cspan\u003eTranscription: \u003c/span\u003e\n \u003c%= if @audio_running? do %\u003e\n \u003cAppWeb.Spinner.spin spin=\"{@audio_running?}\" /\u003e\n \u003c% else %\u003e \n \u003c%= if @transcription do %\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @transcription %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light text-justify\"\u003eWaiting for audio input.\u003c/span\u003e\n \u003c% end %\u003e \n \u003c% end %\u003e\n \u003c/div\u003e\n \u003cbr /\u003e\n\n \u003cdiv :if=\"{@audio_search_result}\"\u003e\n \u003cdiv class=\"border-gray-900/10\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n \u003e\n \u003cimg src=\"{@audio_search_result.url}\" alt=\"found_image\" /\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c!-- Examples --\u003e\n \u003cdiv :if=\"{@display_list?}\" class=\"flex flex-col\"\u003e\n \u003ch3\n class=\"mt-10 text-xl lg:text-center font-light tracking-tight text-gray-900 lg:text-2xl\"\n \u003e\n Examples\n \u003c/h3\u003e\n \u003cdiv class=\"flex flex-row justify-center my-8\"\u003e\n \u003cdiv\n class=\"mx-auto grid max-w-2xl grid-cols-1 gap-x-6 gap-y-20 sm:grid-cols-2\"\n \u003e\n \u003c%= for example_img \u003c- @example_list do %\u003e\n \u003c!-- Loading skeleton if it is predicting --\u003e\n \u003c%= if example_img.predicting? == true do %\u003e\n \u003cdiv\n role=\"status\"\n class=\"flex items-center justify-center w-full h-full max-w-sm bg-gray-300 rounded-lg animate-pulse\"\n \u003e\n \u003cimg src={~p\"/images/spinner.svg\"} alt=\"spinner\" /\u003e\n \u003cspan class=\"sr-only\"\u003eLoading...\u003c/span\u003e\n \u003c/div\u003e\n \u003c% else %\u003e\n \u003cdiv\u003e\n \u003cimg\n id=\"{example_img.url}\"\n src=\"{example_img.url}\"\n class=\"rounded-2xl object-cover\"\n /\u003e\n \u003ch3 class=\"mt-1 text-lg leading-8 text-gray-900 text-center\"\u003e\n \u003c%= example_img.label %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nAs you may have noticed,\nwe've made some changes to the Audio portion of the HTML.\n\n- we check if the `@transcription` assign exists.\n If so, we display the text to the person.\n- we check if the `@audio_search_result` assign is not `nil`.\n If that's the case, the image that is semantically closest\n to the audio transcription is shown to the person.\n\nAnd that's it!\nWe are simply showing the person\nthe results.\n\nAnd with that, you've successfully added\nsemantic search into the application!\nPat yourself on the back! πŸ‘\n\nYou've expanded your knowledge in key areas of machine learning\nand artificial intelligence,\nthat is increasingly becoming more prevalent!\n\n### 5. Tweaking our UI\n\nNow that we have all the features we want in our application,\nlet's make it prettier!\nAs it stands, it's responsive enough.\nBut we can always make it better!\n\nWe're going to show you the changes you're going to need to make\nand then explain it to you what it means!\n\nHead over to `lib/app_web/live/page_live.html.heex` and change it like so:\n\n```html\n\u003cdiv class=\"hidden\" id=\"tracker_el\" phx-hook=\"ActivityTracker\" /\u003e\n\u003cdiv class=\"h-full w-full px-4 py-10 flex justify-center sm:px-6 xl:px-28\"\u003e\n \u003cdiv class=\"flex flex-col justify-start lg:w-full\"\u003e\n \u003cdiv class=\"flex justify-center items-center w-full\"\u003e\n \u003cdiv class=\"w-full 2xl:space-y-12\"\u003e\n \u003cdiv class=\"mx-auto lg:text-center\"\u003e\n\n \u003c!-- Title pill --\u003e\n \u003cp class=\"text-center\"\u003e\n \u003cspan class=\"rounded-full w-fit bg-brand/5 px-2 py-1 text-[0.8125rem] font-medium text-center leading-6 text-brand\"\u003e\n \u003ca\n href=\"https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n \u003e\n πŸ”₯ LiveView\n \u003c/a\u003e\n +\n \u003ca\n href=\"https://github.com/elixir-nx/bumblebee\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n \u003e\n 🐝 Bumblebee\n \u003c/a\u003e\n \u003c/span\u003e\n \u003c/p\u003e\n\n \u003c!-- Toggle Buttons --\u003e\n \u003cdiv class=\"flex justify-center lg:invisible\"\u003e\n \u003cspan class=\"isolate inline-flex rounded-md shadow-sm mt-2\"\u003e\n \u003cbutton id=\"upload_option\" type=\"button\" class=\"relative inline-flex items-center gap-x-1.5 rounded-l-md bg-blue-500 text-white hover:bg-blue-600 px-3 py-2 text-sm font-semibold ring-1 ring-inset ring-gray-300 focus:z-10\"\u003e\n \u003csvg fill=\"none\" viewBox=\"0 0 24 24\" stroke-width=\"1.5\" stroke=\"currentColor\" class=\"-ml-0.5 h-5 w-5 text-white\"\u003e\n \u003cpath stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M3 16.5v2.25A2.25 2.25 0 0 0 5.25 21h13.5A2.25 2.25 0 0 0 21 18.75V16.5m-13.5-9L12 3m0 0 4.5 4.5M12 3v13.5\" /\u003e\n \u003c/svg\u003e\n Upload\n \u003c/button\u003e\n \u003cbutton id=\"search_option\" type=\"button\" class=\"relative -ml-px inline-flex items-center rounded-r-md bg-white px-3 py-2 text-sm font-semibold text-gray-900 ring-1 ring-inset ring-gray-300 hover:bg-gray-50 focus:z-10\"\u003e\n \u003csvg fill=\"none\" viewBox=\"0 0 24 24\" stroke-width=\"1.5\" stroke=\"currentColor\" class=\"-ml-0.5 h-5 w-5 text-gray-400\"\u003e\n \u003cpath stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"m21 21-5.197-5.197m0 0A7.5 7.5 0 1 0 5.196 5.196a7.5 7.5 0 0 0 10.607 10.607Z\" /\u003e\n \u003c/svg\u003e\n Search\n \u003c/button\u003e\n \u003c/span\u003e\n \u003c/div\u003e\n\n \u003c!-- Containers --\u003e\n \u003cdiv class=\"flex flex-col lg:flex-row lg:justify-around\"\u003e\n \u003c!-- UPLOAD CONTAINER --\u003e\n \u003cdiv id=\"upload_container\" class=\"mb-6 lg:px-10\"\u003e\n \u003cp class=\"mt-2 text-center text-3xl font-bold tracking-tight text-gray-900 sm:text-4xl\"\u003e\n Caption your image!\n \u003c/p\u003e\n \u003cdiv class=\"flex gap-x-4 rounded-xl bg-black/5 px-6 py-2 mt-2\"\u003e\n \u003cdiv class=\"flex flex-col justify-center items-center\"\u003e\n \u003csvg fill=\"none\" viewBox=\"0 0 24 24\" stroke-width=\"1.5\" stroke=\"currentColor\" class=\"w-7 h-7 text-indigo-400\"\u003e\n \u003cpath stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 8.25H7.5a2.25 2.25 0 0 0-2.25 2.25v9a2.25 2.25 0 0 0 2.25 2.25h9a2.25 2.25 0 0 0 2.25-2.25v-9a2.25 2.25 0 0 0-2.25-2.25H15m0-3-3-3m0 0-3 3m3-3V15\" /\u003e\n \u003c/svg\u003e\n \u003c/div\u003e\n \u003cdiv class=\"text-sm leading-2 text-justify flex flex-col justify-center\"\u003e\n \u003cp class=\"text-slate-700\"\u003e\n Upload your own image (up to 5MB) and perform image captioning with\n \u003ca\n href=\"https://elixir-lang.org/\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n class=\"font-mono font-medium text-sky-500\"\n \u003e\n Elixir\n \u003c/a\u003e\n !\n \u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003cp class=\"mt-4 text-center text-sm leading-2 text-gray-400\"\u003e\n Powered with\n \u003ca\n href=\"https://elixir-lang.org/\"\n target=\"_blank\"\n rel=\"noopener noreferrer\"\n class=\"font-mono font-medium text-sky-500\"\n \u003e\n HuggingFaceπŸ€—\n \u003c/a\u003e\n transformer models,\n you can run this project locally and perform machine learning tasks with a handful lines of code.\n \u003c/p\u003e\n\n \u003c!-- File upload section --\u003e\n \u003cdiv class=\"border-gray-900/10 mt-4\"\u003e\n \u003cdiv class=\"col-span-full\"\u003e\n \u003cdiv\n class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\n phx-drop-target={@uploads.image_list.ref}\n \u003e\n \u003cdiv class=\"text-center\"\u003e\n \u003c!-- Show image preview --\u003e\n \u003c%= if @image_preview_base64 do %\u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c%= if not @upload_running? do %\u003e\n \u003c.live_file_input upload={@uploads.image_list} class=\"hidden\" /\u003e\n \u003c% end %\u003e\n \u003cimg src={@image_preview_base64} /\u003e\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c% else %\u003e\n \u003csvg\n class=\"mx-auto h-12 w-12 text-gray-300\"\n viewBox=\"0 0 24 24\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M1.5 6a2.25 2.25 0 012.25-2.25h16.5A2.25 2.25 0 0122.5 6v12a2.25 2.25 0 01-2.25 2.25H3.75A2.25 2.25 0 011.5 18V6zM3 16.06V18c0 .414.336.75.75.75h16.5A.75.75 0 0021 18v-1.94l-2.69-2.689a1.5 1.5 0 00-2.12 0l-.88.879.97.97a.75.75 0 11-1.06 1.06l-5.16-5.159a1.5 1.5 0 00-2.12 0L3 16.061zm10.125-7.81a1.125 1.125 0 112.25 0 1.125 1.125 0 01-2.25 0z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003cdiv class=\"mt-4 flex text-sm leading-6 text-gray-600\"\u003e\n \u003clabel\n for=\"file-upload\"\n class=\"relative cursor-pointer rounded-md bg-white font-semibold text-indigo-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-indigo-600 focus-within:ring-offset-2 hover:text-indigo-500\"\n \u003e\n \u003cform id=\"upload-form\" phx-change=\"noop\" phx-submit=\"noop\"\u003e\n \u003clabel class=\"cursor-pointer\"\u003e\n \u003c.live_file_input upload={@uploads.image_list} class=\"hidden\" /\u003e Upload\n \u003c/label\u003e\n \u003c/form\u003e\n \u003c/label\u003e\n \u003cp class=\"pl-1\"\u003eor drag and drop\u003c/p\u003e\n \u003c/div\u003e\n \u003cp class=\"text-xs leading-5 text-gray-600\"\u003ePNG, JPG, GIF up to 5MB\u003c/p\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\n \u003c!-- Show errors --\u003e\n \u003c%= for entry \u003c- @uploads.image_list.entries do %\u003e\n \u003cdiv class=\"mt-2\"\u003e\n \u003c%= for err \u003c- upload_errors(@uploads.image_list, entry) do %\u003e\n \u003cdiv class=\"rounded-md bg-red-50 p-4 mb-2\"\u003e\n \u003cdiv class=\"flex\"\u003e\n \u003cdiv class=\"flex-shrink-0\"\u003e\n \u003csvg\n class=\"h-5 w-5 text-red-400\"\n viewBox=\"0 0 20 20\"\n fill=\"currentColor\"\n aria-hidden=\"true\"\n \u003e\n \u003cpath\n fill-rule=\"evenodd\"\n d=\"M10 18a8 8 0 100-16 8 8 0 000 16zM8.28 7.22a.75.75 0 00-1.06 1.06L8.94 10l-1.72 1.72a.75.75 0 101.06 1.06L10 11.06l1.72 1.72a.75.75 0 101.06-1.06L11.06 10l1.72-1.72a.75.75 0 00-1.06-1.06L10 8.94 8.28 7.22z\"\n clip-rule=\"evenodd\"\n /\u003e\n \u003c/svg\u003e\n \u003c/div\u003e\n \u003cdiv class=\"ml-3\"\u003e\n \u003ch3 class=\"text-sm font-medium text-red-800\"\u003e\n \u003c%= error_to_string(err) %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n\n \u003c!-- Prediction text --\u003e\n \u003cdiv class=\"flex mt-2 space-x-1.5 items-center\"\u003e\n \u003cspan class=\"font-bold text-gray-900\"\u003eDescription: \u003c/span\u003e\n \u003c!-- conditional Spinner or display caption text or waiting text--\u003e\n \u003c%= if @upload_running? do %\u003e\n \u003cAppWeb.Spinner.spin spin={@upload_running?} /\u003e\n \u003c% else %\u003e\n \u003c%= if @label do %\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @label %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light text-justify\"\u003eWaiting for image input.\u003c/span\u003e\n \u003c% end %\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n\n \u003c!-- Examples --\u003e\n \u003c%= if @display_list? do %\u003e\n \u003cdiv :if={@display_list?} class=\"mt-16 flex flex-col\"\u003e\n \u003ch3 class=\"text-xl text-center font-bold tracking-tight text-gray-900 lg:text-2xl\"\u003e\n Examples\n \u003c/h3\u003e\n \u003cdiv class=\"flex flex-row justify-center my-8\"\u003e\n \u003cdiv class=\"mx-auto grid max-w-2xl grid-cols-1 gap-x-6 gap-y-10 sm:grid-cols-2\"\u003e\n \u003c%= for example_img \u003c- @example_list do %\u003e\n \u003c!-- Loading skeleton if it is predicting --\u003e\n \u003c%= if example_img.predicting? == true do %\u003e\n \u003cdiv\n role=\"status\"\n class=\"flex items-center justify-center w-full h-full max-w-sm bg-gray-300 rounded-lg animate-pulse\"\n \u003e\n \u003cimg src={~p\"/images/spinner.svg\"} alt=\"spinner\" /\u003e\n \u003cspan class=\"sr-only\"\u003eLoading...\u003c/span\u003e\n \u003c/div\u003e\n \u003c% else %\u003e\n \u003cdiv\u003e\n \u003cimg id={example_img.url} src={example_img.url} class=\"rounded-2xl object-cover\" /\u003e\n \u003ch3 class=\"mt-1 text-lg leading-8 text-gray-900 text-center\"\u003e\n \u003c%= example_img.label %\u003e\n \u003c/h3\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n\n \u003c!-- AUDIO SEMANTIC SEARCH CONTAINER --\u003e\n \u003cdiv id=\"search_container\" class=\"hidden mb-6 mx-auto lg:block lg:px-10\"\u003e\n \u003ch2 class=\"mt-2 text-3xl font-bold tracking-tight text-gray-900 sm:text-4xl text-center\"\u003e\n ...or search it!\n \u003c/h2\u003e\n\n \u003cdiv class=\"flex gap-x-4 rounded-xl bg-black/5 px-6 py-2 mt-2\"\u003e\n \u003cdiv class=\"flex flex-col justify-center items-center\"\u003e\n \u003csvg fill=\"none\" viewBox=\"0 0 24 24\" stroke-width=\"1.5\" stroke=\"currentColor\" class=\"w-7 h-7 text-indigo-400\"\u003e\n \u003cpath stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M12 18.75a6 6 0 0 0 6-6v-1.5m-6 7.5a6 6 0 0 1-6-6v-1.5m6 7.5v3.75m-3.75 0h7.5M12 15.75a3 3 0 0 1-3-3V4.5a3 3 0 1 1 6 0v8.25a3 3 0 0 1-3 3Z\" /\u003e\n \u003c/svg\u003e\n \u003c/div\u003e\n \u003cdiv class=\"text-sm leading-2 text-justify flex flex-col justify-center\"\u003e\n \u003cp class=\"text-slate-700\"\u003e\n Record a phrase or some key words.\n We'll detect them and semantically search it in our database of images!\n \u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003cp class=\"mt-4 text-center text-sm leading-2 text-gray-400\"\u003e\n After recording your audio, you can listen to it. It will be transcripted automatically into text and appear below.\n \u003c/p\u003e\n \u003cp class=\"text-center text-sm leading-2 text-gray-400\"\u003e\n Semantic search will automatically kick in and the resulting image will be shown below.\n \u003c/p\u003e\n\n \u003c!-- Audio recording button --\u003e\n \u003cform id=\"audio-upload-form\" phx-change=\"noop\" class=\"mt-8 flex flex-col items-center\"\u003e\n \u003c.live_file_input upload={@uploads.speech} class=\"hidden\" /\u003e\n \u003cbutton\n id=\"record\"\n class=\"bg-blue-500 hover:bg-blue-700 text-white font-bold p-4 rounded flex\"\n type=\"button\"\n phx-hook=\"Audio\"\n disabled={@mic_off?}\n \u003e\n \u003cHeroicons.microphone\n outline\n class=\"w-6 h-6 text-white font-bold group-active:animate-pulse\"\n /\u003e\n \u003cspan id=\"text\"\u003eRecord\u003c/span\u003e\n \u003c/button\u003e\n \u003c/form\u003e\n\n \u003c!-- Audio preview --\u003e\n \u003cp class=\"flex flex-col items-center mt-6\"\u003e\n \u003caudio id=\"audio\" controls\u003e\u003c/audio\u003e\n \u003c/p\u003e\n\n \u003c!-- Audio transcription --\u003e\n \u003cdiv class=\"flex mt-2 space-x-1.5 items-center\"\u003e\n \u003cspan class=\"font-bold text-gray-900\"\u003eTranscription: \u003c/span\u003e\n \u003c%= if @audio_running? do %\u003e\n \u003cAppWeb.Spinner.spin spin={@audio_running?} /\u003e\n \u003c% else %\u003e\n \u003c%= if @transcription do %\u003e\n \u003cspan id=\"output\" class=\"text-gray-700 font-light\"\u003e\u003c%= @transcription %\u003e\u003c/span\u003e\n \u003c% else %\u003e\n \u003cspan class=\"text-gray-300 font-light text-justify\"\u003eWaiting for audio input.\u003c/span\u003e\n \u003c% end %\u003e\n \u003c% end %\u003e\n \u003c/div\u003e\n\n \u003c!-- Semantic search result --\u003e\n \u003cdiv :if={@audio_search_result}\u003e\n \u003cdiv class=\"border-gray-900/10\"\u003e\n \u003cdiv class=\"mt-2 flex justify-center rounded-lg border border-dashed border-gray-900/25 px-6 py-10\"\u003e\n \u003cimg src={@audio_search_result.url} alt=\"found_image\" /\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003cspan class=\"text-gray-700 font-light\"\u003e\u003c%= @audio_search_result.description %\u003e\u003c/span\u003e\n \u003c/div\u003e\n\n \u003c/div\u003e\n \u003c/div\u003e\n\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n```\n\nThat may look like a lot, but we've done just a handful of changes!\n\n- we've reestructured our HTML so it's easier to read.\n You just have a few key elements: the pill on top of the page,\n a toggle that we've added (to show the **upload section**\n and the **search section**) and two containers\n with the upload and search section, respectively.\n The code is practically intact for each section.\n\n- we've made minor styling changes to each section.\n On the _upload section_,\n we added a small callout section.\n In the _search section_,\n we added a small calout section as well,\n and added the description of the image\n that is found after the audio transcription occurs.\n\nAnd that's it!\nWhat's important\n**is that only one second is shown at a time on mobile devices**\nand **both sections are shown on desktop devices**\n(over `1024` pixels - [pertaining to the `lg` breakpoint of `TailwindCSS`](https://tailwindcss.com/docs/responsive-design)).\nOn desktop devices, the toggle button should disappear.\n\nTo accomplish these, we need to add a bit of `Javascript` and `CSS` magic. πŸͺ„\nHead over to `assets/js/app.js`\nand add the following code.\n\n```js\ndocument.getElementById(\"upload_option\").addEventListener(\"click\", function () {\n document.getElementById(\"upload_container\").style.display = \"block\";\n document.getElementById(\"search_container\").style.display = \"none\";\n\n document\n .getElementById(\"upload_option\")\n .classList.replace(\"bg-white\", \"bg-blue-500\");\n document\n .getElementById(\"upload_option\")\n .classList.replace(\"text-gray-900\", \"text-white\");\n document\n .getElementById(\"upload_option\")\n .classList.replace(\"hover:bg-gray-50\", \"hover:bg-blue-600\");\n document\n .getElementById(\"upload_option\")\n .getElementsByTagName(\"svg\")[0]\n .classList.replace(\"text-gray-400\", \"text-white\");\n\n document\n .getElementById(\"search_option\")\n .classList.replace(\"bg-blue-500\", \"bg-white\");\n document\n .getElementById(\"search_option\")\n .classList.replace(\"text-white\", \"text-gray-900\");\n document\n .getElementById(\"search_option\")\n .classList.replace(\"hover:bg-blue-600\", \"hover:bg-gray-50\");\n document\n .getElementById(\"search_option\")\n .getElementsByTagName(\"svg\")[0]\n .classList.replace(\"text-white\", \"text-gray-400\");\n});\n\ndocument.getElementById(\"search_option\").addEventListener(\"click\", function () {\n document.getElementById(\"upload_container\").style.display = \"none\";\n document.getElementById(\"search_container\").style.display = \"block\";\n\n document\n .getElementById(\"search_option\")\n .classList.replace(\"bg-white\", \"bg-blue-500\");\n document\n .getElementById(\"search_option\")\n .classList.replace(\"text-gray-900\", \"text-white\");\n document\n .getElementById(\"search_option\")\n .classList.replace(\"hover:bg-gray-50\", \"hover:bg-blue-600\");\n document\n .getElementById(\"search_option\")\n .getElementsByTagName(\"svg\")[0]\n .classList.replace(\"text-gray-400\", \"text-white\");\n\n document\n .getElementById(\"upload_option\")\n .classList.replace(\"bg-blue-500\", \"bg-white\");\n document\n .getElementById(\"upload_option\")\n .classList.replace(\"text-white\", \"text-gray-900\");\n document\n .getElementById(\"upload_option\")\n .classList.replace(\"hover:bg-blue-600\", \"hover:bg-gray-50\");\n document\n .getElementById(\"upload_option\")\n .getElementsByTagName(\"svg\")[0]\n .classList.replace(\"text-white\", \"text-gray-400\");\n});\n```\n\nThe code is self-explanatory.\nWe are changing the styles of the toggle buttons\naccording to the button that is clicked.\n\nThe other thing we need to make sure is that\n**both sections are shown on desktop devices**,\nregardless of what the current section is selected.\nLuckily, we can override styles by adding this piece of code\nto `assets/css/app.css`.\n\n```css\n@media (min-width: 1024px) {\n #upload_container,\n #search_container {\n display: block !important; /* Override any inline styles */\n }\n}\n```\n\nAnd that's it!\nWe can see our slightly refactored UI in all of its glory\nby running `mix phx.server`!\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"https://github.com/dwyl/ping/assets/17494745/bc69395a-e793-4b94-b214-27d5528d868b\"\u003e\n\u003c/p\u003e\n\n## _Please_ star the repo! ⭐️\n\nIf you find this package/repo useful,\nplease star it on GitHub, so that we know! ⭐\n\nThank you! πŸ™\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdwyl%2Fimage-classifier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdwyl%2Fimage-classifier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdwyl%2Fimage-classifier/lists"}