{"id":21672444,"url":"https://github.com/redis-developer/finding-bigfoot-with-semantic-search","last_synced_at":"2025-08-02T02:40:12.249Z","repository":{"id":217711384,"uuid":"744598965","full_name":"redis-developer/finding-bigfoot-with-semantic-search","owner":"redis-developer","description":"Sample application showing how to use semantic search with Redis and JavaScript.","archived":false,"fork":false,"pushed_at":"2024-08-03T14:27:29.000Z","size":64584,"stargazers_count":23,"open_issues_count":2,"forks_count":12,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-12T03:52:53.277Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redis-developer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-17T16:19:54.000Z","updated_at":"2025-02-26T22:37:27.000Z","dependencies_parsed_at":"2024-08-03T15:57:25.739Z","dependency_job_id":null,"html_url":"https://github.com/redis-developer/finding-bigfoot-with-semantic-search","commit_stats":null,"previous_names":["guyroyse/finding-bigfoot-with-semantic-search","redis-developer/finding-bigfoot-with-semantic-search"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/redis-developer/finding-bigfoot-with-semantic-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-developer%2Ffinding-bigfoot-with-semantic-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-developer%2Ffinding-bigfoot-with-semantic-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-developer%2Ffinding-bigfoot-with-semantic-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-developer%2Ffinding-bigfoot-with-semantic-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redis-developer","download_url":"https://codeload.github.com/redis-developer/finding-bigfoot-with-semantic-search/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis-developer%2Ffinding-bigfoot-with-semantic-search/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259894138,"owners_count":22928132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T13:29:16.364Z","updated_at":"2025-06-14T22:39:04.955Z","avatar_url":"https://github.com/redis-developer.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Finding Bigfoot with JavaScript + vector search\n\nThis repository contains a sample application that uses Semantic Search to find Bigfoot sightings in Redis. It uses a lot of popular AI libraries to make it work, but I had three things I wanted to show when I wrote this:\n\n 1. You don't have to use Python to use AI.\n 2. You don't have to plop down a credit card to use AI.\n 3. You can run AI on your local machine.\n\nTo that end, I used JavaScript, and only JavaScript to build a simple web application. There's no Python to be found. Not that Python is bad, just that lots of us like JavaScript too.\n\nI used the following bits of tech to make this possible:\n\n - [Node.js](https://nodejs.org/en)\n - [LangChain.js](https://github.com/langchain-ai/langchainjs)\n - [node-llama-cpp](https://withcatai.github.io/node-llama-cpp/)\n - [Transformers.js](https://xenova.github.io/transformers.js/)\n - [Express](https://expressjs.com/)\n - [Lit](https://lit.dev/)\n - [tailwindcss](https://tailwindcss.com/)\n - [Vite](https://vitejs.dev/)\n\nBut most importantly, I used [Redis](https://redis.io/docs/about/about-stack/) and [Node Redis](https://github.com/redis/node-redis).\n\n\n## Get Started\n\nIf you're a Docker user, once you've cloned the repo you can get up and running quickly with:\n\n```bash\ndocker compose up\n```\n\nThis will launch four containers:\n\n 1. An instance of Redis that's preloaded with a bunch of Bigfoot sightings and exposed on port 6379.\n 2. An API written in Express with endpoints to search, fetch, and add Bigfoot sightings.\n 3. A web app written with Lit and tailwindcss that lets you search and view Bigfoot sightings.\n 4. An nginx gateway that binds and exposes the API and web app on port 8080.\n\n\u003e Note: If you have Redis running on your local machine, the first container won't be able to start. So, shut down existing versions of Redis before you start.\n\nOnce you've done that, you should be able to hit the app by browsing to http://localhost:8080. Search for some sightings. Click on ones that interest you. The search is semantic, so you can search for concepts instead of specific words. Some of my favorite searches include:\n\n - near a body of water at night\n - involving alcohol\n - found a footprint\n - during winter\n - in the desert\n\n\n## If You're Not Using Docker\n\nGetting up and running without Docker is totally possible, but a bit more work. You'll need to make sure you have the following installed before starting:\n\n - A recent version of [Node.js](https://nodejs.org/en). I used version 18, but any version that supports top-level await should work.\n - The latest version of [Redis](https://redis.io/docs/install/install-stack/). Follow the instructions in the link to get it installed *and* running.\n\nYou can confirm that Redis is running by using [RedisInsight](https://redis.io/docs/install/install-redisinsight/) to connect to it.\n\n\n### Install Packages\n\nNow we need to install the packages that the API and the web app need. For the API, run the following in the `api` folder:\n\n```bash\nnpm install\n```\n\nAnd likewise, for the web app, run this in the `web` folder:\n\n```bash\nnpm install\n```\n\nSensing a theme?\n\n\n### Start Services\n\nEverything is installed. Now we can start the API and web app services, each in their own terminal. Start with the API from the `api` folder:\n\n```bash\nnpm start\n```\n\nAnd then with the web app in the `web` folder:\n\n```bash\nnpm run dev\n```\n\nGo ahead and test the service by going to the URL that `npm run dev` provides. Search for some sightings and you'll immediately notice something—there aren't any. This is because when you run without Docker and install your own Redis, you won't have any sightings preloaded. I used Docker to load all the sightings into Redis and since you're not using Docker, you'll need to load some on your own. Which is a great segue to the next topic...\n\n\n## Loading More Sightings\n\nThe API provides an endpoint to load a Bigfoot sighting. You can try it out by posting a JSON document to the API using curl:\n\n```bash\ncurl -X POST http://localhost:8080/api/load \\\n --silent \\\n --header \"Content-Type: application/json\" \\\n --data '{ \"id\": \"65535\", \"observed\": \"I saw Bigfoot at Walmart buying flip-flops. Apparently, he wears a size 17.\", \"county\": \"Athens\", \"state\": \"Ohio\", \"classification\": \"Class A\", \"timestamp\": 205286400 }'\n```\n\n\u003e Note: If you're not using Docker, change the port on the URL to 3000.\n\nWhat this endpoint does is two-fold:\n\n 1. It adds all the data you provided it to a Hash in Redis.\n 2. It adds the `id` and text in the `observed` property to an event stream so that it can be summarized and embedded by another service.\n\nYou can go into Redis Insight and look at this yourself to confirm that it has worked. It will be in the key `bigfoot:sighting:65535`. And, if you look in the stream `bigfoot:sighting:report` you should see it at the top of the stream.\n\nYou can also use the web application to see by browsing to http://localhost:8080/sighting/65535 (or the comparable URL for the web app service if you're not running Docker). Note that there isn't a summary in the results like there is for other sightings. This is because the summary hasn't been created yet. And this is because the embedder service isn't running.\n\nSo, let's launch the embedder service. This will summarize and embed what it reads from the event stream and then update the Hash with that summarization and embedding.\n\n\n### Setting Up and Launching the Embedder\n\nTo run the embedder, you'll need to download the model it makes use of. Normally, I'd have put it in the repo, but the model file is way big and won't fit in GitHub—even if I use Git LFS. So, you need to download it at:\n\nhttps://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/blob/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf\n\nNote that at a little over 4 gigs, this ain't a small file. The download might take a while, but it's easy enough to install. Once you've downloaded it, just copy it to `embedders\\models`.\n\nThe rest is easy as it's just another Node.js app. So, from the `embedder` folder install all the packages:\n\n```bash\nnpm install\n```\n\nAnd then launch the service:\n\n```bash\nnpm start\n```\n\n\u003e Note: If you are using an Intel-based Mac like me, you might need to change how Metal is used. Details are [here](https://withcatai.github.io/node-llama-cpp/guide/Metal) but the magic command is:\n\u003e\n\u003e ```bash\n\u003e npx --no node-llama-cpp download --no-metal\n\u003e ```\n\nSummarization can take a while depending on the computer you are using. On my older Intel-based Mac, it takes between 30-90 seconds to summarize the text. So be patient.\n\nIf you are adding lots of sightings, you can run several embedders in parallel. Just open more terminal windows and run `npm start` in each of them. I typically run 4 of them on my system, which has 32 gigs of RAM.\n\n\n## Loading Lots of Sightings\n\nIf you're not using Docker, you'll need to load all of the Bigfoot sightings. All 4,586 of them. This would be tedious, but fortunately there's another tool in the `loader` folder. You can point this tool at an endpoint (i.e. http://localhost:3000/api/load) and a JSONL file, and it will call that endpoint for each entry in that file.\n\nIt's a Node.js app, like everything here, so you need to install the packages before you can use it:\n\n```bash\nnpm install\n```\n\nOnce there, you can run it. Let's get the help:\n\n```bash\nnpm start -- --help\n```\n\n```bash\nSynopsis\n\n $ npm start -- file [--startingId id] [--endpoint url]\n $ npm start -- --help\n\nOptions\n\n --endpoint url The URL of the API endpoint to post sightings to. Optional.\n Defaults to http://localhost:8080/api/load.\n --startingId id Optional. Skip entries in file until this ID is found.\n --help string Print this usage guide.\n```\n\nConveniently, the `data` folder has a JSONL file with all 4,586 Bigfoot sightings in it. So, to load them, just run the following command:\n\n```bash\nnpm start -- ../data/bfro_reports_geocoded.jsonl --endpoint http://localhost:3000/api/load\n```\n\nThis'll run super fast as it's just adding each sighting to a Hash and an event stream. From there, the embedder(s) will take over.\n\n\n## Next Steps\n\nThere's a lot that could be expanded in this app. Check out the code and see what you can do with it. Here's a couple of ideas:\n\n - The search endpoint can take discrete parameters like `state` or `classification` that enable filtered search. But there are only a few fields implemented and the UI doesn't do anything with them. Add some fields and add a UI to support it.\n - The UI doesn't provide a way to add a Bigfoot sighting directly. It probably should.\n - There's no mechanism to *edit* a Bigfoot sightings. You could add endpoints and the UI to do this. Keep in mind that if you change the `observed` field, you'll need to update to embedding and summary as well. You'll probably need to drop an event on a stream.\n\n\n## Contributing\n\nIf you found this useful, please share it. If you find a bug, a typo, or whatever, send me a pull request and I'll get it merged. If you add more functionality, keep that for yourself. We want others to be able to learn from this codebase as well.\n\nThanks and good luck in your hunt for Bigfoot.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredis-developer%2Ffinding-bigfoot-with-semantic-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredis-developer%2Ffinding-bigfoot-with-semantic-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredis-developer%2Ffinding-bigfoot-with-semantic-search/lists"}