{"id":21250702,"url":"https://github.com/mhw32/seeclickbuy","last_synced_at":"2025-05-10T21:13:03.603Z","repository":{"id":263202718,"uuid":"888715527","full_name":"mhw32/seeclickbuy","owner":"mhw32","description":"🌟 Purchasing as easy as see, click, buy - Chrome extension for online shopping","archived":false,"fork":false,"pushed_at":"2024-11-17T19:41:53.000Z","size":10192,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-31T21:42:45.334Z","etag":null,"topics":["ai","chrome-extension","hackathon","honey","llm","sam2","vision"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mhw32.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-14T22:02:25.000Z","updated_at":"2025-02-21T13:15:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"4827d396-5cc1-4269-b651-9f89ef207f3f","html_url":"https://github.com/mhw32/seeclickbuy","commit_stats":null,"previous_names":["mhw32/seeclickbuy"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mhw32%2Fseeclickbuy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mhw32%2Fseeclickbuy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mhw32%2Fseeclickbuy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mhw32%2Fseeclickbuy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mhw32","download_url":"https://codeload.github.com/mhw32/seeclickbuy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253313286,"owners_count":21888801,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","chrome-extension","hackathon","honey","llm","sam2","vision"],"created_at":"2024-11-21T03:38:29.306Z","updated_at":"2025-05-09T19:40:19.120Z","avatar_url":"https://github.com/mhw32.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SeeClickBuy\n\n**This project was awarded third place at the [Sushi Hackathon 2024](https://sushihackathon.com). Here's the [deck](https://docsend.com/view/vp8tgjwfptiattyn) we pitched to the judges.** \n\n*Trouble finding the perfect item? HATE ADS!?* Buying stuff online shouldn't be so hard.\n\nSeeClickBuy is a chrome extension to buy anything you see on the internet. It allows you to click on images, videos, and more and find the best deal for them. If you want to change the color of the item, or tweak the style, you can chat with an AI bot to generate modified search results. \n\n| ![GIF 1](./assets/clip1.gif) | ![GIF 2](./assets/clip2.gif) | ![GIF 3](./assets/clip3.gif) |\n|----------------------------------------|----------------------------------------|----------------------------------------|\n\nBuilt on top of Meta's Segment-Anything-2 (SAM2) model, Google Lens, and the OpenAI API. SAM2 is used to segment a clicked image and remove background pixels. Google lens can visually find similar images across online marketplaces and return product metadata. We call an LLM to summarize product metadata into a single description for the clicked object. Users can chat with the extension to make edits to this description (e.g. change color or brand). This performs a new call to the Google Shopping API using the updated description.\n\n## Setup\n\nThese instructions are for a linux machine with a GPU e.g. a lambda or runpod machine. You may need to customize this setup for your own machine. \n\n### Download models\n\nDownload a model checkpoint [sam2.1_hiera_large.pt](https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt) and put it in the `ai/checkpoints/` folder.\n\n### Firebase \n\nYou will need a Firebase project, which saves all the clicks and chats users make with the extension. You can create a free Firebase project [here](https://firebase.google.com/) in the console. Enable firebase storage within the project. \n\nOnce you have a project, create a service account in the console and download the `service_account.json` file. Rename it and place it in `server/server/seeclickbuy.json` (see line 5 of `database.py`). Replace line 6 of `database.py` with the bucket name for your firebase storage. \n\n### Frontend Installation\n\n(I am using `node v19.7.0` through `nvm`.) \n\nWe use a chrome extension boilerplate in `odin/`. Most of the files will not be relevant to this project. We recommend paying attention to `pages/popup`, `pages/content-runtime`, and `packages/shared/lib/hooks`.\n\nFrom inside the `odin/` directory, run \n```bash\nnpm install pnpm\n```\nThen run \n```bash\npnpm install\n```\nwhich should create a `node_modules` folder among other things.  \n\nIt is also worth noting line 1 of `shared/lib/hooks/api.ts` which sets the base URL for the web server.\n```\nconst BASE_URL = \"http://localhost:8000\";\n```\nYou will need to change this if you deploy your web server to a remote server e.g. `ngrok`.\n\n### Backend Installation\n\nRun the following to install redis, ngrok, git, and other basic essentials.\n```bash\nbash setup.sh\n```\nRun the following to install a conda environment with PyTorch, and also clone the segment-anything-2 repo. This requires a GPU. \n```bash\nbash install.sh\n```\nThis command will already install requirements for the `ai/` and `server/` folders.\n\n### Environment Variables\n\nSee `server/server/env.template`. Create a file in the same directory called `.env`. You will need a [SERP API key](https://serpapi.com) for google lens queries and an [OpenAI API key](https://platform.openai.com/api-keys) for LLM calls. Add those to `.env` file:\n```bash\nBROKER_URL=redis://localhost:6379/0\nCELERY_BROKER_URL=redis://localhost:6379/0\nCELERY_RESULT_BACKEND=redis://localhost:6379/0\nSERP_API_KEY=...\nOPENAI_API_KEY=...\n```\n\n## Usage\n\nThere are a three components: the chrome extension, the web server, and the worker server. \n\n### Web server\n\nTo start the web server, run \n```bash\nbash start_server.sh\n```\nfrom the `server/` directory. \n\nThis will load a FastAPI server at `localhost:8000`. See `http://localhost:8000/docs` for the API docs. \nOnce you can run this, you can rest assured that you have all the dependencies set up correctly. \n\n### Worker server\n\nTo run inference for user requests, we use a Celery server with workers that can asynchronously pick up tasks from a Redis queue. \n\nTo start the worker server, run \n```bash\nbash start_celery.sh\n```\nfrom the `server/` directory. By default, we just run one worker. You may change this by editing the `--concurrency` flag. \n\n### Chrome extension\n\nTo start the chrome extension in development mode, run\n```bash\npnpm dev\n```\nfrom the `odin/` directory. This will create a `dist/` folder with a build of the extension. \n\nTo load the extension in chrome, go to `chrome://extensions/`, enable \"Developer mode\", click \"Load unpacked\", and select the `dist/` folder. There are many walkthroughs online for this part of the process. \n\nIf you want to build the extension for production, run \n```bash\npnpm build\n```\nfrom the `odin/` directory. This will create a `build/` folder with a build of the extension. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmhw32%2Fseeclickbuy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmhw32%2Fseeclickbuy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmhw32%2Fseeclickbuy/lists"}