Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/gitgoap/ai-snipping-tool
Custom Screenshot with tex extraction | Chat with Gemini
https://github.com/gitgoap/ai-snipping-tool
Last synced: 6 days ago
JSON representation
Custom Screenshot with tex extraction | Chat with Gemini
- Host: GitHub
- URL: https://github.com/gitgoap/ai-snipping-tool
- Owner: gitgoap
- License: mpl-2.0
- Created: 2024-05-10T18:42:36.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2025-01-06T15:16:30.000Z (about 1 month ago)
- Last Synced: 2025-01-31T10:08:27.349Z (13 days ago)
- Language: JavaScript
- Homepage:
- Size: 24.1 MB
- Stars: 34
- Watchers: 1
- Forks: 48
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Overview
## ``` AI Snipping Tool ```
A **Chrome extension** that takes custom screenshots, **extracts text**, and allows users to **ask questions** from the extracted content.
### Features of AI Snipping Tool
1. **Custom Screenshots**: Capture screenshots from Chrome.
2. **Text Extraction**: Extract text from screenshots using Tesseract OCR.
3. **Ask Questions**: Use Google Gemini API to ask questions based on extracted text.
4. **User Preferences**: Configure language and accuracy settings for OCR.
5. **Interactive OCR Results**: View and interact with extracted text by copying it, saving it, or saving the screenshot.
## Chrome Extension popup
`Step 1:` Take the screenshot
`Step 2:` Copy the *obtained text* or proceed further
`Step 3:` Fill *Google Gemini* API key and ask related *question*
Prerequisite: `Google Gemini API`
## Steps to get Gemini API key
- Sign up at `https://ai.google.dev/aistudio`
- Click on `Get API key`
## Built With
👉 Tech Stacks:
## Getting Started 👩💻
> ⚠️Prerequisites
>
> - Recommended to have the latest version of Google Chrome
> - Make sure to switch on Chrome's `developer mode`
To get a local copy up and running follow these simple steps
## Installation Guide
https://github.com/user-attachments/assets/a8769d70-4d1e-4b93-a2f5-7692ab313b3f
1. Open Git Bash and change directory
``` sh
cd path
```
3. Clone the repo
```sh
git clone https://github.com/gitgoap/AI-Snipping-Tool.git
```
3. Open Google Chrome
4. Click **3 dots** at top right corner
5. Go to Extensions> Manage Extensions
6. Click **Load Unpack** and select the repo where you saved initially while cloning
## Contribution Guide
- Make sure to raise an issue before raising a Pull Request.
- Get the issue assiged.
- Mention the issue number (`Eg: #4`) while raising a Pull Request in the description.
## Star History
## Problem Statement
We often need to type out text from videos, images, or thumbnails on websites. This can be a tedious and error-prone process, especially if the text is long or has tricky stuff like website links, technical words, code examples, or math equations. This issue comes up on YouTube, where useful info is shown in videos or thumbnail images.
## Some of the scenarios where this problem arises:
- Manually typing the displayed link is inefficient and prone to errors.
- Copying the references given in the presentation footer is difficult
- Copying code directly to the editor is not possible
- Copying text from documents shared as images on **social media** is tricky
- People may require this text to feed into LLMs to get a `summary`
# Working
## Our Chrome extension is utilizing **Tesseract OCR** for character recognition in the image:
1. The extension's background script ```worker.js``` listens for user actions, such as clicking the extension's icon or using the context menu.
2. When the user initiates a screenshot, the extension captures the visible tab using ```chrome.tabs.captureVisibleTab``` and gets the screenshot data as a PNG image.
3. The extension then injects several scripts (```helper.js```, ```response.js```, ```elements.js```, and ```custom-elements.min.js```) into the current tab's context. These scripts are responsible for rendering the OCR result and handling the OCR process.
4. The ocr-result custom element, defined in ```elements.js```, is used to perform the OCR operation. This custom element utilizes the ```tesseract.js``` library, which is a pure **JavaScript port** of the **Tesseract OCR** engine.
5. The ocr-result element is configured with user preferences such as language, accuracy, and other settings fetched from the Chrome extension's local storage.
6. The captured screenshot data (PNG image) is passed to the **ocr-result element**, which then runs the **Tesseract OCR** engine on the image to extract the text.
7. The extracted text is rendered within the **ocr-result element**, allowing the user to view and interact with the OCR results.
> **Tesseract OCR**: It is an open-source optical character recognition (OCR) engine developed by Google. It is designed to extract text from images and documents without a text layer, outputting the document in various formats such as plain text, HTML, PDF, and more. **Tesseract** supports recognition of over **100 languages** "out of the box" and is highly customizable.
# Future Scope / Features
- [ ] **YouTube Video Screenshot:** Capture text in the current YouTube video frame with just one click of a popup button.
- [ ] **Answers questions just with screenshots:** This feature is possible by utilizing the multimodal feature of Google Gemini.
Our Contributors ❤️
Thank you for contributing to our repository.
