https://github.com/paschembri/reading-notes

AI Assistant that runs locally and provides reading notes for a given folder
https://github.com/paschembri/reading-notes

Last synced: 4 days ago
JSON representation

AI Assistant that runs locally and provides reading notes for a given folder

• Host: GitHub
• URL: https://github.com/paschembri/reading-notes
• Owner: paschembri
• License: apache-2.0
• Created: 2023-10-04T13:57:29.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-10-12T08:01:06.000Z (over 1 year ago)
• Last Synced: 2024-11-22T19:21:06.363Z (2 months ago)
• Language: Python
• Size: 138 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Reading Notes AI

AI Assistant that runs locally and provides reading notes for a given folder

## What's this ?

This is an experiment to show how an AI-augmented knowledge management system can provide insights and information about a large document collection.

Reading Notes AI uses the recently released model from Mistral AI and runs locally (i.e. offline).

## Disclaimer

This is still experimental: hallucinations are to be expected.

## How does it work ?

I use the unstructured library to convert any supported document into text (it even works on images, see the `assets` directory), then I use Mistral AI Instruct to generate a summary with PyLLMCore:

```python

from llm_core.assistants import Summarizer, LLaMACPPAssistant

...

summarizer = Summarizer(

    model=args.model_name, assistant_cls=LLaMACPPAssistant

)

...

summarizer.fast_summarize(content)

```

## Why is it slow ?

Running this script on a M1 MacBook Pro can go as fast as 13 tokens / sec (for token sampling) - that is approx 1 min 30s per page generated.

As an example processing this repo took 2min24sec.

## Launching the analysis on a folder

1. Install:

```shell

python3 -m venv venv

venv/bin/python3 -m pip install -r requirements.txt

# If you have a M1/M2 Apple Silicon

CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64" venv/bin/python3 -m install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python

```

2. Launch

```shell

venv/bin/python3 app.py /path/to/a/directory/containing/any/documents

```

3. Wait (it will take a while)

Go look for the file reading_notes.txt

A sample result from running this code on the repository itself is in reading_notes.txt