https://github.com/yobibyte/compressor

Because we don't have enough time to read everything
https://github.com/yobibyte/compressor

Last synced: 9 days ago
JSON representation

Because we don't have enough time to read everything

• Host: GitHub
• URL: https://github.com/yobibyte/compressor
• Owner: yobibyte
• License: mit
• Created: 2023-11-29T21:40:38.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2024-09-07T05:46:59.000Z (2 months ago)
• Last Synced: 2024-09-07T06:44:38.102Z (2 months ago)
• Language: Python
• Size: 298 KB
• Stars: 83
• Watchers: 4
• Forks: 10
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Compressor

Because we do not have time to read everything.

**Daily Arxiv summaries**: [https://yobibyte.github.io/arxiv_compressed.html](https://yobibyte.github.io/arxiv_compressed.html)

![](compressor.jpg)

Compressor is an LLM-based scientific literature / talks summarisation project started by [yobibyte](https://twitter.com/y0b1byte).

It is heavily relying on [llama.cpp](https://github.com/ggerganov/llama.cpp) and [HuggingFace](https://huggingface.co/) models.

Compressor is under active development, you are entering unchartered waters when using it.

I will be happy to any feedback / feature requests, and, please, send PRs.

## Usecases

1. Get arxiv link, summarise.

2. Get all papers submitted to Arxiv at a date (usually published today). Summarise each.

3. Get a pdf, summarise. Not yet implemented.

4. Get an audio of a talk, get a script, summarise. WIP.

5. Summarise all papers accepted to some conference on OpenReview. 

6. Summarise all talks of a particular conference. Future plans.

## Architecture

Crawler -> Compressor -> Reporter

## Big TODOs

* Current version does summarisation based on abstracts only. Add full-text support.

* Better exception handling. Right now, postprocessing LLM outputs might fail from time to time requiring rerunning the compressor.