https://github.com/moritztng/fltr

Like grep but for natural language questions. Based on Mistral 7B or Mixtral 8x7B.
https://github.com/moritztng/fltr

cli grep grep-like llama llama-2 llm localllama mistral mixtral mixtral-8x7b operating-system rust

Last synced: 3 months ago
JSON representation

Like grep but for natural language questions. Based on Mistral 7B or Mixtral 8x7B.

• Host: GitHub
• URL: https://github.com/moritztng/fltr
• Owner: moritztng
• License: mit
• Created: 2023-12-06T01:03:02.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-03-13T11:39:01.000Z (about 1 year ago)
• Last Synced: 2024-10-18T23:13:39.109Z (7 months ago)
• Topics: cli, grep, grep-like, llama, llama-2, llm, localllama, mistral, mixtral, mixtral-8x7b, operating-system, rust
• Language: Rust
• Homepage:
• Size: 63.5 MB
• Stars: 374
• Watchers: 6
• Forks: 7
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
Like grep but for natural language questions. Based on Mistral 7B or Mixtral 8x7B.

Text file input tokens per second:

- Nvidia RTX 3070 with 8GB memory

    - Mistral 7B: **~52 tok/s**

    - Mixtral 8x7B: **~28 tok/s**

- Intel I5-6500 with 8GB memory

    - Mistral 7B: **~5 tok/s**

    - Mixtral 8x7B: **~2 tok/s**

# Installation

## Linux (x86_64) & macOS (x86_64 & arm64)

If nvidia driver that supports cuda 12.1 exists, it installs cuda version, else cpu version. Replace `small` with `large` to install Mixtral 8x7B. It's ~7GB or ~48GB.

```bash

curl https://raw.githubusercontent.com/moritztng/fltr/main/install.sh -o install.sh && bash install.sh small && export PATH=$PATH:~/Fltr

```

# Quickstart

Add `--large` for Mixtral 8x7B.

```bash

fltr --file emails.txt --prompt "Is the following email spam? Email:" --batch-size 32

```

It will output all lines in the file where the answer is yes.