Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/gali1/ollama-cli-or-webui
This project provides a dual-interface tool for generating text responses using large language models.
https://github.com/gali1/ollama-cli-or-webui
cli command-line flask huggingface huggingface-transformers interface interfaces llama llm llm-inference natural-language-processing nlp nlp-parsing rest-api service text-generation transformers transformers-models web
Last synced: 12 days ago
JSON representation
This project provides a dual-interface tool for generating text responses using large language models.
- Host: GitHub
- URL: https://github.com/gali1/ollama-cli-or-webui
- Owner: gali1
- License: mit
- Created: 2024-08-04T08:03:42.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-09-02T19:06:14.000Z (4 months ago)
- Last Synced: 2024-11-12T00:18:32.614Z (about 2 months ago)
- Topics: cli, command-line, flask, huggingface, huggingface-transformers, interface, interfaces, llama, llm, llm-inference, natural-language-processing, nlp, nlp-parsing, rest-api, service, text-generation, transformers, transformers-models, web
- Language: Python
- Homepage:
- Size: 80.1 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
```markdown
# Use LLM Using CLI or WEB-UI
## Overview
This project provides two methods to generate text responses:
- A Flask-based web service.
- A CLI-based (non-Flask) application.
Both methods can use either the Transformers library with
Hugging Face models or Llama (via `llama_cpp`).
## Prerequisites
1. **Python 3.7 or higher**: Ensure that you have Python installed.
Download it from [python.org](https://www.python.org/downloads/).
2. **Virtual Environment (recommended)**: It's recommended to create a
virtual environment to manage dependencies.
3. **API Keys/Access**: Ensure you have the necessary
API keys or access tokens for the Llama model if using it. (Optional)
## Setup
### 1. Clone the Repository
```bash
git clone https://github.com/gali1/ollama-cli-or-webui.git
cd ollama-cli-or-webui
```
### 2. Create and Activate a Virtual Environment
```bash
python -m venv venv
source venv/bin/activate # On Windows use: venv\Scripts\activate
```
### 3. Install Dependencies
Create a `requirements.txt` file or use the following command to install the necessary packages:
```bash
pip install flask python-dotenv transformers torch llama-cpp-python
```
### 4. Configure Environment Variables
Create a `.env` file in the project root directory and set the required environment variables. Example `.env` file:
```
MODEL_NAME=gpt2
LLAMA_MODEL_PATH=/path/to/llama/model
MAX_TOKENS=256
TEMPERATURE=0.7
TOP_P=0.95
TOP_K=50
REPETITION_PENALTY=1.2
NO_REPEAT_NGRAM_SIZE=4
NUM_BEAMS=1
BATCH_SIZE=128
```
**Note:** Adjust `LLAMA_MODEL_PATH` to point to your Llama model. If using the Transformers model, `LLAMA_MODEL_PATH` is not needed.
### 5. Ensure Model and Tokenizer Files
- **Transformers**: Ensure you have internet access or the models are available locally.
- **Llama**: Ensure the model file is available at the specified path.
## Running the Application
### 1. Launch the Flask Application
To start the Flask server, use:
```bash
python llama-direct-llm.py
```
Replace `llama-direct-llm.py` with the appropriate script name if necessary.
### 2. Access the CLI Application
To interact via CLI, run:
```bash
python llama-direct-llm-c.py
```
**Example CLI Usage:**
```bash
> provide java code that generates 10 random numbers.
```
**Example Response:**
```bash
import java.util.Random;
public class Main {
public static void main(String[] args) {
Random rand = new Random();
for (int i = 0; i < 10; i++) {
int randomNum = rand.nextInt((100 - 1) + 1) + 1;
System.out.println("Random number " + (i + 1) + " : " + randomNum);
}
}
}
// Here we are creating a Java program that generates 10 random numbers between 1 and 100.
// The "rand.nextInt((100 - 1) + 1) + 1" part of the code is used to generate the random numbers.
// The numbers generated are from the range 1 to 100 (inclusive).
// "nextInt((100 - 1) + 1)" generates a random number between 0 (inclusive) to 100 (inclusive). Adding 1 shifts that range to 1 to 100.
```
### 3. Access the Web Service
By default, the Flask app will run on `http://127.0.0.1:9898`. You can send a POST request to the `/generate` endpoint with a JSON payload:
**Example Request:**
```bash
curl -X POST http://127.0.0.1:9898/generate -H "Content-Type: application/json" -d '{"prompt": "Tell me a joke."}'
```
**Example Response:**
```json
{
"response": "Here is a joke: Why did the scarecrow win an award? Because he was outstanding in his field!"
}
```
## Troubleshooting
- **Missing Environment Variables**: Ensure all required environment variables are set in the `.env` file.
- **Model Loading Issues**: Verify model paths and configurations are correct.
- **Dependency Issues**: Ensure all dependencies are installed and compatible with your Python version.
---
Feel free to reach out if you have any questions or run into issues. Happy coding!
---