https://github.com/cs194group/multi-agent-misconceptions
A multi-agent system predicting student misconceptions using the Exchange-of-Thought (EoT) framework, utilizing dspy and OpenAI to process questions, generate misconceptions, and evaluate semantic similarities.
https://github.com/cs194group/multi-agent-misconceptions
dspy eot machine-learning multi-agent-systems openai-api
Last synced: 6 days ago
JSON representation
A multi-agent system predicting student misconceptions using the Exchange-of-Thought (EoT) framework, utilizing dspy and OpenAI to process questions, generate misconceptions, and evaluate semantic similarities.
- Host: GitHub
- URL: https://github.com/cs194group/multi-agent-misconceptions
- Owner: CS194Group
- Created: 2024-11-22T00:34:12.000Z (6 months ago)
- Default Branch: final
- Last Pushed: 2024-12-20T06:34:11.000Z (5 months ago)
- Last Synced: 2025-05-06T21:08:34.198Z (13 days ago)
- Topics: dspy, eot, machine-learning, multi-agent-systems, openai-api
- Language: Jupyter Notebook
- Homepage: https://cs194group-misconception-webpage-home-ff9be8.streamlit.app
- Size: 18.3 MB
- Stars: 4
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Multi-Agent Misconception Analysis System
A multi-agent system designed to enhance learning environments by predicting student misconceptions based on their answers to questions. This project uses **dspy** and **OpenAI** to process questions, generate potential misconceptions, and evaluate semantic similarities between these misconceptions, facilitating a deeper understanding of student reasoning.
## Features
- **Multi-Agent Communication**: Implements the Exchange-of-Thought (EoT) framework inspired by large language model communication paradigms like Memory, Report, Relay, and Debate.
- **Misconception Prediction**: Identifies the reasoning behind incorrect answers provided by students.
- **Semantic Evaluation**: Analyzes and ranks misconceptions based on their similarity to student reasoning.
## Getting Started
### Prerequisites
- Python 3.8+
- `dspy` library
- OpenAI API key
### Installation
1. Clone the repository:
```bash
git clone https://github.com/CS194Group/multi-agent-misconceptions.git
cd multi-agent-misconceptions
```
2. Install required dependencies:
```bash
pip install -r requirements.txt
```
### Usage
#### Training and Evaluation
Run the main script to process questions and evaluate misconceptions:
`bash
python main.py
`
#### Data
Place your training, testing, and validation datasets in the `data/` directory.
### Project Structure
- `agents/`: Contains multi-agent logic for generating and evaluating misconceptions.
- `data/`: Includes training, testing, and validation datasets.
- `dataloader.py`: Preprocessing and data loading utilities.
- `evaluation.py`: Performance evaluation and reporting tools.
- `main.py`: Entry point for training and inference.
- `predictmode.py`: Prediction logic for generating misconceptions.
### Structure of .env File
```bash
LAMBDA_API_KEY=""
LAMBDA_API_BASE="https://api.lambdalabs.com/v1/"
LAMBDA_API_MODEL="llama3.1-70b-instruct-berkeley"
OPENAI_API_KEY=""
WANDB_API_KEY=""
```