https://github.com/13point5/aggregated-chat-analysis
Aggregating chat analysis with AI tools
https://github.com/13point5/aggregated-chat-analysis
llm openai python
Last synced: 3 months ago
JSON representation
Aggregating chat analysis with AI tools
- Host: GitHub
- URL: https://github.com/13point5/aggregated-chat-analysis
- Owner: 13point5
- License: mit
- Created: 2024-04-14T03:38:59.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-14T08:37:25.000Z (over 2 years ago)
- Last Synced: 2025-03-14T03:27:27.605Z (over 1 year ago)
- Topics: llm, openai, python
- Language: Jupyter Notebook
- Homepage:
- Size: 364 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Aggregated Chat Analysis - Essay Feedback
As students use platforms like Khanmigo and MagicSchool.ai for tasks like getting feedback on their essays and "chat" with
the tool to improve their learning, it will quickly become difficult for teachers to keep up with the volume of chat data.
This project aims to experiment with methods to aggregate chat data to provide teachers with data to improve instruction.
For the first experiment we will focus on a tool that gives feedback on essays.
## Data
I used ChatGPT to generate the student essays and follow-up chat data. I used MagicSchool.ai's tool for providing feedback on essays for the AI messages.
## Architecture
I've used a simple `Evaluation` data model to analyse **ONLY** the AI messages. I know that this approach has drawbacks but I want to keep it simple for now to see if it's remotely useful.
### Evaluation Data Model
For each of the criteria below, we extract the following data:
- `strengths`
- `weaknesses`
- `suggestions`
**Criteria**:
1. **Introduction**
- Clarity of thesis statement
- Engagement and relevance of opening statements
2. **Structure**
- Organization and clarity of paragraphs
- Logical flow of ideas
3. **Argumentation**
- Strength and clarity of arguments
- Use of critical reasoning
4. **Evidence**
- Relevance and quality of evidence
- Use of citations and references
5. **Conclusion**
- Restatement of thesis
- Summary of main points
- Closing statements
## Setup
```bash
poetry install
```
## Usage
The only experiment I have is in `exp1.ipynb` which is self-explanatory.