https://github.com/ksm26/video-analysis-agent

Last synced: about 2 months ago
JSON representation

Host: GitHub
URL: https://github.com/ksm26/video-analysis-agent
Owner: ksm26
Created: 2025-06-29T11:39:49.000Z (3 months ago)
Default Branch: main
Last Pushed: 2025-06-29T17:35:33.000Z (3 months ago)
Last Synced: 2025-06-29T18:28:08.377Z (3 months ago)
Language: Python
Size: 1.01 MB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# 🎥 Hercules Video Analysis Agent

## 📋 Project Overview

This project implements an automated Video Analysis Agent for Hercules test runs.

The agent evaluates whether the test run was executed as planned by comparing:

✅ The agent's Planning Log (thoughts/steps)
✅ Video recording(s) of the run
✅ The final test output

**Deviation reports** are generated indicating if any claimed action was skipped, altered, or missing in the video evidence.

---

## ⚙️ Features

- Modular, scalable Python codebase
- Step-by-step video inspection with YOLOv8-based action detection
- Final test output validation
- Lightweight AI assistance using `flan-t5-small` (runs on low-resource machines)
- Generates `.txt` and `.html` deviation reports
- Fully configurable via `configs/settings.py`

---

## 🗂️ Project Structure

video_analysis_agent/

├── agent\
│   └── base_agent.py\
├── config\
│   ├── __init__.py\
│   └── settings.py\
├── data\
│   ├── planning_logs\
│   │   └── run1.txt\
│   ├── test_outputs\
│   │   └── run1_output.txt\
│   └── videos\
│   └── run1.mp4\
├── models\
│   └── yolov8s.pt\
├── reports\
├── requirements.txt\
├── run_agent.py\
├── run_agent_langchain.py\
├── src\
│   ├── deviation_engine.py\
│   ├── __init__.py\
│   ├── input_handler.py\
│   ├── output_checker.py\
│   ├── planning_parser.py\
│   ├── report_generator.py\
│   └── video_analyzer.py\
└── tools\
│   ├── ai_tools.py\
│ ├── __init__.py

---
## 🚀 How to Run the Agent

### 1️⃣ Setup

Install requirements:

```bash
pip install -r requirements.txt
```

### 2️⃣ Place Input Files
Videos → `data/videos/`

Planning Logs (`.txt`) → `data/planning_logs/`

Final Output Files → `data/test_outputs/`

Ensure filenames align (e.g., `run1.mp4`, `run1.txt`, `run1_output.txt`).

### 3️⃣ Run the Agent

```bash
python run_agent.py
```

Reports are generated in `reports/` with timestamps.

Example:

`reports/run1_detailed_report_20240628_153020.txt` \
`reports/run1_detailed_report_20240628_153020.html`

### 📊 Outlines
`Video → Frames → YOLO Detections → AI Agent` \
` AI Agent: `\
` - Uses LLM to parse Planning Log (extract steps)` \
` - Matches steps to YOLO results (Was action observed?) `\
` - Flags deviations or missing actions` \
` - Generates final report with reasoning`

### 📊 Sample Output

```bash
Test Report Details
Test Suite: run1
==================================================
Total Duration: 87.63 sec
Total Token Used: 43
Total Cost Estimate: 9e-05 USD

Test Result Summary
--------------------------------------------------
Steps Passed: 0
Steps Failed: 3

Detailed Steps:
--------------------------------------------------
Step 1: Click "Login"
Result: ❌ Deviation
Notes: Action not found in video

Step 2: Enter Password
Result: ❌ Deviation
Notes: Action not found in video

Step 3: Submit Form
Result: ❌ Deviation
Notes: Action not found in video

Output File: data/test_outputs/run1_output.txt
Proofs Video: data/videos/run1.mp4
Planner Thoughts Log: ./log_files/run1_planner_thoughts.log
Chat Messages Log: ./log_files/run1_chat_messages.log
```

### 🛠 Requirements
- Python 3.8+
- Tested with flan-t5-small for AI tasks
- Uses YOLOv8 for action detection
- Low hardware requirements (8GB RAM compatible)

### 📚 References
- [Hercules GitHub](https://github.com/test-zeus-ai/testzeus-hercules)

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/ksm26/video-analysis-agent

Awesome Lists containing this project

README