https://github.com/breadrock1/watchtower
Watchtower is adapter-service for cloud or fs for file processing using LLM.
https://github.com/breadrock1/watchtower
async docsearch elasticsearch fsnotify go golang load-files search
Last synced: about 1 month ago
JSON representation
Watchtower is adapter-service for cloud or fs for file processing using LLM.
- Host: GitHub
- URL: https://github.com/breadrock1/watchtower
- Owner: breadrock1
- Created: 2024-01-18T16:58:05.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2026-03-31T16:50:00.000Z (3 months ago)
- Last Synced: 2026-03-31T17:37:41.554Z (3 months ago)
- Topics: async, docsearch, elasticsearch, fsnotify, go, golang, load-files, search
- Language: Go
- Homepage:
- Size: 840 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Watchtower Metaverse project
[](https://github.com/breadrock1/watchtower/actions/workflows/pull-request.yml)
[](https://www.linux.org/ "Go to Linux homepage")
[](https://www.apple.com/ "Go to Apple homepage")
[](https://www.microsoft.com/ "Go to Apple homepage")
## Overview
Watchtower is a project designed to monitor S3 file events for further AI processing throught [Doc-Search](https://github.com/breadrock1/doc-searcher/blob/master/README.md). This service is been designed for listening creating/uploading new files into cloud storage to download it
and start processing to extract text content, build and store knowledge graph of content entities and store to Doc-Search service.
### Domain
There are following domains:
```
domain
|----> File Storage (core)
| |----> Bucket
| | |----> Context: bucket management into s3 object storage
| | |----> Services: IBucketManager
| |----> Object
| |----> Context: object management into s3 object storage
| |----> Services: IObjectManager
|
|----> Task Processing (support)
| |----> Task
| | |----> Context: task management into storage
| | |----> Services: ITaskStorage
| |----> Message
| |----> Context: task queue management
| |----> Services: ITaskQueue
```
And there are usecases:
```
usecase
|----> Storage Use Case
| |----> CRUD of bucket and object
| |----> generate share URL of stored object
| |----> upload file to storage and create new task processing event
|
|----> Task Use Case
| |----> task management into storage and queue
|
|----> Orchestrator (process)
| |----> combined both usecases to common upload and processing file pipeline
| |----> task processing stages like recognizing and indexing by uploading files
```
There is context map:
```
+----------------+
| Orchesttator |
+--------+-------+
|
┌───────────┴───────────┐
▼ ▼
+----------------+ +-------------+
| StorageUseCase | | TaskUseCase |
+----------------+ +-------------+
| |
▼ ▼
+----------------+ +-------------+
| Storage Domain | | Task Domain |
+----------------+ +-------------+
```
Context data flow:
```
HTTP Request
│
▼
HTTP Handler (ServerState)
│
▼
Orchestrator (orchestrator)
├── StorageUseCase (application)
│ │
│ ▼
│ Storage (domain)
│
└── TaskUseCase (application)
│
▼
Task (domain)
```
## Features
- Task event based - create new event for processing by file uploading;
- Tasks management - using RabbitMQ and Redis for tasks management of processing;
- Text extracting - extract text from PDF, DOCX, and TXT files by OCR and LLM;
- Document storing - storing document object to Doc-Search service;
- Embeddings computing (removed) - computing file text content embeddings by pre-trained model for semantic-search.
- Stateless scalable architecture - stateless service that is guarantied by RabbitMQ and Redis services.
## Quick Start
1. Clone the repository:
```shell
git clone /watchtower.git
cd watchtower
```
2. Build docker image from sources:
```shell
docker build -t watchtower:latest .
```
3. Edit configs file `configs/production.toml` or `.env` file to launch docker compose services
4. Start the application using Docker Compose:
```shell
docker compose up -d watchtower
```
5. The application should now be running. Check the logs with:
```shell
docker compose logs -f
```