{"id":29035531,"url":"https://github.com/breadrock1/watchtower","last_synced_at":"2026-05-14T15:35:57.440Z","repository":{"id":217919902,"uuid":"745109507","full_name":"breadrock1/watchtower","owner":"breadrock1","description":"Watchtower is adapter-service for cloud or fs for file processing using LLM.","archived":false,"fork":false,"pushed_at":"2026-03-31T16:50:00.000Z","size":860,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-31T17:37:41.554Z","etag":null,"topics":["async","docsearch","elasticsearch","fsnotify","go","golang","load-files","search"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/breadrock1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-01-18T16:58:05.000Z","updated_at":"2026-03-31T16:45:36.000Z","dependencies_parsed_at":"2024-11-11T19:15:52.698Z","dependency_job_id":"4afffdc3-8857-4d7e-8066-84f34083b272","html_url":"https://github.com/breadrock1/watchtower","commit_stats":null,"previous_names":["breadrock1/doc-notifier","breadrock1/doc-watcher","breadrock1/watchtower"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/breadrock1/watchtower","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breadrock1%2Fwatchtower","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breadrock1%2Fwatchtower/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breadrock1%2Fwatchtower/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breadrock1%2Fwatchtower/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/breadrock1","download_url":"https://codeload.github.com/breadrock1/watchtower/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breadrock1%2Fwatchtower/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33031286,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","docsearch","elasticsearch","fsnotify","go","golang","load-files","search"],"created_at":"2025-06-26T12:10:46.956Z","updated_at":"2026-05-14T15:35:57.387Z","avatar_url":"https://github.com/breadrock1.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Watchtower Metaverse project\n\n[![Pull-Request](https://github.com/breadrock1/watchtower/actions/workflows/pull-request.yml/badge.svg)](https://github.com/breadrock1/watchtower/actions/workflows/pull-request.yml)\n\n[![Target - Linux](https://img.shields.io/badge/OS-Linux-blue?logo=linux\u0026logoColor=white)](https://www.linux.org/ \"Go to Linux homepage\")\n[![Target - MacOS](https://img.shields.io/badge/OS-MacOS-blue?logo=linux\u0026logoColor=white)](https://www.apple.com/ \"Go to Apple homepage\")\n[![Target - Windows](https://img.shields.io/badge/OS-Windows-blue?logo=linux\u0026logoColor=white)](https://www.microsoft.com/ \"Go to Apple homepage\")\n\n## Overview\n\nWatchtower is a project designed to monitor S3 file events for further AI processing throught [Doc-Search](https://github.com/breadrock1/doc-searcher/blob/master/README.md). This service is been designed for listening creating/uploading new files into cloud storage to download it \nand start processing to extract text content, build and store knowledge graph of content entities and store to Doc-Search service.\n\n![architecture.png](docs/architecture.png)\n\n### Domain\n\nThere are following domains:\n\n```\ndomain\n   |----\u003e File Storage (core)\n   |        |----\u003e Bucket\n   |        |       |----\u003e Context: bucket management into s3 object storage\n   |        |       |----\u003e Services: IBucketManager\n   |        |----\u003e Object\n   |                |----\u003e Context: object management into s3 object storage\n   |                |----\u003e Services: IObjectManager\n   |\n   |----\u003e Task Processing (support)\n   |        |----\u003e Task\n   |        |       |----\u003e Context: task management into storage\n   |        |       |----\u003e Services: ITaskStorage\n   |        |----\u003e Message\n   |                |----\u003e Context: task queue management\n   |                |----\u003e Services: ITaskQueue\n```\n\nAnd there are usecases:\n\n```\nusecase\n   |----\u003e Storage Use Case\n   |        |----\u003e CRUD of bucket and object\n   |        |----\u003e generate share URL of stored object\n   |        |----\u003e upload file to storage and create new task processing event\n   |\n   |----\u003e Task Use Case\n   |        |----\u003e task management into storage and queue\n   |\n   |----\u003e Orchestrator (process)\n   |        |----\u003e combined both usecases to common upload and processing file pipeline\n   |        |----\u003e task processing stages like recognizing and indexing by uploading files\n```\n\nThere is context map:\n\n```\n           +----------------+\n           |  Orchesttator  |\n           +--------+-------+\n                    |\n        ┌───────────┴───────────┐\n        ▼                       ▼\n+----------------+         +-------------+\n| StorageUseCase |         | TaskUseCase |\n+----------------+         +-------------+\n        |                        |\n        ▼                        ▼\n+----------------+         +-------------+\n| Storage Domain |         | Task Domain |\n+----------------+         +-------------+\n```\n\nContext data flow:\n\n```\nHTTP Request\n     │\n     ▼\nHTTP Handler (ServerState)\n     │\n     ▼\nOrchestrator (orchestrator)\n    ├── StorageUseCase (application)\n    │       │\n    │       ▼\n    │    Storage (domain)\n    │\n    └── TaskUseCase (application)\n            │\n            ▼\n          Task (domain)\n```\n\n## Features\n\n - Task event based                - create new event for processing by file uploading;\n - Tasks management                - using RabbitMQ and Redis for tasks management of processing;\n - Text extracting                 - extract text from PDF, DOCX, and TXT files by OCR and LLM;\n - Document storing                - storing document object to Doc-Search service;\n - Embeddings computing (removed)  - computing file text content embeddings by pre-trained model for semantic-search. \n - Stateless scalable architecture - stateless service that is guarantied by RabbitMQ and Redis services.\n\n## Quick Start\n\n1. Clone the repository:\n\n    ```shell\n    git clone \u003crepository-url\u003e/watchtower.git\n    cd watchtower\n    ```\n\n2. Build docker image from sources:\n\n    ```shell\n   docker build -t watchtower:latest .\n    ```\n\n3. Edit configs file `configs/production.toml` or `.env` file to launch docker compose services\n\n4. Start the application using Docker Compose:\n\n    ```shell\n    docker compose up -d watchtower \u003cother-needed-services\u003e\n    ```\n\n5. The application should now be running. Check the logs with:\n\n    ```shell\n    docker compose logs -f\n    ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbreadrock1%2Fwatchtower","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbreadrock1%2Fwatchtower","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbreadrock1%2Fwatchtower/lists"}