{"id":31468179,"url":"https://github.com/thescottyb/square-tools","last_synced_at":"2026-04-02T02:51:12.292Z","repository":{"id":317451885,"uuid":"1067333083","full_name":"TheScottyB/square-tools","owner":"TheScottyB","description":"MongoDB-based caching and change tracking system for Square catalog items with Photos Library integration","archived":false,"fork":false,"pushed_at":"2025-09-30T23:16:05.000Z","size":18,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-01T01:11:23.078Z","etag":null,"topics":["api","bash","cache","change-tracking","cli","macos","mongodb","photos","python","square"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TheScottyB.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-30T17:53:47.000Z","updated_at":"2025-09-30T23:16:09.000Z","dependencies_parsed_at":"2025-10-01T01:12:02.050Z","dependency_job_id":"8c5c4782-548f-431f-9186-fb0512fae6ad","html_url":"https://github.com/TheScottyB/square-tools","commit_stats":null,"previous_names":["thescottyb/square-tools"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/TheScottyB/square-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScottyB%2Fsquare-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScottyB%2Fsquare-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScottyB%2Fsquare-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScottyB%2Fsquare-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TheScottyB","download_url":"https://codeload.github.com/TheScottyB/square-tools/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScottyB%2Fsquare-tools/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277900424,"owners_count":25896988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-01T02:00:09.286Z","response_time":88,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","bash","cache","change-tracking","cli","macos","mongodb","photos","python","square"],"created_at":"2025-10-01T19:55:12.493Z","updated_at":"2026-04-02T02:51:12.277Z","avatar_url":"https://github.com/TheScottyB.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Square Items MongoDB Cache System\n\nA comprehensive caching and change tracking system for Square catalog items using MongoDB.\n\n## 🏗️ System Overview\n\nThis system provides:\n- **Local MongoDB cache** of all Square catalog items\n- **Change detection** with before/after snapshots\n- **Audit trail** of all modifications\n- **AI-powered image processing** with automated background removal\n- **CLI tools** for easy management\n- **Photo integration** from Photos Library to Square\n- **Gemini AI chat** for image analysis and assistance\n\n## 📁 Project Structure\n\n```\n~/Workspace/square-tools/\n├── bin/\n│   ├── square_cache.sh           # CLI wrapper for cache operations\n│   ├── photos_to_square.sh       # Photos Library → Square uploader\n│   ├── process_and_upload.sh     # AI-powered photo processing + upload\n│   ├── gemini_chat.sh           # Interactive Gemini chat interface\n│   └── browse_photos.sh          # Photos Library browser\n├── cache-system/\n│   ├── square_cache_manager.py   # Core MongoDB cache manager\n│   ├── bg_removal_service.py     # Background removal service (Remove.bg, Gemini, Banana)\n│   ├── test_bg_removal.py        # System health checker\n│   └── demo_change_tracking.sh   # Demonstration script\n├── data/\n│   ├── photo_exports/            # Exported photos directory\n│   └── logs/                     # Application logs\n├── config.sh                     # Configuration file\n├── IMAGE_PROCESSING.md           # Image processing documentation\n├── QUICKSTART_BG_REMOVAL.md      # Quick start guide\n├── DEMO.md                       # Quick demos and examples\n└── README.md                     # This documentation\n```\n\n## 🗄️ MongoDB Database Schema\n\n### Database: `square_cache`\n\n#### Collections:\n\n1. **`catalog_items`** - Cached Square catalog items\n   ```json\n   {\n     \"_id\": \"ObjectId\",\n     \"id\": \"ANE5SXKQR4JZ6AYEZDO26IMX\",\n     \"type\": \"ITEM\", \n     \"version\": 1759194139169,\n     \"item_data\": {\n       \"name\": \"Trading Places - Rare Video 8 Format (1983)\",\n       \"image_ids\": [\"ZDHVCXFYL7MNUQV6TAQTXH4G\", \"...\"],\n       ...\n     },\n     \"content_hash\": \"620e6650a2732e5a...\",\n     \"cached_at\": \"2025-09-30T02:31:13.439Z\"\n   }\n   ```\n\n2. **`change_snapshots`** - Before/after change tracking\n   ```json\n   {\n     \"_id\": \"ObjectId\",\n     \"item_id\": \"ANE5SXKQR4JZ6AYEZDO26IMX\",\n     \"item_name\": \"Trading Places - Rare Video 8 Format (1983)\",\n     \"change_type\": \"update\",\n     \"timestamp\": \"2025-09-30T02:35:00.000Z\",\n     \"before_data\": { /* complete item before change */ },\n     \"after_data\": { /* complete item after change */ },\n     \"differences\": {\n       \"item_data.image_ids\": {\n         \"before\": null,\n         \"after\": [\"ZDHVCXFYL7MNUQV6TAQTXH4G\", \"JUOA5BTTKGJOXWHJ62RR4I62\"]\n       }\n     },\n     \"square_version_before\": 1759194139169,\n     \"square_version_after\": 1759194200000\n   }\n   ```\n\n3. **`sync_log`** - Sync operation history\n   ```json\n   {\n     \"_id\": \"ObjectId\", \n     \"timestamp\": \"2025-09-30T02:31:12.920Z\",\n     \"total_items\": 31,\n     \"created_count\": 31,\n     \"updated_count\": 0,\n     \"changes_detected\": 31,\n     \"duration_seconds\": 0.532\n   }\n   ```\n\n4. **`bg_removal_cache`** - Processed image cache (AI background removal)\n   ```json\n   {\n     \"_id\": \"ObjectId\",\n     \"image_hash\": \"sha256_hash_of_original\",\n     \"processed_path\": \"/path/to/processed_image.png\",\n     \"metadata\": {\n       \"provider\": \"remove_bg\",\n       \"status\": \"success\",\n       \"cost\": 0.002,\n       \"timestamp\": \"2025-12-20T03:22:00Z\",\n       \"original_size\": 1024000,\n       \"processed_size\": 512000\n     },\n     \"cached_at\": \"2025-12-20T03:22:00Z\"\n   }\n   ```\n\n## 🚀 Quick Start\n\n### Runtime Policy Note\n\nPrivileged operations are gated by runtime policy:\n\n- `runtime/capability_matrix.json`\n- `runtime/operation_policy.json`\n- `bin/agent_preflight.sh`\n\nExample:\n\n```bash\nbin/agent_preflight.sh --operation square_cache_sync --runtime \"${SQUARE_RUNTIME_ID:-local_cli}\"\n```\n\n### 1. Prerequisites\n\n```bash\n# Install MongoDB\nbrew tap mongodb/brew\nbrew install mongodb-community@8.0\nbrew services start mongodb-community@8.0\n\n# Install Python dependencies\npip3 install pymongo requests Pillow\n```\n\n### 2. Set Environment Variables\n\n```bash\n# Required for Square API\nexport SQUARE_TOKEN=\"YOUR_SQUARE_TOKEN_HERE\"\n\n# Optional: For AI background removal (at least one recommended)\nexport REMOVEBG_API_KEY=\"your_remove_bg_api_key\"  # Recommended\nexport GEMINI_API_KEY=\"your_gemini_api_key\"        # For chat \u0026 image analysis\n```\n\n### 2.5 Security Gate\n\nRun secret scanning before commit/release:\n\n```bash\n./bin/secret_scan.sh\n```\n\n### 3. Initial Setup\n\n```bash\n# Load configuration (creates directories)\nsource ~/Workspace/square-tools/config.sh\n\n# Check system status\nsquare_cache.sh status\n\n# Perform initial sync (creates baseline cache)\nsquare_cache.sh sync\n\n# View all changes\nsquare_cache.sh changes\n```\n\n## 📝 CLI Commands\n\n### Cache Management\n\n```bash\n# Sync all items from Square to MongoDB cache\nsquare_cache.sh sync\n\n# Show recent changes\nsquare_cache.sh changes\nsquare_cache.sh changes --since 2025-09-30\n\n# Generate detailed change report\nsquare_cache.sh report\nsquare_cache.sh report --output json\n\n# Search cached items\nsquare_cache.sh search \"Trading Places\"\n\n# Get specific item details\nsquare_cache.sh item ANE5SXKQR4JZ6AYEZDO26IMX\n\n# Check system status\nsquare_cache.sh status\n```\n\n### Photo Management\n\n```bash\n# Browse Photos Library\nbrowse_photos.sh recent 10\nbrowse_photos.sh search \"IMG_7232\"\nbrowse_photos.sh random 5\nbrowse_photos.sh export 27569\n\n# Upload photos to Square (basic)\nphotos_to_square.sh 27569 \"YOUR_SQUARE_TOKEN\"\n\n# Process with AI background removal and upload\nprocess_and_upload.sh 27569 --remove-bg --preview\nprocess_and_upload.sh 27569 --remove-bg --skip-upload\nprocess_and_upload.sh 27569 --remove-bg --provider remove_bg\n```\n\n### AI Image Processing\n\n```bash\n# Remove background from image (Python interface)\npython3 cache-system/bg_removal_service.py image.jpg remove_bg\n\n# Chat with Gemini about text\ngemini_chat.sh flash\n\n# Chat with Gemini about an image\ngemini_chat.sh flash image.jpg\n\n# Test system health\npython3 cache-system/test_bg_removal.py\n```\n\n### Demo \u0026 Testing\n\n```bash\n# Run change tracking demonstration\n~/Workspace/square-tools/cache-system/demo_change_tracking.sh\n```\n\n## 🔄 Change Detection Process\n\n### 1. Content Hashing\n- Each item gets a SHA256 hash of its content (excluding volatile fields like `updated_at`, `version`)\n- Changes detected by comparing hashes between Square API and cached versions\n\n### 2. Before/After Snapshots\n- **BEFORE**: Complete item data from MongoDB cache\n- **AFTER**: Complete item data from Square API\n- **DIFFERENCES**: Field-by-field comparison showing exactly what changed\n\n### 3. Change Types\n- **`create`**: New item added to Square catalog\n- **`update`**: Existing item modified (with detailed diff)\n- **`delete`**: Item removed from Square catalog (planned feature)\n\n## 📊 Real-World Example\n\n### Before State (Trading Places Item):\n```json\n{\n  \"name\": \"Trading Places - Rare Video 8 Format (1983)\",\n  \"version\": 1759194139169,\n  \"item_data\": {\n    // NO image_ids field - item has no images\n  }\n}\n```\n\n### After State (Images Attached):\n```json\n{\n  \"name\": \"Trading Places - Rare Video 8 Format (1983)\", \n  \"version\": 1759194200000,\n  \"item_data\": {\n    \"image_ids\": [\n      \"ZDHVCXFYL7MNUQV6TAQTXH4G\",  // 1A2E186B-A015-4A86-84B6-C1F76EC9810D.jpeg\n      \"JUOA5BTTKGJOXWHJ62RR4I62\"   // 5748D4B2-F1DC-4BB6-84E4-90B56DCA4059.jpeg\n    ]\n  }\n}\n```\n\n### Change Detection Output:\n```\n🔄 Trading Places - Rare Video 8 Format (1983)\n   Type: update\n   Changes: item_data.image_ids, version, updated_at\n   Version: 1759194139169 → 1759194200000\n   \n   BEFORE: No images attached\n   AFTER:  2 images attached\n```\n\n## 🎯 Use Cases\n\n### 1. Audit Trail\n- Track all changes to your Square catalog\n- See who changed what and when\n- Compliance and business intelligence\n\n### 2. Content Management\n- Backup/restore item configurations  \n- Bulk operations planning\n- A/B testing tracking\n\n### 3. Integration Monitoring\n- Detect unauthorized changes\n- Monitor third-party integrations\n- Quality assurance workflows\n\n### 4. Analytics\n- Track catalog growth over time\n- Monitor pricing changes\n- Image attachment patterns\n\n## 🧪 Testing \u0026 Validation\n\n### Current System State\n- ✅ **31 items cached** from Square catalog\n- ✅ **All items have baseline snapshots** (before state)\n- ✅ **Trading Places item cached** with NO images (perfect test case)\n- ✅ **2 images uploaded** to Square catalog ready for attachment\n- ✅ **Change detection ready** to track image attachment\n\n### Test Workflow\n1. **Baseline established**: All items cached with current state\n2. **Make changes**: Attach images via Square Dashboard or API\n3. **Sync \u0026 detect**: `./square_cache.sh sync` will detect changes\n4. **View changes**: `./square_cache.sh changes` shows before/after\n\n## 🔧 Technical Details\n\n### Change Detection Algorithm\n```python\ndef _detect_changes(self, square_item: Dict) -\u003e Optional[ChangeSnapshot]:\n    item_id = square_item['id']\n    cached_item = self.items_collection.find_one({\"id\": item_id})\n    \n    if not cached_item:\n        return ChangeSnapshot(change_type='create', ...)\n    \n    square_hash = self._calculate_hash(square_item)\n    cached_hash = cached_item.get('content_hash', '')\n    \n    if square_hash != cached_hash:\n        differences = self._find_differences(cached_item, square_item)\n        return ChangeSnapshot(change_type='update', differences=differences, ...)\n    \n    return None\n```\n\n### Performance Optimizations\n- **MongoDB indexes** on key fields (id, timestamp, item_name)\n- **Content hashing** for efficient change detection\n- **Cursor-based pagination** for large catalogs\n- **Batch operations** for bulk changes\n\n### Error Handling\n- **Connection retry logic** for MongoDB and Square API\n- **Partial sync recovery** if operations fail\n- **Comprehensive logging** of all operations\n- **Data validation** before cache updates\n\n## 🎉 Success Metrics\n\nThe system successfully provides:\n\n1. **Complete catalog cache**: All 31 Square items cached locally in MongoDB\n2. **Change tracking**: Before/after snapshots for all modifications  \n3. **Audit trail**: Complete history of when and what changed\n4. **CLI tools**: Easy-to-use command-line interface\n5. **Photo integration**: Direct upload from macOS Photos Library\n6. **Real-time sync**: Detect changes on-demand or scheduled\n7. **Detailed reporting**: JSON and human-readable change reports\n\nThis creates a robust foundation for Square catalog management, change tracking, and business intelligence.\n\n---\n\n## 🚀 Next Steps\n\nTo see the change tracking in action:\n\n1. **Attach the uploaded images** to Trading Places item in Square Dashboard\n2. **Run sync**: `./square_cache.sh sync` \n3. **View changes**: `./square_cache.sh changes --since 2025-09-30`\n4. **Generate report**: `./square_cache.sh report`\n\nThe system will detect and document the exact changes made, creating a complete before/after snapshot for audit and analysis purposes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthescottyb%2Fsquare-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthescottyb%2Fsquare-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthescottyb%2Fsquare-tools/lists"}