{"id":39727729,"url":"https://github.com/richardscottoz/comic-analysis","last_synced_at":"2026-03-04T19:00:31.929Z","repository":{"id":280214858,"uuid":"941316512","full_name":"RichardScottOZ/Comic-Analysis","owner":"RichardScottOZ","description":"Work on looking at analysis of comics with data science and deep learning","archived":false,"fork":false,"pushed_at":"2026-02-26T02:31:57.000Z","size":9788,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-26T07:45:17.326Z","etag":null,"topics":["calibre","closure-lite-simple","comic","comic-analysis","comics","composition","deep-learning","embeddings","fusion","multimodal","panels","paratext","reading"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RichardScottOZ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-02T02:00:29.000Z","updated_at":"2026-02-26T02:32:00.000Z","dependencies_parsed_at":"2026-03-04T19:00:30.086Z","dependency_job_id":null,"html_url":"https://github.com/RichardScottOZ/Comic-Analysis","commit_stats":null,"previous_names":["richardscottoz/comic-analysis"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/RichardScottOZ/Comic-Analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RichardScottOZ%2FComic-Analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RichardScottOZ%2FComic-Analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RichardScottOZ%2FComic-Analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RichardScottOZ%2FComic-Analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RichardScottOZ","download_url":"https://codeload.github.com/RichardScottOZ/Comic-Analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RichardScottOZ%2FComic-Analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30090037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T18:31:08.343Z","status":"ssl_error","status_checked_at":"2026-03-04T18:31:07.708Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calibre","closure-lite-simple","comic","comic-analysis","comics","composition","deep-learning","embeddings","fusion","multimodal","panels","paratext","reading"],"created_at":"2026-01-18T10:56:57.222Z","updated_at":"2026-03-04T19:00:31.888Z","avatar_url":"https://github.com/RichardScottOZ.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Comic Analysis: Project Overview\n\nThis document outlines the ongoing work in analyzing comics using data science techniques, focusing on multimodal modeling for content understanding and querying.\n\n# Models Available\nCosmo Page Stream Segmentation - https://huggingface.co/RichardScottOZ/cosmo-v4\n   - trained on a 25K page subset of more modern comics from multi companies and multiple genres and styles (not manga)\nClosure Lite Simple - https://huggingface.co/RichardScottOZ/comics-analysis-closure-lite-simple\n   - composition and structural analysis\n   \n\n## 1. Project Goals \u0026 Evolution\n\n*   **Version 1 (Closure Lite Framework):**\n    *   **Aim:** To combine page image and VLM text for querying, with panel and reading order recognition.\n    *   **Key Choice:** Utilized 384-dimensional \"lite\" embeddings to allow for testing on retail GPUs.\n    *   **Model:** Based on the [RichardScottOZ/comics-analysis-closure-lite-simple](https://huggingface.co/RichardScottOZ/comics-analysis-closure-lite-simple) model, trained as a context-denoise variant.\n*   **Version 2 (Current Focus):** Initiating work towards a new, more robust modeling framework, particularly for Page Stream Segmentation.\n\n## 2. Core Framework: Closure Lite\n\nThe Closure Lite Framework is a multimodal fusion model designed to analyze images, text, and panel reading order.\n\n*   **Objective:** Generate multimodal fused embeddings that are queryable for similarity.\n    *   Embeddings are efficiently stored in Zarr format (e.g., 80K \"perfect match subset\" takes ~500MB).\n    *   An interface (Flask-based) exists for querying.\n*   **Design Principle:** Usable on a reasonable retail GPU (hence 384-dim embeddings).\n*   **Model Variants:** `base`, `denoise`, `context`, `context-denoise`.\n\n## 3. Data Sources \u0026 Preparation\n\n*   **Primary Data:** Comic pages from various sources (Amazon/Comixology, Dark Horse, Calibre-managed collections like Humble Bundle and DriveThruComics).\n*   **Future Data:** Exploring new Humble Bundles and potential scraping (e.g., Neon Ichigan).\n*   **Initial Processing:**\n    *   **Conversion:** Comic files (PDF, CBZ, etc.) are converted into individual page images.\n    *   **VLM Text Extraction:** Vision-Language Models (VLMs) are used to extract text from each page, preceding general OCR due to its comic-specific challenges.\n    *   **Panel Detection:** Fast-RCNN is used to detect panel bounding boxes and coordinates.\n    *   **DataSpec Integration:** All extracted data is joined into a `DataSpec` format for modeling.\n*   **Dataset Size:** Currently working with a \"Calibre\" subset of ~330K pages (compared to ~805K Amazon pages).\n\n## 4. Basic Workflow\n\n1.  **Find Comics:** Gather digital comic files.\n2.  **Convert to Pages:** Extract images from containers (PDF, CBZ, etc.).\n3.  **VLM Analysis:** Use VLMs to get panel text for each page.\n4.  **R-CNN Panel Detection:** Get panel bounding boxes and coordinates.\n5.  **Data Integration:** Join data into `DataSpec` (e.g., using `coco_to_dataspec`).\n6.  **Perfect Match Subset:** Create high-quality subsets for training.\n7.  **Model Training:** Train the Closure Lite model (using `closure_lite_dataset`, `closure_lite_simple_framework`, `train_closure_lite_simple_with_list`).\n8.  **Embedding Generation:** Run the trained model to generate queryable embeddings.\n9.  **Querying:** Access embeddings via code or a dedicated interface.\n\n## 5. Perfect Match Dataset Notes\n\n*   **Goal:** Ensure text alignment with the correct panels (same number of panels from R-CNN and VLM).\n*   **Challenge:** Initial alignment between Fast-RCNN and various VLM runs was only ~25%, indicating a need to improve this step or consider alternative approaches (e.g., OCR for Fast-RCNN).\n\n## 6. Key Results from Closure Lite (80K Calibre Subset, Context-Denoise)\n\n### Panel Similarity\n*   An early model showed panel similarities, with the model cutting things off at 12 panels.\n![alt text](images/image-1.png)\n\n### Page Similarity\n*   The model demonstrated understanding of page composition and structure.\n![alt text](images/image-2.png)\n\n### Clustering (UMAP Embeddings)\n*   UMAP embeddings revealed distinct clusters.\n![alt text](images/image-3.png)\n*   **Landscape Cluster:** A \"ribbon cluster\" was identified as a \"landscape cluster\" (pages with landscape orientation), which is less common in comics.\n![alt text](images/image-4.png)\n*   **Data Noise:** An adjoining part of this cluster was found to be an error, containing extracted pages from non-comic content (e.g., computing books), highlighting the need for better data filtering.\n\n### Ablation Study (Vision-Only Embeddings)\n*   Clustering of vision-only embeddings showed distinct patterns.\n![alt text](images/image-5.png)\n*   **Cluster 8:** Identified as \"all single panel very dark images\" and \"heavy text one page images,\" with a few multi-panel pages. A disconnected group of \"all white images\" formed a monochromatic sub-cluster.\n*   **Aspect Ratios:** Vision-only clustering also picked out significant differences in panel aspect ratios.\n![alt text](images/image-6.png)\n\n## 7. Relevant Research \u0026 Future Tasks\n\n### Relevant Research\n*   **Survey:** \"One missing piece in Vision and Language: A Survey on Comics Understanding\" (https://arxiv.org/abs/2409.09502v1)\n*   **Neural Networks \u0026 Transformers:** \"Investigating Neural Networks and Transformer Models for Enhanced Comic Decoding\" (https://dl.acm.org/doi/10.1007/978-3-031-70645-5_10)\n*   **Image Indexing:** \"Digital Comics Image Indexing Based on Deep Learning\" (https://www.researchgate.net/publication/326137469_Digital_Comics_Image_Indexing_Based_on_Deep_Learning)\n*   **Synthesis of Graphic Novels:** \"A Deep Learning Pipeline for the Synthesis of Graphic Novels\" (https://computationalcreativity.net/iccc21/wp-content/uploads/2021/09/ICCC_2021_paper_52.pdf)\n\n*   **ComicsPAP:** https://github.com/llabres/ComicsPAP\n=======\n*   **Papers:**\n    *   https://arxiv.org/abs/2503.08561\n    *   https://www.researchgate.net/publication/389748978_ComicsPAP_understanding_comic_strips_by_picking_the_correct_panel?\n    _tp=eyJjb250ZXh0Ijp7ImZpcnN0UGFnZSI6InB1YmxpY2F0aW9uIiwicGFnZSI6InB1YmxpY2F0aW9uIn19\n        - https://huggingface.co/VLR- ComicsPap models\n        - https://github.com/emanuelevivoli/CoMix - CoMix repo\n\n    *   https://www.researchgate.net/publication/326137469_Digital_Comics_Image_Indexing_Based_on_Deep_Learning\n    *   https://computationalcreativity.net/iccc21/wp-content/uploads/2021/09/ICCC_2021_paper_52.pdf\n\n\n### Tasks of Interest\n*   Dialogue transcription, which necessitates a robust panel detection pipeline.\n\n## 8. Pipeline Details\n\n*   Refer to the main [CoMix](https://github.com/RichardScottOZ/CoMix) repository for the Faster RCNN panel detection and processing to make coco style fiels\n*   The code in `detections_2000ad` contains adaptations for inference, distinct from evaluation.\n*   Detailed functions are in `https://github.com/RichardScottOZ/CoMix/tree/main/benchmarks/detections`.\n*   **Technical Stack:** PyTorch (CUDA 11.8 recommended), transformers, xarray and zarr.\n\n## 9. VLM Model Experiments \u0026 Possibilities\n\nExtensive testing has been conducted with various Vision-Language Models for text extraction and analysis:\n\n*   **Gemini 4B:** Handles basics, but fails on some perpetually (reason yet to be understood; Fourier analysis suggested).\n*   **Gemma 12B:** Can handle some failures of Gemini 4B.\n*   **Mistral 3.1:** Generally identifies \"character A and character B,\" can handle Gemini failures.\n*   **Qwen 2.5 VL Instruct:** Failed on the same cases as Gemini 4B.\n*   **Phi4:** Not very good.\n*   **Llama 11B:** Failed to process.\n*   **Gemini Flash 1.5:** Capable of handling missing/null captions and characters.\n*   **Gemini Flash 2.5 Lite:** Good performance, but 8x more expensive than Gemma 4B (which can run locally). Experienced quite a few connection errors with Google. On the hardest 660 samples, had 1/3 errors and 1/5 JSON errors.\n*   **GPT Nano 4.1:** Reported \"unsupported image type,\" indicating lower capability than Google's models.\n*   **Meta Llama 4 Scout:** Much better, with 300/500 success on remaining images, and better cost efficiency (0.08/0.3 vs 0.10/0.40 for Gemini Flash Lite 2.5). However, encountered significant problems with the GMI Cloud provider.\n*   *UPDATE\n*   **GLM 4.6v performanve is in the Gemini Class maybe somewhere between 2.5 and 3 in this error\n*   **Amazon Nova Lite - has some capability but nowhere near Gemini class\n\n## 10. Embeddings Generation Strategy\n\n*   **To Generate:** Panel embeddings (P), Page embeddings (E_page), Reading order embeddings.\n*   **Dataset Coverage:** Amazon perfect matches (212K pages), CalibreComics perfect matches (80K), and a combined dataset of all high-quality samples.\n*   **Technical Approach:** Batch Processing Script.\n\n## 11. Domain capability\n* **Humble Bundle:** If you download everything you will get your book and rpg collections too - which can happily go through pdf and epub extractions and end up with images.\n- Framework handled this, the 'deep learning / linux / programming' book images clusters as their own small part connected to a ribbon, so not ideal to include but a useful result.  So what if you put rpg art in?  Lots of single page 'panels' obtained most likely - and would cluster in the 'no/low test text' UMAP sections\n\n## 12. Future Research\n\n*   Refer to [documentation/future_work/](https://github.com/RichardScottOZ/Comic-Analysis/tree/main/documentation/future_work).\n\n## Version 2\n\n### 1. Paratext\n- Need to work out comic classes inside a comic that will work for 'modern comics' as opposed to old public domain comics\n- Cover, credits and indicia, splash, cover interior for trades, back matter text, back matter art, previews etc.\n#### Problems\n- single issues\n- trade paperbacks\n- omnibuses, collections that are larger\n- anthologies  - art/narrative changes 'inside' a comic - e.g. 2000AD\n    - anthologies inside omnibuses\n\n\n#### a. Zero shot classification\n- Trying bart for the above as have many VLM json files with some signal of what pages might include in 'overall summary' for example.\n\n#### b. Page Stream Segmentation (PSS) \u0026 CoSMo\n\n*   **Importance:** PSS is crucial for the next version of the project, feeding page type markers into multimodal fusion.\n*   **CoSMo Model:**\n    *   [GitHub Repository](https://github.com/mserra0/CoSMo-ComicsPSS) (includes arXiv paper link).\n    *   Specifically designed for PSS.\n    *   Uses Qwen 2.5 VL 32B for OCR.\n    *   **Consideration:** Is Gemma as good/cost-effective for OCR?\n    *   **Current Status:** No pre-trained model available for direct testing; requires training a new one.\n\n- notebook in Data Analysis - comic training\n    - try first pass classes\n    - cover, credits, advertisements, art (previews, internal covers etc. hopefully eventually)), text, back_cover - which could be ads or previews often so might not be a useful classification\n\n#### 2. OCR\n- Needed for everything - page / panel / characters\n- Discussion on Deepseek OCR references Qwen here too: https://news.ycombinator.com/item?id=43549072\n    - note bounding box commentary\n    - also https://pyimagesearch.com/2025/06/09/object-detection-and-visual-grounding-with-qwen-2-5/\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichardscottoz%2Fcomic-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frichardscottoz%2Fcomic-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichardscottoz%2Fcomic-analysis/lists"}