{"id":22530214,"url":"https://github.com/cpietsch/vikus-iiif-generator","last_synced_at":"2025-08-11T09:09:24.164Z","repository":{"id":39483287,"uuid":"436208585","full_name":"cpietsch/vikus-IIIF-generator","owner":"cpietsch","description":"VIKUS IIIF Generator","archived":false,"fork":false,"pushed_at":"2023-07-23T21:37:40.000Z","size":625,"stargazers_count":17,"open_issues_count":7,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-03T16:12:54.340Z","etag":null,"topics":["docker","glam","vikus","vikus-viewer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cpietsch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-12-08T10:38:22.000Z","updated_at":"2025-02-01T23:18:23.000Z","dependencies_parsed_at":"2025-05-27T06:12:49.555Z","dependency_job_id":"4eb0e4bf-5769-4a5e-a623-c82ec574f090","html_url":"https://github.com/cpietsch/vikus-IIIF-generator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cpietsch/vikus-IIIF-generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpietsch%2Fvikus-IIIF-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpietsch%2Fvikus-IIIF-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpietsch%2Fvikus-IIIF-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpietsch%2Fvikus-IIIF-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cpietsch","download_url":"https://codeload.github.com/cpietsch/vikus-IIIF-generator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpietsch%2Fvikus-IIIF-generator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268628767,"owners_count":24281023,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-03T02:00:12.545Z","response_time":2577,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","glam","vikus","vikus-viewer"],"created_at":"2024-12-07T07:18:45.744Z","updated_at":"2025-08-11T09:09:24.141Z","avatar_url":"https://github.com/cpietsch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VIKUS IIIF Generator\nThis GitHub Repo contains code to build a VIKUS IIIF generator instance, which can be used to view IIIF Collections (Presentation 3.x API) with [VIKUS Viewer](https://github.com/cpietsch/vikus-viewer). The code crawls all manifests in a IIIF Collection (v3) recursively, downloads images of each canvas, creates spritesheets, extracts metadata, and generates similarity layouts using UMAP and Rasterfairy. The code also includes a REST API and websocket for real-time progress updates, and allows for the download of a ZIP archive for the VIKUS Viewer. The code uses redis cache for manifests and feature vectors and also extract embeddings with CLIP using huggingface transformers.\n\n\u003cimg width=\"1166\" alt=\"vikusdocker\" src=\"https://user-images.githubusercontent.com/129529/186490880-1dffef6b-ba03-479c-985d-d32941d440b7.png\"\u003e\n\n\n## Features\n\n- Crawling of all manifests in a [IIIF Collection](https://iiif.io/api/presentation/3.0/) (v3) recursively\n- Asynchronous queueing for crawling manifests and images\n- Downloads an image of each canvas and save it to disk\n- Create spritesheets\n- Extract metadata from manifests and generate keywords using [spacy](https://spacy.io/models/xx)\n- Extract embeddings with CLIP using [huggingface transformers](https://huggingface.co/docs/transformers/model_doc/clip)\n- Redis cache for manifests and feature vectors\n- Generate similarity layouts with [UMAP](https://umap-learn.readthedocs.io/en/latest/) and [Rasterfairy](https://github.com/Quasimondo/RasterFairy)\n- REST Api\n- Websocket real time progress\n- Download data ZIP archive for VIKUS Viewer\n\n## RUN\n\nTo start up the Docker instance, run:\n\n```\ndocker compose up\n```\n\nIf you have a GPU available, you can increase the speed of CLIP features extraction by running:\n\n```\ndocker compose -f docker-compose.gpu.yml up\n```\n\nIf you need to rebuild the image, run:\n\n```\ndocker-compose down\ndocker-compose up --force-recreate --build -d\n```\n\n### CLI\n\nTo run the CLI, run:\n\n```\ndocker exec -it vikus-docker-vikusdocker-1 python cli.py https://resource.swissartresearch.net/manifest/zbz-collection-100\n```\nYou can also use the CLI with the following [options](https://github.com/cpietsch/vikus-docker/blob/main/scripts/files/defaults.json):\n\n```\npython cli.py https://resource.swissartresearch.net/manifest/zbz-collection-100 collection.worker=1 collection.depth=1\n```\n\n## Usage\n\nTo use the VIKUS IIIF generator, open the frontend in a browser at http://localhost:8000/frontend/. Paste an IIIF Collection (v3) URL and click \"Create\". Click \"Generate\" to run all the steps.\n\nIf you want, you can fiddle with the settings of each step and/or run them individually. Once the process is finished, you can open a VIKUS Viewer preview, or download the data ZIP and extract it into the /data folder of the VIKUS Viewer.\n\n## Tech\n- Python3\n- Redis\n- Huggingface Transformers\n- CLIP\n- Spacy\n- UMAP\n- Rasterfairy\n- FastAPI\n- Svelte\n- Docker\n\n## API\nOpen the VIKUS IIIF generator API in a browser:\nhttp://localhost:3000/docs\n\n\n## Sub-projects\n- https://github.com/cpietsch/vikus-docker-frontend/ (svelte)\n- https://github.com/cpietsch/vikus-viewer\n\n## Requirements\n- Docker\n- Docker Compose\n\n## Notebooks\nTo start the notebook server on port 5000 instead of the server, connect to the vikusdocker and run\n- `pip install jupyter-lab`\n- `jupyter-lab --port 5000`\n\n## Funding\nThe work on this project was made possible through the [Swiss Art Research Infrastructure (SARI)](https://swissartresearch.net/) in the framework of the [Bilder der Schweiz Online](https://bso.swissartresearch.net) project funded by the [Stiftung Familie Fehlmann](https://stiftung-familie-fehlmann.ch/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpietsch%2Fvikus-iiif-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpietsch%2Fvikus-iiif-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpietsch%2Fvikus-iiif-generator/lists"}