{"id":17099905,"url":"https://github.com/mseitzer/pattern-spotting","last_synced_at":"2025-04-12T23:52:23.565Z","repository":{"id":108935213,"uuid":"116686272","full_name":"mseitzer/pattern-spotting","owner":"mseitzer","description":"Image retrieval system for historical documents using deep features","archived":false,"fork":false,"pushed_at":"2018-01-09T21:08:53.000Z","size":2545,"stargazers_count":10,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-12T23:52:13.981Z","etag":null,"topics":["convolutional-neural-networks","deep-learning","image-retrieval","object-retrieval","pattern-spotting"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mseitzer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-01-08T14:23:50.000Z","updated_at":"2021-07-07T22:43:53.000Z","dependencies_parsed_at":"2023-07-07T12:45:18.448Z","dependency_job_id":null,"html_url":"https://github.com/mseitzer/pattern-spotting","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mseitzer%2Fpattern-spotting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mseitzer%2Fpattern-spotting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mseitzer%2Fpattern-spotting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mseitzer%2Fpattern-spotting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mseitzer","download_url":"https://codeload.github.com/mseitzer/pattern-spotting/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248647246,"owners_count":21139081,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convolutional-neural-networks","deep-learning","image-retrieval","object-retrieval","pattern-spotting"],"created_at":"2024-10-14T15:11:45.503Z","updated_at":"2025-04-12T23:52:23.545Z","avatar_url":"https://github.com/mseitzer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Historic Image Retrieval using Deep Features\n\nThis repository contains the implementation of an image retrieval system. \nThe project was implemented using Python, Tensorflow and Keras. \nThe implementation follows [this paper](https://arxiv.org/abs/1511.05879) from Tolias et al, who make use of global image descriptors obtained from Convolutional Neural Networks. \nFor ease of testing, the project also includes a simple web frontend which allows to query the extracted image descriptor database interactively.\n\nThe system was designed for the application of finding recurring occurences of patterns in historical documents, e.g. a certain sign. \nA system like this could potentially simplify the life of historians, who currently have to sift through heaps of scanned documents by hand.\n\nContributions of this project:\n- Providing a flexible implementation of an image retrieval system\n- Introducing a [new dataset](#notary-charters-dataset) for pattern spotting on historical documents\n\nSee also the [research report](http://www.max-seitzer.de/files/static/pattern-spotting.pdf) for an evaluation of the system and possible future directions.\n\n![Web frontend](resources/web_frontend.png)\n\n## Setup\n\n#### Prerequisites:\n\n- Get all dependencies using [Conda](https://conda.io/): `conda env create -n img-retrieval -f environment.yml`\n- Activate the environment: `source activate img-retrieval`\n- Setup the folder structure: `make setup`\n- Download the notary charters dataset: `make dataset-notary-charters`\n\n#### Building the image descriptor database:\n\n```DATASET=notary_charters IMAGE_DIR=data/interim/notary_charters/notary_charters make repr```\n\nHere, `DATASET` contains the name referring to the generated dataset, and `IMAGE_DIR` is the directory containing the images used for the image database. \nIt might take a while until the process is finished.\n\n#### Query the image database using the commandline:\n\n```cmd/query --features features/notary_charters --model VGG16 \u003cquery_image\u003e```\n\n#### Query the image database using the web frontend:\n\n- Setup the image metadata database: `cmd/modify_database.py --root-dir data/ create database/notary_charters.db data/interim/notary_charters/notary_charters.csv`\n\n- Start the web server on localhost:\n```\ncd web\n./main.py --config config_notary_charters.txt\n```\n\n- You can access the web frontend on localhost:5000\n\n#### Evaluating the performance of the retrieval system:\n\n```make evaluate-notary-charters```\n\nThe evaluation uses hand-labeled pattern occurences, which is why it is only available for the notary charters dataset. \nThe evaluation scripts reports:\n- retrieval performance as mAP, i.e. the mean average precision over all correctly retrieved images\n- localization performance as mAP, i.e. the mean average precision over all correctly retrieved and localized patterns\n- localization performance as IoU, i.e. the intersection over union score of all correctly retrieved patterns\n\n## Repository Structure\n\n```\ncmd/          # Commandline scripts to interact with the library\ndata/         # Images and annotations\nmodels/       # Evaluation and trained models, if any\nsrc/          # The main library\nsrc/data      # Code setting up the datasets\nsrc/features  # Code extracting the image features\nsrc/models    # Code loading the CNNs\nsrc/search    # Code implementing the retrieval algorithm\nweb/          # The web frontend\n```\n\n## Notary charters dataset\n\nNotary charters were used in the classical and medieval eras as legal documents. \nEach notary had his own sign to identify himself and prove the authenticity of the document. \nIn order to make it harder to fake them, the signs were really graphically elaborate. \nHistorians often search for corresponding signs as they link the documents on which they appear together. \nMore on the historical importance of charters can be found [here](http://charlemagneseurope.ac.uk/charter-basics/).\n\n![Notary charters example](resources/notary_charter.png)\nImage property of [HHSTA Salzburg](http://www.monasterium.net/mom/AT-HHStA/SbgE/AUR_1420_IV_15/charter)\n\nAs in the domain of historical documents there is next to no annotated data available (we know only of [DocExplore](http://spotting.univ-rouen.fr/), but they provide no annotations), we had to build our own dataset in order to evaluate our approach.\nWe now release our dataset with bounding boxes of notary signs here.\nNote that all images fall under the terms of use of Monasterium, see [here](http://monasterium.net/mom/terms-of-use)!\n\nThe dataset consists of 975 images of notary charters, all from [monasterium.net](http://www.monasterium.net/mom/home).\nOn those images, 606 search patterns (different notary signs) were hand-annotated with bounding boxes. \nThen, 295 of the patterns were sorted into 77 correspondency groups with on average 3.8 patterns per group. \nThe remaining 311 patterns appear only once and have no corresponding patterns on other images.\n\nThere is also a second, larger dataset which consists of the 977 notary charter images and 31215 distractor images which make the retrieval task harder.\n\n## References\n\n- [Giorgos Tolias, Ronan Sicre, Hervé Jégou - Particular object retrieval with integral max-pooling of CNN activations](https://arxiv.org/abs/1511.05879)\n- [Matlab implementation of Particular object retrieval with integral max-pooling of CNN activations](http://cmp.felk.cvut.cz/~toliageo/soft.html)\n- [Monasterium for the notary charters images](http://monasterium.net/mom/home)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmseitzer%2Fpattern-spotting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmseitzer%2Fpattern-spotting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmseitzer%2Fpattern-spotting/lists"}