{"id":26356543,"url":"https://github.com/ds4sd/deepsearch-examples","last_synced_at":"2025-07-21T08:02:20.373Z","repository":{"id":56818768,"uuid":"520438042","full_name":"DS4SD/deepsearch-examples","owner":"DS4SD","description":"Examples using the Deep Search functionalities","archived":false,"fork":false,"pushed_at":"2024-05-22T14:07:21.000Z","size":17072,"stargazers_count":31,"open_issues_count":3,"forks_count":13,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-05-22T14:52:42.166Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DS4SD.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-02T09:44:30.000Z","updated_at":"2024-05-27T14:11:48.049Z","dependencies_parsed_at":"2023-02-13T00:00:23.568Z","dependency_job_id":"ca470842-79df-413e-a823-7b52f49688d2","html_url":"https://github.com/DS4SD/deepsearch-examples","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DS4SD/deepsearch-examples","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DS4SD","download_url":"https://codeload.github.com/DS4SD/deepsearch-examples/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-examples/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266263057,"owners_count":23901354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-16T13:31:00.971Z","updated_at":"2025-07-21T08:02:20.354Z","avatar_url":"https://github.com/DS4SD.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep Search Examples\n\n![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10-blue)\n[![Jupyter](https://img.shields.io/static/v1?label=jupyter\u0026message=py\u0026color=orange\u0026logo=jupyter)](https://mybinder.org/v2/gh/DS4SD/deepsearch-examples/main)\n[![License MIT](https://img.shields.io/github/license/ds4sd/deepsearch-toolkit)](https://opensource.org/licenses/MIT)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Docs](https://img.shields.io/badge/website-live-brightgreen)](https://ds4sd.github.io/deepsearch-toolkit/)\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/DS4SD/deepsearch-examples/main)\n\n\nIn this repository we showcase some common usage of Deep Search\nfor **Document conversion** as well as **Data and Knowledge exploration**.\n\n\n## Quick links\n\n- [Deep Search Toolkit](https://github.com/ds4sd/deepsearch-toolkit)\n- [Documentation](https://ds4sd.github.io/deepsearch-toolkit/)\n\n## Examples\n\n### Setup and usage\n\nEach example starts by defining its input parameters. This is supported by Pydantic\nSettings, allowing automated loading from a `.env` file or env vars. Furthermore, access\nis based on [Profiles][profiles]. Unless otherwise configured, the profile used is the active one.\n\n### Document conversion\n\n|    | Name              | Description |\n| -- | ----------------- | ----------- |\n| 1. | [Convert documents quick start][doc_conv_quick_start] | Full example on programmatic document conversion \u003cbr /\u003e [\u003cimg src=\".readme_resources/doc_conversion_quick_start.png\" width=\"300px\" /\u003e][doc_conv_quick_start] |\n| 2. | [Convert documents with custom settings][doc_conv_custom_pipeline] | Full example on programmatic document conversion with custom conversion settings \u003cbr /\u003e [\u003cimg src=\".readme_resources/doc_conversion_quick_start.png\" width=\"300px\" /\u003e][doc_conv_custom_pipeline] |\n| 3. | [Visualize bounding boxes][doc_conv_visualize_bbox] | Visualize the bbox of the text elements \u003cbr /\u003e [\u003cimg src=\"./.readme_resources/visualize_bbox.png\" width=\"300px\" /\u003e][doc_conv_visualize_bbox] |\n| 4. | [Extract figures from documents][doc_conv_extract_figures] | Given a PDF file, extract the figures \u003cbr /\u003e [\u003cimg src=\".readme_resources/extract_figures.png\" width=\"300px\" /\u003e][doc_conv_extract_figures] |\n| 5. | [Extract tables][doc_conv_extract_tables] | Given a PDF file, extract the tables \u003cbr /\u003e [\u003cimg src=\".readme_resources/extract_tables.png\" width=\"300px\" /\u003e][doc_conv_extract_tables] |\n\n### NLP on Documents\n\n|    | Name              | Description |\n| -- | ----------------- | ----------- |\n| 1. | [NLP on documents](examples/nlp_on_documents/nlp_on_documents.ipynb)* | A few quick examples on how to apply NLP models on documents (eg extracting key-terms) |\n| 2. | [Reference Parsing](examples/nlp_for_references/nlp_for_references.ipynb)* | Examples on how to parse references from Documents |\n| 3. | [Material Extraction](examples/nlp_for_materials/nlp_for_materials.ipynb)* | Examples on how to extract materials from Documents |\n\n### Data queries\n\nThis section will showcase examples which query data processed via Deep Search.\n\n|    | Name              | Description |\n| -- | ----------------- | ----------- |\n| 1. | [Data query quick start][data_query_quick_start] | Example listing data collections, making search in one and more document collections, using source for projection \u003cbr /\u003e [\u003cimg src=\".readme_resources/data_query_quick_start.png\" width=\"300px\" /\u003e][data_query_quick_start] |\n| 2. | [Chemistry search queries][data_query_chemistry] | Search the chemistry databases for known molecules \u003cbr /\u003e [\u003cimg src=\".readme_resources/data_query_chemistry.png\" width=\"300px\" /\u003e][data_query_chemistry] |\n| 3. | [Chemistry and patent searches via PatCID][data_query_chemistry_patcid] | Explore the chemistry databases using substructure and similarity searches and navigate to the world-wide patents which reference molecules \u003cbr /\u003e [\u003cimg src=\".readme_resources/data_query_chemistry_patcid.png\" width=\"300px\" /\u003e][data_query_chemistry] |\n| 4. | [Snippets and aggregations in data queries][data_query_snippets] | Extract snippets in search queries and leverage aggregations for exploratory analysis \u003cbr /\u003e [\u003cimg src=\".readme_resources/data_query_snippets.png\" width=\"300px\" /\u003e][data_query_snippets] |\n\n\n### Semantic queries \u0026 RAG\n\nThis section will showcase examples of semantic capabilitilies in the area of Q\u0026A using RAG.\n\n|    | Name              | Description |\n| -- | ----------------- | ----------- |\n| 1. | [QA quick start][qa_quick_start] | Get started with semantic ingestion, RAG, and retrieval.\u003cbr /\u003e [\u003cimg src=\".readme_resources/documentqa.png\" width=\"300px\" /\u003e][qa_quick_start] |\n| 2. | [QA deep dive][qa_deep_dive] | Explore advanced RAG and semantic retrieval capabilities.\u003cbr /\u003e [\u003cimg src=\".readme_resources/qa_doc_collection.png\" width=\"300px\" /\u003e][qa_deep_dive] |\n\n\n### Bring your own\n\nThis section will showcase examples for bringing your own documents, csv data, nlp models and more.\n\n|    | Name              | Description |\n| -- | ----------------- | ----------- |\n| 1. | [Bring your own PDF][bring_your_own_pdf] | Upload your own PDF documents, search on them and export the result as JSON files. \u003cbr /\u003e [\u003cimg src=\".readme_resources/bring_your_own_pdf.png\" width=\"300px\" /\u003e][bring_your_own_pdf] |\n| 2. | [Bring your own converted documents][bring_your_own_converted_documents] | Upload your documents already formatted as JSON. \u003cbr /\u003e [\u003cimg src=\".readme_resources/bring_your_own_converted_documents.png\" width=\"300px\" /\u003e][bring_your_own_converted_documents] |\n| 3. | Bring your own DataFrame | Bring your own DataFrame from CSV, XLSX, etc and explore the content in a knowledge graph \u003cbr /\u003e |\n\n\n### Attachments and metadata\n\nThis section will showcase examples for managing index item attachments and metadata.\n\n|    | Name              | Description |\n| -- | ----------------- | ----------- |\n| 1. | [Manage attachments][manage_attachments] | Manage index item attachments \u003cbr /\u003e [\u003cimg src=\".readme_resources/manage_attachments.png\" width=\"300px\" /\u003e][manage_attachments] |\n\n\n### Knowledge graphs\n\nThis section will showcase examples related to the use of knowledge graphs (KGs) in Deep Search.\n\n|    | Name               | Description |\n| -- | ------------------ | ----------- |\n| 1. | [Using Deep Search KGs with PyTorch Geometric][kg_download_quick_start] | Download knowledge graphs from Deep Search and import them in PyTorch Geometric. \u003cbr /\u003e [\u003cimg src=\".readme_resources/kg_download_quick_start.png\" width=\"300px\" /\u003e][kg_download_quick_start] |\n\n### Integrations\n\nThis section will showcase examples related to the integration of Deep Search with other tools and utilities.\n\n|    | Name               | Description |\n| -- | ------------------ | ----------- |\n| 1. | [Annotations on argilla.io][integration_argilla] | Use [argilla.io](https://argilla.io/) for annotating the content of documents. \u003cbr /\u003e [\u003cimg src=\".readme_resources/integration_argilla.png\" width=\"300px\" /\u003e][integration_argilla] |\n\n\n\n## Example dependencies\n\nThe examples contained in this catalog depend on the `deepsearch-toolkit` as well as\nother modules needed for the showcase demonstrated (e.g. `pandas`, `matplotlib`, `rdkit`, etc).\nPlease refer to the poetry `pyproject.toml` or  `requirements.txt` for a complete list.\n\nPython dependencies are installed with\n\n```console\npip install -r requirements.txt\n```\n\nAdditionally, some examples rely on system packages. When this is the case, the README of the individual\nexample will contain more details on which package is required.\nThe auxiliary file [apt.txt](./apt.txt) list all such packages for a Debian-bases OS. They can be installed with\n\n```console\nxargs sudo apt-get install \u003c apt.txt\n```\n\n### Windows compatibility\n\nNote that some examples require dependencies that are not available on Windows platform. We flagged those examples with an asterisk `*` in the index above.\n\n\n## License\n\nThe `Deep Search Toolkit` codebase is under MIT license.\nFor individual model usage, please refer to the model licenses found in the original packages.\n\n[doc_conv_quick_start]: ./examples/document_conversion_quick_start/\n[doc_conv_custom_pipeline]: ./examples/document_conversion_custom_settings/\n[doc_conv_visualize_bbox]: ./examples/document_conversion_visualize_bbox/\n[doc_conv_extract_figures]: ./examples/document_conversion_extract_figures/\n[doc_conv_extract_tables]: ./examples/document_conversion_extract_tables/\n[data_query_quick_start]: ./examples/data_query_quick_start/\n[data_query_snippets]: ./examples/data_query_snippets/\n[data_extract_figures]: ./examples/data_extract_figures/\n[data_query_chemistry]: ./examples/data_query_chemistry/\n[data_query_chemistry_patcid]: ./examples/data_query_chemistry_patcid/\n[qa_quick_start]: ./examples/qa/qa_quick_start.ipynb\n[qa_deep_dive]: ./examples/qa/qa_deep_dive.ipynb\n[qa_doc_collection]: ./examples/qa_doc_collection\n[bring_your_own_pdf]: ./examples/bring_your_own_pdf/\n[bring_your_own_converted_documents]: ./bring_your_own_converted_documents/\n[manage_attachments]: ./examples/attachment_management/\n[kg_download_quick_start]: ./examples/kg_download_quick_start/\n[integration_argilla]: ./examples/integration_argilla/\n[json_export]: ./examples/json_export/\n[profiles]: https://ds4sd.github.io/deepsearch-toolkit/#set-up-your-profile\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds4sd%2Fdeepsearch-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fds4sd%2Fdeepsearch-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds4sd%2Fdeepsearch-examples/lists"}