https://github.com/ds4sd/deepsearch-examples
Examples using the Deep Search functionalities
https://github.com/ds4sd/deepsearch-examples
Last synced: about 1 year ago
JSON representation
Examples using the Deep Search functionalities
- Host: GitHub
- URL: https://github.com/ds4sd/deepsearch-examples
- Owner: DS4SD
- License: mit
- Created: 2022-08-02T09:44:30.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2024-05-22T14:07:21.000Z (about 2 years ago)
- Last Synced: 2024-05-22T14:52:42.166Z (about 2 years ago)
- Language: Python
- Size: 16.3 MB
- Stars: 31
- Watchers: 1
- Forks: 13
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Deep Search Examples
[](https://mybinder.org/v2/gh/DS4SD/deepsearch-examples/main)
[](https://opensource.org/licenses/MIT)
[](https://github.com/psf/black)
[](https://ds4sd.github.io/deepsearch-toolkit/)
[](https://mybinder.org/v2/gh/DS4SD/deepsearch-examples/main)
In this repository we showcase some common usage of Deep Search
for **Document conversion** as well as **Data and Knowledge exploration**.
## Quick links
- [Deep Search Toolkit](https://github.com/ds4sd/deepsearch-toolkit)
- [Documentation](https://ds4sd.github.io/deepsearch-toolkit/)
## Examples
### Setup and usage
Each example starts by defining its input parameters. This is supported by Pydantic
Settings, allowing automated loading from a `.env` file or env vars. Furthermore, access
is based on [Profiles][profiles]. Unless otherwise configured, the profile used is the active one.
### Document conversion
| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [Convert documents quick start][doc_conv_quick_start] | Full example on programmatic document conversion
[
][doc_conv_quick_start] |
| 2. | [Convert documents with custom settings][doc_conv_custom_pipeline] | Full example on programmatic document conversion with custom conversion settings
[][doc_conv_custom_pipeline] |
| 3. | [Visualize bounding boxes][doc_conv_visualize_bbox] | Visualize the bbox of the text elements
[
][doc_conv_visualize_bbox] |
| 4. | [Extract figures from documents][doc_conv_extract_figures] | Given a PDF file, extract the figures
[
][doc_conv_extract_figures] |
| 5. | [Extract tables][doc_conv_extract_tables] | Given a PDF file, extract the tables
[
][doc_conv_extract_tables] |
### NLP on Documents
| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [NLP on documents](examples/nlp_on_documents/nlp_on_documents.ipynb)* | A few quick examples on how to apply NLP models on documents (eg extracting key-terms) |
| 2. | [Reference Parsing](examples/nlp_for_references/nlp_for_references.ipynb)* | Examples on how to parse references from Documents |
| 3. | [Material Extraction](examples/nlp_for_materials/nlp_for_materials.ipynb)* | Examples on how to extract materials from Documents |
### Data queries
This section will showcase examples which query data processed via Deep Search.
| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [Data query quick start][data_query_quick_start] | Example listing data collections, making search in one and more document collections, using source for projection
[
][data_query_quick_start] |
| 2. | [Chemistry search queries][data_query_chemistry] | Search the chemistry databases for known molecules
[
][data_query_chemistry] |
| 3. | [Chemistry and patent searches via PatCID][data_query_chemistry_patcid] | Explore the chemistry databases using substructure and similarity searches and navigate to the world-wide patents which reference molecules
[
][data_query_chemistry] |
| 4. | [Snippets and aggregations in data queries][data_query_snippets] | Extract snippets in search queries and leverage aggregations for exploratory analysis
[
][data_query_snippets] |
### Semantic queries & RAG
This section will showcase examples of semantic capabilitilies in the area of Q&A using RAG.
| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [QA quick start][qa_quick_start] | Get started with semantic ingestion, RAG, and retrieval.
[
][qa_quick_start] |
| 2. | [QA deep dive][qa_deep_dive] | Explore advanced RAG and semantic retrieval capabilities.
[
][qa_deep_dive] |
### Bring your own
This section will showcase examples for bringing your own documents, csv data, nlp models and more.
| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [Bring your own PDF][bring_your_own_pdf] | Upload your own PDF documents, search on them and export the result as JSON files.
[
][bring_your_own_pdf] |
| 2. | [Bring your own converted documents][bring_your_own_converted_documents] | Upload your documents already formatted as JSON.
[
][bring_your_own_converted_documents] |
| 3. | Bring your own DataFrame | Bring your own DataFrame from CSV, XLSX, etc and explore the content in a knowledge graph
|
### Attachments and metadata
This section will showcase examples for managing index item attachments and metadata.
| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [Manage attachments][manage_attachments] | Manage index item attachments
[
][manage_attachments] |
### Knowledge graphs
This section will showcase examples related to the use of knowledge graphs (KGs) in Deep Search.
| | Name | Description |
| -- | ------------------ | ----------- |
| 1. | [Using Deep Search KGs with PyTorch Geometric][kg_download_quick_start] | Download knowledge graphs from Deep Search and import them in PyTorch Geometric.
[
][kg_download_quick_start] |
### Integrations
This section will showcase examples related to the integration of Deep Search with other tools and utilities.
| | Name | Description |
| -- | ------------------ | ----------- |
| 1. | [Annotations on argilla.io][integration_argilla] | Use [argilla.io](https://argilla.io/) for annotating the content of documents.
[
][integration_argilla] |
## Example dependencies
The examples contained in this catalog depend on the `deepsearch-toolkit` as well as
other modules needed for the showcase demonstrated (e.g. `pandas`, `matplotlib`, `rdkit`, etc).
Please refer to the poetry `pyproject.toml` or `requirements.txt` for a complete list.
Python dependencies are installed with
```console
pip install -r requirements.txt
```
Additionally, some examples rely on system packages. When this is the case, the README of the individual
example will contain more details on which package is required.
The auxiliary file [apt.txt](./apt.txt) list all such packages for a Debian-bases OS. They can be installed with
```console
xargs sudo apt-get install < apt.txt
```
### Windows compatibility
Note that some examples require dependencies that are not available on Windows platform. We flagged those examples with an asterisk `*` in the index above.
## License
The `Deep Search Toolkit` codebase is under MIT license.
For individual model usage, please refer to the model licenses found in the original packages.
[doc_conv_quick_start]: ./examples/document_conversion_quick_start/
[doc_conv_custom_pipeline]: ./examples/document_conversion_custom_settings/
[doc_conv_visualize_bbox]: ./examples/document_conversion_visualize_bbox/
[doc_conv_extract_figures]: ./examples/document_conversion_extract_figures/
[doc_conv_extract_tables]: ./examples/document_conversion_extract_tables/
[data_query_quick_start]: ./examples/data_query_quick_start/
[data_query_snippets]: ./examples/data_query_snippets/
[data_extract_figures]: ./examples/data_extract_figures/
[data_query_chemistry]: ./examples/data_query_chemistry/
[data_query_chemistry_patcid]: ./examples/data_query_chemistry_patcid/
[qa_quick_start]: ./examples/qa/qa_quick_start.ipynb
[qa_deep_dive]: ./examples/qa/qa_deep_dive.ipynb
[qa_doc_collection]: ./examples/qa_doc_collection
[bring_your_own_pdf]: ./examples/bring_your_own_pdf/
[bring_your_own_converted_documents]: ./bring_your_own_converted_documents/
[manage_attachments]: ./examples/attachment_management/
[kg_download_quick_start]: ./examples/kg_download_quick_start/
[integration_argilla]: ./examples/integration_argilla/
[json_export]: ./examples/json_export/
[profiles]: https://ds4sd.github.io/deepsearch-toolkit/#set-up-your-profile