{"id":26356623,"url":"https://github.com/ds4sd/deepsearch-glm","last_synced_at":"2025-04-06T16:10:29.636Z","repository":{"id":109427394,"uuid":"595117796","full_name":"DS4SD/deepsearch-glm","owner":"DS4SD","description":"Create fast graph language models from converted PDF documents for knowledge extraction and Q\u0026A.","archived":false,"fork":false,"pushed_at":"2025-01-27T04:19:26.000Z","size":45791,"stargazers_count":48,"open_issues_count":4,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-30T15:07:19.731Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DS4SD.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-30T12:39:31.000Z","updated_at":"2025-03-13T20:57:02.000Z","dependencies_parsed_at":"2024-09-09T16:58:44.650Z","dependency_job_id":"d21441e6-5dc1-488d-a36a-582318154c57","html_url":"https://github.com/DS4SD/deepsearch-glm","commit_stats":null,"previous_names":[],"tags_count":77,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-glm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-glm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-glm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DS4SD%2Fdeepsearch-glm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DS4SD","download_url":"https://codeload.github.com/DS4SD/deepsearch-glm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247509221,"owners_count":20950232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-16T13:31:24.428Z","updated_at":"2025-04-06T16:10:29.618Z","avatar_url":"https://github.com/DS4SD.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Graph Language Models\n\n[![PyPI version](https://img.shields.io/pypi/v/deepsearch-glm)](https://pypi.org/project/deepsearch-glm/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/deepsearch-glm)](https://pypi.org/project/deepsearch-glm/)\n[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Pybind11](https://img.shields.io/badge/build-pybind11-blue)](https://github.com/pybind/pybind11/)\n[![Platforms](https://img.shields.io/badge/platform-macos%20|%20linux%20|%20windows-blue)](https://github.com/DS4SD/deepsearch-glm/)\n[![License MIT](https://img.shields.io/github/license/DS4SD/deepsearch-glm)](https://opensource.org/licenses/MIT)\n![PyPI - Downloads](https://img.shields.io/pypi/dm/deepsearch-glm)\n\n## Getting Started\n\n### Finding entities and relations via NLP on text and documents\n\nTo get easily started, simply install the `deepsearch-glm` package from PyPi. This can be\ndone using the traditional `pip install deepsearch-glm` or via poetry `poetry add deepsearch-glm`.\n\nBelow, you can find the code-snippet to process pieces of text,\n\n```python\nfrom deepsearch_glm.utils.load_pretrained_models import load_pretrained_nlp_models\nfrom deepsearch_glm.nlp_utils import init_nlp_model, print_on_shell\n\nload_pretrained_nlp_models(force=False, verbose=False)\nmdl = init_nlp_model()\n\n# from Wikipedia (https://en.wikipedia.org/wiki/France)\ntext = \"\"\"\nFrance (French: [fʁɑ̃s] Listen), officially the French Republic\n(French: République française [ʁepyblik fʁɑ̃sɛz]),[14] is a country\nlocated primarily in Western Europe. It also includes overseas regions\nand territories in the Americas and the Atlantic, Pacific and Indian\nOceans,[XII] giving it one of the largest discontiguous exclusive\neconomic zones in the world.\n\"\"\"\n\nres = mdl.apply_on_text(text)\nprint_on_shell(text, res)\n```\n\nThe last command will print the pandas dataframes on the shell and provides the\nfollowing output,\n\n```sh\ntext:\n\n   #France (French: [fʁɑ̃s] Listen), officially the French Republic\n(French: République française [ʁepyblik fʁɑ̃sɛz]),[14] is a country\nlocated primarily in Western Europe. It also includes overseas regions\nand territories in the Americas and the Atlantic, Pacific and Indian\nOceans, giving it one of the largest discontiguous exclusive economic\nzones in the world.\n\nproperties:\n\n         type label  confidence\n0  language    en    0.897559\n\ninstances:\n\n  type         subtype               subj_path      char_i    char_j  original\n-----------  --------------------  -----------  --------  --------  ---------------------------------------------------------------------\nsentence                           #                   1       180  France (French: [fʁɑ̃s] Listen), officially the French Republic\n                                                                    (French: République française [ʁepyblik fʁɑ̃sɛz]),[14] is a country\n                                                                    located primarily in Western Europe.\nterm         single-term           #                   1         8  #France\nexpression   wtoken-concatenation  #                   1         8  #France\nparenthesis  round brackets        #                   9        36  (French: [fʁɑ̃s] Listen)\nexpression   wtoken-concatenation  #                  18        28  [fʁɑ̃s]\nterm         single-term           #                  29        35  Listen\nterm         single-term           #                  53        68  French Republic\nparenthesis  round brackets        #                  69       125  (French: République française [ʁepyblik fʁɑ̃sɛz])\nterm         single-term           #                  78       100  République française\nterm         single-term           #                 112       124  fʁɑ̃sɛz]\nparenthesis  reference             #                 126       130  [14]\nnumval       ival                  #                 127       129  14\nterm         single-term           #                 136       143  country\nterm         single-term           #                 165       179  Western Europe\nsentence                           #                 181       373  It also includes overseas regions and territories in the Americas and\n                                                                    the Atlantic, Pacific and Indian Oceans, giving it one of the largest\n                                                                    discontiguous exclusive economic zones in the world.\nterm         single-term           #                 198       214  overseas regions\nterm         enum-term-mark-3      #                 207       230  regions and territories\nterm         single-term           #                 219       230  territories\nterm         single-term           #                 238       246  Americas\nterm         enum-term-mark-4      #                 255       290  Atlantic, Pacific and Indian Oceans\nterm         single-term           #                 255       263  Atlantic\nterm         single-term           #                 265       272  Pacific\nterm         single-term           #                 277       290  Indian Oceans\nterm         single-term           #                 313       359  largest discontiguous exclusive economic zones\nterm         single-term           #                 367       372  world\n```\n\nThe NLP can also be applied on entire documents which were converted using\nDeep Search. A simple example is shown below,\n\n```python\nfrom deepsearch_glm.utils.load_pretrained_models import load_pretrained_nlp_models\nfrom deepsearch_glm.nlp_utils import init_nlp_model, print_on_shell\n\nload_pretrained_nlp_models(force=False, verbose=False)\nmdl = init_nlp_model()\n\nwith open(\"\u003cpath-to-json-file-of-converted-pdf-doc\u003e\", \"r\") as fr:\n    doc = json.load(fr)\n\nenriched_doc = mdl.apply_on_doc(doc)\n```\n\n### Creating Graphs from NLP entities and relations in document collections\n\nTo create graphs, you need two ingredients, namely,\n\n1. a collection of text or documents\n2. a set of NLP models that provide entities and relations\n\nBelow is a code snippet to create the graph using these basic ingredients,\n\n```python\nodir = \"\u003couput-dir-to-save-graph\u003e\"\njson_files = [\"json-file of converted PDF document\"]\nmodel_names = \"\u003clist of NLP models:langauge;term;verb;abbreviation\u003e\"\n\nglm = create_glm_from_docs(odir, json_files, model_names)\t\n```\n\n### Querying Graphs \n\nTBD\n\n## Install for development\n\n### Python installation\n\nTo use the python interface, first make sure all dependencies are installed. We use [poetry](https://python-poetry.org/docs/)\nfor that. To install all the dependent python packages and get the python bindings, simply execute,\n\n```sh\npoetry install --all-extras\n```\n\n### CXX compilation\n\nTo compile from scratch, simply run the following command in the `deepsearch-glm` root folder to\ncreate the `build` directory,\n\n```sh\ncmake -B ./build; \n```\n\nNext, compile the code from scratch,\n\n```sh\ncmake --build ./build -j\n```\n\n## Run using the Python Interface\n\n### NLP and GLM examples\n\n_Note: Some of the examples require to convert PDF documents with Deep Search. For this to run, it is required to install the [deepsearch-toolkit](https://github.com/DS4SD/deepsearch-toolkit) package. You can install it with `pip install deepsearch-glm[toolkit]`._\n\nTo run the examples, simply do execute the scripts as `poetry run python \u003cscript\u003e \u003cinput\u003e`. Examples are,\n\n1. **apply NLP on document(s)**\n```sh\npoetry run python ./deepsearch_glm/nlp_apply_on_docs.py --pdf './data/documents/articles/2305.*.pdf' --models 'language;term'\n```\n2. **analyse NLP on document(s)**\n```sh\npoetry run python ./deepsearch_glm/nlp_apply_on_docs.py --json './data/documents/articles/2305.*.nlp.json' \n```\n3. **create GLM from document(s)**\n```sh\npoetry run python ./deepsearch_glm/glm_create_from_docs.py --pdf ./data/documents/reports/2022-ibm-annual-report.pdf\n```\n\n### Deep Search utilities\n\nTo use the Deep Search capabilities, it is required to install the [deepsearch-toolkit](https://github.com/DS4SD/deepsearch-toolkit) package.\nYou can install it with `pip install deepsearch-glm[toolkit]`.\n\n\n1. **Query and download document(s)**\n```sh\npoetry run python ./deepsearch_glm/utils/ds_query.py --index patent-uspto --query \"\\\"global warming potential\\\" AND \\\"etching\\\"\"\n```\n2. **Converting PDF document(s) into JSON**\n```sh\npoetry run python ./deepsearch_glm/utils/ds_convert.py --pdf './data/documents/articles/2305.*.pdf'\"\n```\n\n## Run using CXX executables\n\nIf you like to be bare-bones, you can also use the executables for NLP and GLM's directly. In general, we\nfollow a simple scheme of the form\n\n```sh\n./nlp.exe -m \u003cmode\u003e -c \u003cJSON-config file\u003e\n./glm.exe -m \u003cmode\u003e -c \u003cJSON-config file\u003e\n```\n\nIn both cases, the modes can be queried directly via the `-h` or `--help`\n\n```sh\n./nlp.exe -h\n./glm.exe -h\n```\n\nand the configuration files can be generated,\n\n```sh\n./nlp.exe -m create-configs\n./glm.exe -m create-configs\n```\n\n### Natural Language Processing (NLP)\n\nAfter you have generated the configuration files (see above), you can\n\n1. train simple NLP models\n```sh\n./nlp.exe -m train -c nlp_train_config.json\n```\n2. leverage pre-trained models\n```sh\n./nlp.exe -m predict -c nlp_predict.example.json\n```\n\n### Graph Language Models (GLM)\n\n1. create a GLM\n```sh\n./glm.exe -m create -c glm_config_create.json\n```\n2. explore interactively the GLM\n```sh\n./glm.exe -m explore -c glm_config_explore.json\n```\n\n## Testing\n\nTo run the tests, simply execute (after installation),\n\n```sh\npoetry run pytest ./tests -vvv -s\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds4sd%2Fdeepsearch-glm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fds4sd%2Fdeepsearch-glm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds4sd%2Fdeepsearch-glm/lists"}