{"id":26829808,"url":"https://github.com/scai-bio/datastew","last_synced_at":"2025-08-05T04:24:56.847Z","repository":{"id":246754201,"uuid":"822570156","full_name":"SCAI-BIO/datastew","owner":"SCAI-BIO","description":"Python library for intelligent data stewardship using Large Language Model (LLM) embeddings","archived":false,"fork":false,"pushed_at":"2025-04-29T13:47:46.000Z","size":1421,"stargazers_count":5,"open_issues_count":10,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-29T14:50:48.623Z","etag":null,"topics":["data-harmonization","data-stewardship","large-language-models"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/datastew/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SCAI-BIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-01T11:54:23.000Z","updated_at":"2025-04-29T13:47:47.000Z","dependencies_parsed_at":"2024-10-24T21:29:51.677Z","dependency_job_id":"237b8444-5673-4fa0-80ed-3ca4d70710b9","html_url":"https://github.com/SCAI-BIO/datastew","commit_stats":null,"previous_names":["scai-bio/datastew"],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCAI-BIO%2Fdatastew","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCAI-BIO%2Fdatastew/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCAI-BIO%2Fdatastew/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCAI-BIO%2Fdatastew/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SCAI-BIO","download_url":"https://codeload.github.com/SCAI-BIO/datastew/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251526080,"owners_count":21603461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-harmonization","data-stewardship","large-language-models"],"created_at":"2025-03-30T13:19:15.621Z","updated_at":"2025-04-30T06:07:33.583Z","avatar_url":"https://github.com/SCAI-BIO.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# datastew\n\n![tests](https://github.com/SCAI-BIO/datastew/actions/workflows/tests.yml/badge.svg) ![GitHub Release](https://img.shields.io/github/v/release/SCAI-BIO/datastew)\n\nDatastew is a python library for intelligent data harmonization using Large Language Model (LLM) vector embeddings.\n\n## Installation\n\n```bash\npip install datastew\n```\n\n## Usage\n\n### Harmonizing excel/csv resources\n\nYou can directly import common data models, terminology sources or data dictionaries for harmonization directly from a\ncsv, tsv or excel file. An example how to match two separate variable descriptions is shown in\n[datastew/scripts/mapping_excel_example.py](datastew/scripts/mapping_excel_example.py):\n\n```python\nfrom datastew.process.parsing import DataDictionarySource\nfrom datastew.process.mapping import map_dictionary_to_dictionary\n\n# Variable and description refer to the corresponding column names in your excel sheet\nsource = DataDictionarySource(\"source.xlxs\", variable_field=\"var\", description_field=\"desc\")\ntarget = DataDictionarySource(\"target.xlxs\", variable_field=\"var\", description_field=\"desc\")\n\ndf = map_dictionary_to_dictionary(source, target)\ndf.to_excel(\"result.xlxs\")\n```\n\nThe resulting file contains the pairwise variable mapping based on the closest similarity for all possible matches\nas well as a similarity measure per row.\n\nPer default this will use the local MiniLM model, which may not yield the optimal performance. If you got an OpenAI API\nkey it is possible to use their embedding API instead. To use your key, create a Vectorizer model and pass it to the\nfunction:\n\n```python\nfrom datastew.embedding import Vectorizer\nfrom datastew.process.mapping import map_dictionary_to_dictionary\n\nvectorizer = Vectorizer(\"text-embedding-ada-002\", key=\"your_api_key\")\ndf = map_dictionary_to_dictionary(source, target, vectorizer=vectorizer)\n```\n\n---\n\n### Creating and using stored mappings\n\nA simple example how to initialize an in memory database and compute a similarity mapping is shown in\n[datastew/scripts/mapping_db_example.py](datastew/scripts/mapping_db_example.py):\n\n1) Initialize the repository and embedding model:\n\n    ```python\n    from datastew.embedding import Vectorizer\n    from datastew.repository import WeaviateRepository\n    from datastew.repository.model import Terminology, Concept, Mapping\n\n    repository = WeaviateRepository(mode='remote', path='localhost', port=8080)\n    vectorizer = Vectorizer()\n    # vectorizer = Vectorizer(\"text-embedding-ada-002\", key=\"your_key\") # Use this line for higher accuracy if you have an OpenAI API key\n    ```\n\n2) Create a baseline of data to map to in the initialized repository. Text gets attached to any unique concept of an\nexisting or custom vocabulary or terminology namespace in the form of a mapping object containing the text, embedding,\nand the name of sentence embedder used. Multiple Mapping objects with textually different but semantically equal\ndescriptions can point to the same Concept.\n\n    ```python\n    terminology = Terminology(\"snomed CT\", \"SNOMED\")\n\n    text1 = \"Diabetes mellitus (disorder)\"\n    concept1 = Concept(terminology, text1, \"Concept ID: 11893007\")\n    mapping1 = Mapping(concept1, text1, vectorizer.get_embedding(text1), vectorizer.model_name)\n\n    text2 = \"Hypertension (disorder)\"\n    concept2 = Concept(terminology, text2, \"Concept ID: 73211009\")\n    mapping2 = Mapping(concept2, text2, vectorizer.get_embedding(text2), vectorizer.model_name)\n\n    repository.store_all([terminology, concept1, mapping1, concept2, mapping2])\n    ```\n\n3) Retrieve the closest mappings and their similarities for a given text:\n\n```python\ntext_to_map = \"Sugar sickness\" # Semantically similar to \"Diabetes mellitus (disorder)\"\nembedding = vectorizer.get_embedding(text_to_map)\n\nresults = repository.get_closest_mappings(embedding, similarities=True, limit=2)\n\nfor result in results:\n    print(result)\n```\n\noutput:\n\n```python\nsnomed CT \u003e Concept ID: 11893007 : Diabetes mellitus (disorder) | Diabetes mellitus (disorder) | Similarity: 0.4735338091850281\nsnomed CT \u003e Concept ID: 73211009 : Hypertension (disorder) | Hypertension (disorder) | Similarity: 0.2003161907196045\n```\n\nYou can also import data from file sources (csv, tsv, xlsx) or from a public API like OLS. An example script to\ndownload \u0026 compute embeddings for SNOMED from ebi OLS can be found in\n[datastew/scripts/ols_snomed_retrieval.py](datastew/scripts/ols_snomed_retrieval.py).\n\n---\n\n### Embedding visualization\n\nYou can visualize the embedding space of multiple data dictionary sources with t-SNE plots utilizing different\nlanguage models. An example how to generate a t-sne plot is shown in\n[datastew/scripts/tsne_visualization.py](datastew/scripts/tsne_visualization.py):\n\n```python\nfrom datastew.embedding import Vectorizer\nfrom datastew.process.parsing import DataDictionarySource\nfrom datastew.visualisation import plot_embeddings\n\n# Variable and description refer to the corresponding column names in your excel sheet\ndata_dictionary_source_1 = DataDictionarySource(\"source1.xlsx\", variable_field=\"var\", description_field=\"desc\")\ndata_dictionary_source_2 = DataDictionarySource(\"source2.xlsx\", variable_field=\"var\", description_field=\"desc\")\n\nvectorizer = Vectorizer()\nplot_embeddings([data_dictionary_source_1, data_dictionary_source_2], vectorizer=vectorizer)\n```\n\n![t-SNE plot](./docs/tsne_plot.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscai-bio%2Fdatastew","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscai-bio%2Fdatastew","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscai-bio%2Fdatastew/lists"}