{"id":25166007,"url":"https://github.com/zbmed-semtec/doc2vec-doc-relevance","last_synced_at":"2025-04-03T17:23:09.714Z","repository":{"id":229213754,"uuid":"477695471","full_name":"zbmed-semtec/doc2vec-doc-relevance","owner":"zbmed-semtec","description":"An approach exploring and assessing literature-based doc-2-doc recommendations using a doc2vec and applying to the RELISH dataset.","archived":false,"fork":false,"pushed_at":"2025-01-23T22:02:58.000Z","size":10013,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-09T05:47:24.042Z","etag":null,"topics":["doc2doc-relevance","document-embeddings","document-similarity","ontoclue","phase-one","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zbmed-semtec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":"codemeta.json"}},"created_at":"2022-04-04T12:42:04.000Z","updated_at":"2025-01-23T22:03:00.000Z","dependencies_parsed_at":"2025-01-23T23:18:27.433Z","dependency_job_id":"28af6e35-8949-4885-9e43-4c688944a471","html_url":"https://github.com/zbmed-semtec/doc2vec-doc-relevance","commit_stats":null,"previous_names":["zbmed-semtec/doc2vec-doc-relevance"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zbmed-semtec%2Fdoc2vec-doc-relevance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zbmed-semtec%2Fdoc2vec-doc-relevance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zbmed-semtec%2Fdoc2vec-doc-relevance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zbmed-semtec%2Fdoc2vec-doc-relevance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zbmed-semtec","download_url":"https://codeload.github.com/zbmed-semtec/doc2vec-doc-relevance/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247043815,"owners_count":20874187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["doc2doc-relevance","document-embeddings","document-similarity","ontoclue","phase-one","python"],"created_at":"2025-02-09T05:47:28.704Z","updated_at":"2025-04-03T17:23:09.691Z","avatar_url":"https://github.com/zbmed-semtec.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)\n\n\n[![SWH](https://archive.softwareheritage.org/badge/swh:1:dir:5939540bdda33571b0969e520360b80d98b2bf2b/)](https://archive.softwareheritage.org/swh:1:dir:5939540bdda33571b0969e520360b80d98b2bf2b;origin=https://github.com/zbmed-semtec/doc2vec-doc-relevance;visit=swh:1:snp:876472bcf3941f5ccde23d20465f2ece7ae1d80f;anchor=swh:1:rev:20b6036fd63891bef25d6af364c6b17066a86416)\n\n\n# Doc2Vec-Doc-relevance\nThis repository focuses on an approach exploring and assessing literature-based doc-2-doc recommendations using the Doc2Vec technique with its application to the RELISH dataset. \n\n\n## Table of Contents\n\n1. [About](#about)\n2. [Input Data](#input-data)\n3. [Pipeline](#pipeline)\n    1. [Generate Embeddings](#generate-embeddings)\n        - [Create Tagged Documents](#create-tagged-documents)\n        - [Generate and Train Doc2Vec models](#generate-and-train-doc2vec-models)\n          - [Parameters](#parameters)\n    2. [Format embeddings](#format-embeddings)\n    3. [Calculate Cosine Similarity](#calculate-cosine-similarity)\n    4. [Evaluation](#evaluation)\n        - [Precision@N](#precisionn)\n        - [nDCG@N](#ndcgn)\n7. [Code Implementation](#code-implementation)\n8. [Getting Started](#getting-started)\n9. [Tutorial](#tutorial)\n\n\n\n## About\n\n Our approach involves employing the [doc2vec](https://arxiv.org/pdf/1405.4053v2.pdf) model, which extends the popular word2vec technique to capture document-level semantics. By encoding documents and their textual content into fixed-length vectors, doc2vec facilitates similarity calculations and enables meaningful comparisons between documents. This approach is harnessed to derive insightful doc-2-doc recommendations within the realm of biomedical research, specifically employing the RELISH dataset. In order to do so, we employ the [doc2vec model](https://radimrehurek.com/gensim/models/doc2vec.html) from the [Gensim](https://radimrehurek.com/gensim/index.html) library.\n\n## Input Data\n\nThe input data for this method consists of preprocessed tokens derived from the RELISH documents. These tokens are stored in the **RELISH.npy file**, which contains preprocessed arrays comprising PMIDs, document titles, and abstracts. These arrays are generated through an extensive preprocessing pipeline, as elaborated in the [relish-preprocessing repository](https://github.com/zbmed-semtec/relish-preprocessing). Within this preprocessing pipeline, both the title and abstract texts undergo several stages of refinement: structural words are eliminated, text is converted to lowercase, and finally, tokenization is employed, resulting in arrays of individual words.\n\n## Pipeline\n\nThis section outlines the progression from generating document embeddings to conducting hyperparameter optimization and ultimately evaluating the effectiveness of the approach.\n\n### Generate Embeddings\nThe following section outlines the process of generating document-level embeddings for each PMID of the RELISH corpus.\n\n#### Create Tagged Documents \nIn this initial step, we create  `TaggedDocuments `, which associate each PMID with a corresponding list of words. Here, we combine the abstract and title of each document into a unified paragraph (or document). This unified text serves as the input for our Doc2Vec model, allowing it to capture the semantic meaning of the entire document.\n#### Generate and Train Doc2Vec models \nIn the second phase, we construct and train Doc2Vec models with customizable hyperparameters. These models are designed to understand the relationships between documents and words in a high-dimensional vector space. We employ the parameters shown below in order to generate our models.\n##### Parameters\n\n+ **dm:** {1,0} Refers to the training algorithm. If dm=1, distributed memory is used otherwise, distributed bag of words is used.\n+ **vector_size:** It represents the dimensions of the generated embeddings, with options of 200, 300 and 400 in our case.\n+ **window:** Represents the maximum distance between the current and predicted word, with values fof 5,6 and 7 in our case.\n+ **epochs:** Refers to the number of iterations over the training dataseta and is set at 15 in this context.\n+ **min_count:** It is the minimum number of appearances a word must have to not be ignored by the algorithm and is configured at a minimum of 5.\n\n### Format embeddings\nAfter model training, we can extract document-level embeddings. These embeddings are numerical vectors that represent the content and context of each document in a continuous vector space. These embeddings are stored by the model, associated with each PMID. For further downstream document similarity calculations, we format and save these embeddings for each document with its PMID as a dataframe in a pickle file. Each specific set of hyperparameter combination results in having a separate pickle file.\n\n## Calculate Cosine Similarity\nTo assess the similarity between two documents within the RELISH corpus, we employ the Cosine Similarity metric. This process enables the generation of a 4-column matrix containing cosine similarity scores for existing pairs of PMIDs within our corpus. For a more detailed explanation of the process, please refer to this [documentation](https://github.com/zbmed-semtec/medline-preprocessing/tree/main/code/Cosine_Similarity).\n\n## Evaluation\n\n### Precision@N\n\nIn order to evaluate the effectiveness of this approach, we make use of Precision@N. Precision@N measures the precision of retrieved documents at various cutoff points (N).We generate a Precision@N matrix for existing pairs of documents within the RELISH corpus, based on the original RELISH JSON file. The code determines the number of true positives within the top N pairs and computes Precision@N scores. The result is a Precision@N matrix with values at different cutoff points, including average scores. For detailed insights into the algorithm, please refer to this [documentation](https://github.com/zbmed-semtec/medline-preprocessing/tree/main/code/Precision%40N_existing_pairs).\n\n\n### nDCG@N\n\nAnother metric used is the nDCG@N (normalized Discounted Cumulative Gain). This ranking metric assesses document retrieval quality by considering both relevance and document ranking. It operates by using a TSV file containing relevance and cosine similarity scores, involving the computation of DCG@N and iDCG@N scores. The result is an nDCG@N matrix for various cutoff values (N) and each PMID in the corpus, with detailed information available in the [documentation](https://github.com/zbmed-semtec/medline-preprocessing/tree/main/code/Evaluation).\n\n## Code Implementation\n\n\nThe [`run_embeddings.py`](./code/generate_embeddings/run_embeddings.py) serves as a comprehensive wrapper function, supporting the creation of tagged documents, model generation, training, embedding generation, and the subsequent storage of these embeddings as pickle files. Individual functions for each task are provided in the other two code scripts:\n\n+ [`embeddings.py`](./code/generate_embeddings/embeddings.py) : Creation of tagged documents from input tokens, creation and training of Doc2Vec models, generation of embeddings. \n+ [`embeddings.dataframe.py`](./code/generate_embeddings/embeddings_dataframe.py) : Creates a dataframe of embeddings with its corresponding PMID, sorts and stores it as a pickle file.\n\n\n## Getting Started\n\nTo get started with this project, follow these steps:\n\n### Step 1: Clone the Repository\nFirst, clone the repository to your local machine using the following command:\n\n###### Using HTTP:\n\n```\ngit clone https://github.com/zbmed-semtec/doc2vec-doc-relevance.git\n```\n\n###### Using SSH:\nEnsure you have set up SSH keys in your GitHub account.\n\n```\ngit clone git@github.com:zbmed-semtec/doc2vec-doc-relevance.git\n```\n\n### Step 2: Create a virtual environment and install dependencies\n\nTo create a virtual environment within your repository, run the following command:\n\n```\npython3 -m venv .venv \nsource .venv/bin/activate   # On Windows, use '.venv\\Scripts\\activate'\n\n#If you have any difficulties to activate the env on Windows, try below commands:\n'.venv\\Scripts\\activate.ps1'  # PowerShell\n'.venv\\Scripts\\activate.bat'  # Command prompt\n```\n\nTo confirm if the virtual environment is activated and check the location of yourPython interpreter, run the following command:\n\n```\nwhich python    # On Windows command prompt, use 'where python'\n                # On Windows PowerShell, use 'Get-Command python'\n```\nThe code is stable with python 3.6 and higher. The required python packages are listed in the requirements.txt file. To install the required packages, run the following command:\n\n```\npip install -r requirements.txt\n```\n\nTo deactivate the virtual environment after running the project, run the following command:\n\n```\ndeactivate\n```\n\n### Step 3: Dataset\n\nUse the Download_Dataset.sh script to download the Split Dataset by running the following commands:\n\n\n```\nchmod +777 Download_Dataset.sh\n./Download_Dataset.sh\n```\n\nThis script makes sure that the necessary folders are created and the files are downloaded in the corresponding folders as shown below.\n\n```\n📦 /doc2vec-doc-relevance\n└─ data\n   └─ Input\n      ├─ Tokens\n      │  ├─ relish.npy\n      └─ Ground_truth\n         └─ relevance_matrix.tsv\n```\nThe file *relish.npy* is in the NumPy binary format (.npy), which is specifically used to store NumPy arrays efficiently. These arrays contain the PMID, title, and abstract for each document.\n\nIn contrast, *relevance_matrix.tsv* is a Tab-separated Values file, similar to CSV but using tabs as delimiters. It stores tabular data with four columns: PMID1 | PMID2 | Relevance | WMD Similarity.\n\nReference: Tab-separated values (TSV) file format:  \n[![FAIRsharing DOI](https://img.shields.io/badge/DOI-10.25504%2FFAIRsharing.a978c9-blue)](https://doi.org/10.25504/FAIRsharing.a978c9)\n\n### Step 4: Generate Embeddings\nThe [`embeddings.py`](/code/embeddings.py) script uses the RELISH Tokenized npy file as input. You can easily adapt it for different values and parameters by modifying the [`hyperparameters.yaml`](./code/hyperparameters.yaml) Make sure to have the RELISH Tokenized.npy file within the directory under the data folder.\n\n```\npython3 code/embeddings.py [-i INPUT PATH] [-o OUTPUT PATH] [-p PARAMS]\n```\n\nYou must pass the following arguments:\n\n+ -i/ --input : File path to the RELISH tokenized .npy file.\n+ -o/ --output : File path to the resulting embeddings in pickle file format.\n+ -p/ --params : File path to the hyperparameters YAML file.\n\nTo run this script, please execute the following command:\n\n```\npython3 code/embeddings.py --input data/Input/Tokens/relish.npy --output data/embeddings --params code/hyperparameters.yaml \n```\n\nThe script will create Doc2Vec models, generate embeddings, and store them in separate directories. You should expect to find a total of 18 files corresponding to the various models, embeddings, and embedding pickle files.\n\n### Step 5: Calculate Cosine Similarity\nIn order to generate the cosine similarity matrix and execute this [script](/code/generate_cosine_existing_pairs.py), run the following command:\n\n```\npython3 code/generate_cosine_existing_pairs.py [-i INPUT] [-e EMBEDDINGS] [-o OUTPUT] \n```\n\nYou must pass the following four arguments:\n\n+ -i/ --input : File path to the RELISH relevance matrix in the TSV format.\n+ -e/ --embeddings : File path to the embeddings in the pickle file format.\n+ -o/ --output : File path for the output 4 column cosine similarity matrix.\n\nFor example, if you are running the code from the code folder and have the RELISH relevance matrix in the data folder, run the cosine matrix creation for the first hyperparameter as:\n\n```\npython3 code/generate_cosine_existing_pairs.py -i data/Input/Ground_truth/relevance_matrix.tsv -e data/embeddings/embeddings_0.pkl -o data/cosine/cosine_similarity_0.tsv\n```\n\nNote: You would have to run the above command for every hyperparameter configuration by changing the file name for the embedding's pickle file or use the following shell script to generate all files at once.\n\n```\nfor VALUE in {0..17};do\npython3 code/generate_cosine_existing_pairs.py -i data/Input/Ground_truth/relevance_matrix.tsv -e data/embeddings/embeddings_pickle_${VALUE}.pkl -o data/cosine/cosine_similarity_${VALUE}.tsv\ndone\n```\n\n### Step 6: Precision@N\nIn order to calculate the Precision@N scores and execute this [script](/code/precision.py), run the following command:\n\n```\npython3 code/precision.py [-i COSINE FILE PATH]  [-o OUTPUT PATH]  [-c CLASSES]\n```\n\nYou must pass the following two arguments:\n\n+ -i/ --cosine_file_path: path to the 4-column cosine similarity existing pairs RELISH file: (tsv file)\n+ -o/ --output_path: path to save the generated precision matrix: (tsv file)\n+ -c/ --classes: Number of classes for class distribution (2 or 3)\n\nFor example, if you are running the code from the code folder and have the cosine similarity TSV file in the data folder, run the precision matrix creation for the first hyperparameter as:\n\n```\npython3 code/precision.py -i data/cosine/cosine_similarity_0.tsv -o data/precision_three_classes/precision_0.tsv -c 3\n```\n\nNote: You would have to run the above command for every hyperparameter configuration by changing the file name for the cosine similarity file or use the following shell script to generate all files at once.\n\n\n```\nfor VALUE in {0..17};do\npython3 code/precision.py -c data/cosine_similarity_${VALUE}.tsv -o data/precision_three_classes/precision_${VALUE}.tsv -c 3\ndone\n```\n\nNote: Make sure to re-run the above command by changing the classes for a different class distribution.\n\n### Step 7: nDCG@N\nIn order to calculate nDCG scores and execute this [script](/code/calculate_gain.py), run the following command:\n\n```\npython3 code/calculate_gain.py [-i INPUT]  [-o OUTPUT]\n```\n\nYou must pass the following two arguments:\n\n+ -i / --input: Path to the 4 column cosine similarity existing pairs RELISH TSV file.\n+ -o/ --output: Output path along with the name of the file to save the generated nDCG@N TSV file.\n\nFor example, if you are running the code from the code folder and have the 4 column RELISH TSV file in the data folder, run the matrix creation for the first hyperparameter as:\n\n```\npython3 code/calculate_gain.py -i data/cosine/cosine_similarity_0.tsv -o data/gain/ndcg_0.tsv\n```\n\nNote: You would have to run the above command for every hyperparameter configuration by changing the file name for the cosine similarity file or use the following shell script to generate all files at once.\n\n```\nfor VALUE in {0..17};do\npython3 code/calculate_gain.py -i data/cosine/cosine_similarity_${VALUE}.tsv -o data/gain/ndcg_${VALUE}.tsv\ndone\n```\n\n### Step 8: Compile Results\n\nIn order to compile the average result values for Precison@ and nDCG@N and generate a single TSV file each, please use this [script](code/show_avg.py).\n\nYou must pass the following two arguments:\n\n+ -i / --input: Path to the directory consisting of all the precision matrices/gain matrices.\n+ -o/ --output: Output path along with the name of the file to save the generated compiled Precision@N / nDCG@N TSV file.\n\n\nIf you are running the code from the code folder, run the compilation script as:\n\n```\npython3 code/show_avg.py -i data/gain/ -o data/results_gain.tsv\n```\n\nNOTE: Please do not forget to put a `'/'` at the end of the input file path.\n\n## Tutorial\n+ A [tutorial](./docs/embeddings/) is accessible in the form of Jupyter notebook for the generation of embeddings.\n+ A [tutorial](./docs/cosine_similarity/) is accessible in the form of Jupyter notebook for the computing cosine similarity values.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzbmed-semtec%2Fdoc2vec-doc-relevance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzbmed-semtec%2Fdoc2vec-doc-relevance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzbmed-semtec%2Fdoc2vec-doc-relevance/lists"}