{"id":19863538,"url":"https://github.com/sandialabs/galen-view","last_synced_at":"2025-05-02T04:31:10.644Z","repository":{"id":57432944,"uuid":"259477352","full_name":"sandialabs/galen-view","owner":"sandialabs","description":"Python software for visualizing and exploring the CORD-19 Dataset","archived":false,"fork":false,"pushed_at":"2020-07-15T14:49:55.000Z","size":10472,"stargazers_count":4,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-19T07:50:56.284Z","etag":null,"topics":["scr-2504","snl-data-analysis","snl-visualization"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sandialabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-27T23:07:34.000Z","updated_at":"2024-08-01T13:35:54.000Z","dependencies_parsed_at":"2022-08-27T21:10:54.161Z","dependency_job_id":null,"html_url":"https://github.com/sandialabs/galen-view","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgalen-view","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgalen-view/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgalen-view/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandialabs%2Fgalen-view/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sandialabs","download_url":"https://codeload.github.com/sandialabs/galen-view/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251986784,"owners_count":21675951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scr-2504","snl-data-analysis","snl-visualization"],"created_at":"2024-11-12T15:15:03.928Z","updated_at":"2025-05-02T04:31:05.635Z","avatar_url":"https://github.com/sandialabs.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Description\n\ngalen-view is a visualizer for exploring the CORD-19 dataset on a stand-alone \ndesktop computer.  The code is intended to be small and straight-forward, \nenabling researchers to expand and modify the code.\n\nThis software was written as part of an effort at Sandia National\nLaboratories described\n[here](https://github.com/sandialabs/galen-view/blob/master/SAND2020-4510-RapidResponseDataScienceForCOVID-19.pdf)\n\nThis code is named after Galen, a second century Greek physician who\ndocumented the Antonine Plague in the Roman Empire.\n\n# Installation \n\n## Create a python 3 environment\n\ngalen-view was written and tested in python 3 and uses syntax specific to \npython 3.  To get started, create a python 3 environment.\n\nFor example, if you are using anaconda, you can execute the following:\n\n```bash\nconda create -n galenenv python=3\nconda activate galenenv\n```\n\nIf you are using Anaconda on Windows 10, also type:\n```bash\nconda install holoviews\n```\n\n## Install the wheel file\n\nInstalling the wheel file will install galen-view along with it's various\ndependencies.  galen-view is based on holoviews, so there are a large number\nof dependencies.\n\nThis can be done using:\n\n```bash\npip install galen-view\n```\n\n# Data Download and Configuration\n\nThe module sandia.galen.dataprep contains several methods for downloading, \nanalyzing, and indexing the cord-19 dataset.  For convenience, the module\ncan be executed as an \"all-in-one\" call do perform all these funcions and\nprepare the data for galen-view.  \n\nIf you are behind a proxy, you will need to set the environment variables\nin order to download the data.  This can be accomplished by setting the \nhttp_proxy and https_proxy environment variables.  If the\n\"pip install\" step above worked, it is highly likely that you already have\nthem set properly. Note that this is only necessary for the \ndownload and not for galen-view to work after that.  \n\nThis will download files and create more.  It is recommended to do this\nin a fresh directory.\n\nTo create a new directory:\n\n```base\nmkdir galen_data\ncd galen_data\n```\n\nThen run one of the following two commands:\n\nFor a faster startup time that only analyzes the first 2000 documents\n(You may have to use \"python\" rather than \"python3\"):\n\n```bash\npython3 -m sandia.galen.dataprep test\n```\n\nTo analyze all the downloaded documents, execute the following.  \n\n(You may have to use \"python\" rather than \"python3\").\n\nThis can take hourse, but you only have to do it once:\n\n```bash\npython3 -m sandia.galen.dataprep\n```\n\nYou will see progress during\neach step.  You should see something similar to the following:\n\n```\n$ python3 -m sandia.galen.dataprep\nInitializing Data in Current Directory\nChecking for Data in this Directory\nDownloading the metadata file\nDownloading the CORD-19 Dataset\nProcessing https://path/to/biorxiv_medrxiv.tar.gz ... Done.\nProcessing https://path/to/comm_use_subset.tar.gz ... Done.\nProcessing https://path/to/noncomm_use_subset.tar.gz ... Done.\nProcessing https://path/to/pmc_custom_license.tar.gz ... Done.\nProcessing https:///custom_license.tar.gz ... Done.\nProcessing https:///cord19_specter_embeddings_2020-04-03.tar.gz ... Done.\nProcessing https:///cord19_specter_embeddings_2020-04-10.tar.gz ... Done.\n========\nMaking the Coordinates to Visualize the Data\n========\nBuilding a searchable index from the documents\n..........\n..........\n..........\n[There will be more lines like this]\n..........\n..........\n..........\nDone. Committing Index\nDone Indexing\n========\nDone.  Run \"jupyter-notebook GalenView.ipynb\" and run all cells to explore the data.\n```\n\n# Usage\n\nsandia.galen.dataprep's final step is to copy a jupyter notebook\ninto the directory that can be used to view and explore the data.\nYou can open this notebook in a variety of ways, but the following\nwill open it directly:\n\n```bash\njupyter-notebook GalenView.ipynb\n```\n\nAfter executing this, a web browser should open with the jupyter\nnotebook.  You still have to run all the cells in the notebook to\nactivate and use the visualization.\n\n\u003ctable style=\"width:100%\"\u003e\n\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src=\"https://raw.githubusercontent.com/sandialabs/galen-view/master/other/20200422-GalenView_RunNotebookMenu.png\" alt=\"Run Notebook Menu\"\u003e\n\u003ctd\u003eAfter the notebook loads, click \"Restart \u0026 Runn All\" from the menu at the top of the page.\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src=\"https://raw.githubusercontent.com/sandialabs/galen-view/master/other/20200422-GalenView_RunNotebookButton.png\" alt=\"Run Notebook Button\"\u003e\n\u003ctd\u003eA window will appear prompting you to restart the notebook.  Click the red button.\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\nOn the page, you will now see the full interface.  Each point in the\ngraph represents a document.  Hovering your mouse over a point or\ngroup of points will display the title.  Clicking on a point will\ncause the title and first 1500 characters (if full text is included in\nthe CORD19 dataset) on the right hand side.  Clicking on the\n\"Document\" tab will show the full text of the document.  \n\n\u003ctable style=\"width:100%\"\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src=\"https://raw.githubusercontent.com/sandialabs/galen-view/master/other/20200422-GalenView_Controls.png\" alt=\"Controls\"\u003e\u003c/td\u003e\n\u003ctd\u003eThis interface lets you explore the various clusters of data by panning \nand zooming.  The individual controls are described at the left.\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nYou can type in a query in the Search box and press enter.  The points\ncorresponding to the matching documents will turn into red triangles.\nNote that the documents are indexed using a python library called\n\"whoosh\" which contains a query language that lets you use advanced\nsyntax such as boolean and proximity queries.  More information on\nthat query syntax can be found\n[here](https://whoosh.readthedocs.io/en/latest/querylang.html).\n\nThis following is a short description of each part of the user interface:\n\n\u003cimg src=\"https://raw.githubusercontent.com/sandialabs/galen-view/master/other/20200422-GalenView_InterfaceExplanation.png\" alt=\"interface explanation\"\u003e\n\nIn the above example, the user has searched for the term \"mers.\"  The\ndocuments that reference \"mers\" are mostly found in a particular\ncluster at the lower left.  Note that because of how the layout\nalgorithm works, the cluster in your interface may be different.  By\nusing the controls, the user can zoom in on that cluster.  The\nfollowing image shows what that might look like.  Note that the \"mers\"\nmatches are in a cluster, but you can see similar clusters nearby.\nThrough hovering, zomming, panning, and clicking, the user can explore\nthe data set.\n\n\u003cimg src=\"https://raw.githubusercontent.com/sandialabs/galen-view/master/other/20200422-GalenView_InterfaceZoomed.png\" alt=\"zoomed interface\"\u003e\n\n# Coloring\n\n(Note: This section requres python development expertise)\n\nIn the paper associated with this codebase, compression-based\nanalytics were used to color the documents according to how well they\nmatched snippets of text provided by subject matter experts.  The code\nto perform the compression-based analytics is not included in this\ncodebase.  However, it is possible to score documents in many\ndifferent ways (even by something like year of publication).  If you\nhave a methodology for scoring the documents, you can get them into \nthe visualization using the following steps:\n\n1. Run the \"Data Download and Configuration\" step above.  When it\nfinishes, there will be a file called \"metadata.df.pickle.\" in the\ndata directory.  This is a pandas DataFrame that can be read using \npandas.read_pickle.  The index (and integer) corresponds with the \ninteger used by cotools in the cord-19-tools and is how the tool \nidentifies the documents.  \n\n2. Create a score per document using whatever means is useful for \nyour subject matter experts.\n\n3. Append those scores as a new column to the DataFrame from step 1.  \nWrite it back out to metadata.df.pickle.\n\n4. Create a Jupyter Notebook and create the visualization using the \nfollowing code (where \"myscore\" is the name of column you added):\n\n```python\nfrom sandia.galen.viewer import DocumentSetExplorer\ndisplay(HTML(\"\u003cstyle\u003e.container { width:100% !important; }\u003c/style\u003e\"))\nhv.extension('bokeh')\nd = DocumentSetExplorer(scoring_cols=[\"myscore\"])\nd.main_viz()\n```\n\nIf you want to test how this works before trying your own score,\nyou can skip steps 2 and 3 and use \"X\" in place of \"myscore.\"  This will\nsimply color the points according to their X value.\n\n# Notes and TOOD\n\n* A large percentage of the documents do not have associated with\n  them.  All documents are arranged in the 2D space using both the\n  title and content.  But the search only searches the content unless\n  you explicitly specify the title (using title:query).  This is by\n  design but can be confusing when navigating the space.  This could\n  be changed in a future version.\n\n* There currently is no way to set aside a document.  This is entirely\n  possible within the holoviews framework but is not yet impelemented.\n\n* It would be useful to include author, date published, journal, and\n  other fields in the metadata.  This information for many of the\n  documents are in the metadata file and could be incorporated in the\n  hover and visualization.\n\n* Links to the original documents that include tables and figures is\n  important.  For the documents that contain pubmed ids in the\n  metadata, this is straight forward to do.\n\n* Implement better parsing of the documents, especially adding an\n  ability to jump directly to different sections, depending on the\n  specific question being asked.  Sometimes, for example, it would be\n  helpful to jump directly to the abstract.  In other cases, jumping\n  directly to the \"methods\" section would be best.  Parsing research\n  papers in this way is hard, but to the extent it can be done, the\n  right hand summary pane could be customizable.\n\n* Add in paragraph spacing to increase readability.  Right now, the\n  document display used that cotools returns.  But the underlying text\n  is generally stored in a way that is better broken up and could be\n  displayed with better paragraph breaks.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsandialabs%2Fgalen-view","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsandialabs%2Fgalen-view","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsandialabs%2Fgalen-view/lists"}