{"id":18345701,"url":"https://github.com/digital-science/dimensions-network-gen","last_synced_at":"2025-04-06T08:31:49.687Z","repository":{"id":53861675,"uuid":"486303131","full_name":"digital-science/dimensions-network-gen","owner":"digital-science","description":"Science mapping tool using Dimensions and Google BigQuery.","archived":false,"fork":false,"pushed_at":"2022-09-22T13:28:33.000Z","size":35162,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-21T20:51:12.934Z","etag":null,"topics":["covid19-data","dimensions","network-analysis","publication-data","python","science-of-science","scientometrics","visualization","vosviewer"],"latest_commit_sha":null,"homepage":"https://digital-science.github.io/dimensions-network-gen/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/digital-science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-27T18:13:20.000Z","updated_at":"2023-10-05T06:56:37.000Z","dependencies_parsed_at":"2023-01-18T18:34:16.332Z","dependency_job_id":null,"html_url":"https://github.com/digital-science/dimensions-network-gen","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digital-science%2Fdimensions-network-gen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digital-science%2Fdimensions-network-gen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digital-science%2Fdimensions-network-gen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digital-science%2Fdimensions-network-gen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/digital-science","download_url":"https://codeload.github.com/digital-science/dimensions-network-gen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247457278,"owners_count":20941900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["covid19-data","dimensions","network-analysis","publication-data","python","science-of-science","scientometrics","visualization","vosviewer"],"created_at":"2024-11-05T21:09:11.050Z","updated_at":"2025-04-06T08:31:45.831Z","avatar_url":"https://github.com/digital-science.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dimensions Network Generation tool\n\nA Python tool that streamlines the process of creating scientific networks visualizations (aka *science maps*), by using data from [Dimensions on Google BigQuery](https://www.dimensions.ai/products/bigquery/). \nCurrenlty the only output visualization supported is [VOSviewer](https://www.vosviewer.com/). More visualizations might be added in the future.\n\n## What it looks like\n\n* [Live Application](https://digital-science.github.io/dimensions-network-gen/)\n* [Screenshots](#screenshots)\n\n## How it works\n\nIt is possible to generate network analyses on the whole COVID19 database, or using a selected subset of data. This is achieved by letting users input any SQL query defining a COVID-19 document subset of interest (e.g. a group of journals, or a group of countries).\n\nThe Python application deals with the extraction of data from BigQuery and calculation of the network representation. Currently we have included two possible network calculations:\n1. **Concept co-occurrence network**. This query generates two-concept pairs and counts how many publications are shared between these concepts (note: concepts in Dimensions are publication-level keywords normalised and weighted based on a relevancy score).\n2. **Organisation network**. This query generates two-organisations pairs (from the authors affiliations) and counts how many publications are shared between these organisations.\n\nFinally, the data extracted from BigQuery gets converted into a VOSviewer JSON file and packaged up into an HTML application that can be viewed in a browser. The Python library also includes a local server component that can be used to view the files locally on a computer.\n\n\n## Datasets\n\nBy default the tool uses the [Dimensions COVID-19 dataset](https://console.cloud.google.com/marketplace/product/digitalscience-public/covid-19-dataset-dimensions). The dataset is openly available on the Google Cloud Marketplace and contains all published articles and preprints, grants, clinical trials, and research datasets from Dimensions.ai that are related to COVID-19.\n\nAt time of writing (May 2022), the dataset contains:\n\n* 1M+  Publications and preprints\n* 16k+ Grants\n* 41k+ Patents\n* 14k+ Clinical Trials\n* 32k+ Research Datasets\n* 36k+ Research Organizations\n\nData model: see the [official documentation](https://docs.dimensions.ai/bigquery/data-sources.html).\n\n### Using the full Dimensions dataset\n\nUsers with an active subscription to the full [Dimensions on Google BigQuery](https://www.dimensions.ai/products/bigquery) dataset can perform network analyses using all data in Dimensions, not just the COVID19 subset.  \n\nIn order to do so, pass the `--fulldimensions` (or `-f`) option when invoking the script. E.g.\n\n```\n$ dimensions-networks {SQL_QUERY_FILE} --fulldimensions\n```\n\n### Accessing BigQuery \n\nIn order to access the Dimensions datasets, you need to be able to connect to [Google BigQuery](https://cloud.google.com/bigquery/) using Python. This means:\n\n* **Installing the SDK**. Installing \u0026 authorizing the the Google Cloud SDK, \"gcloud,\" available [directly from Google](https://cloud.google.com/sdk/docs/install). If you can open a terminal and the `gcloud` command is recognized, it has been sufficiently configured.\n* **Setting up a GCP project**. Each time you interact with BigQuery, you need to specify which GCP project you are using. This is generally used for resources access management. More info [here](https://docs.dimensions.ai/bigquery/gcp-setup.html).\n\nNote: newly created projects which have no associated billing account provide a [sandbox](https://cloud.google.com/bigquery/docs/sandbox) experience, providing initial access to the [free tier](https://cloud.google.com/free) of BigQuery provided by Google. The free tier is more than enough for using this library.  \n\n\n\n## Installation\n\nWith Python 3.9 and [virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/index.html)\n\n```bash\n$ git clone git@github.com:digital-science/dimensions-network-gen.git\n$ mkvirtualenv dimensions-networks\n$ pip install -r requirements.txt\n$ pip install -e .\n```\n\n\n## Running\n\nAfter installation, you can run the application by calling `dimensions-networks`.\n\n```bash\n$ dimensions-networks\nUsage: dimensions-networks [OPTIONS] [FILENAME]...\n\n  dimensions-networks: a tool for creating network visualizations powered by\n  data from Dimensions on Google BigQuery. Example:\n\n  dimensions-networks {QUERY_FILE}\n\n  QUERY_FILE. File name containing the GBQ query to be converted into a\n  network. If a folder is passed, all files in the folder will be processed.\n\nOptions:\n  -i, --buildindex      Just build the index page listing out previously\n                        created networks.\n  -f, --fulldimensions  Query using the full Dimensions dataset, instead of\n                        the COVID19 subset (note: requires subscription).\n  -r, --runserver       Run the webserver.\n  -p, --port INTEGER    Specify the port on which the webserver should listen\n                        for connections (default: 8009).\n  --verbose             Verbose mode\n  --help                Show this message and exit.\n```\n\n## Input files\n\nVisualizations are generated based on a subset of publications that you can define using SQL.\n\n* Each visualization is triggered by a corresponding SQL file containing a query and, optionally, some configuration directives. \n* SQL queries are stored in the `topics` directory.\n* Each file should contain a single SQL query that returns a list of Dimensions publication IDs **in a field called `id`**.\n* File names should be of the format `$title.sql`.\n  * For example, a file called `archaeology.sql` will create a network listed under the title \"archaeology\".\n\nE.g. these are the contents of `last_30_days.sql`:\n\n```sql\nselect id\nfrom `covid-19-dimensions-ai.data.publications`\nwhere \nEXTRACT(DATE FROM date_inserted) \u003e= DATE_ADD(CURRENT_DATE(), INTERVAL -30 DAY)\n```\n\nAnd this is how you're generate a network from that query:\n\n```bash\n$ dimensions-networks topics/last_30_days.sql\n```\n\n\n\n### Network configuration\n\nNetwork configuration options can be optionally defined in the SQL files before your query, as a series of commented lines starting with a predefined keyword. For example:\n\n```sql\n-- network_types: concepts, organizations\n-- max_nodes: 400 \n-- min_edge_weight: 3\n-- min_concept_relevance: 0.5 \n-- min_concept_frequency: 4\n\nselect id\nfrom `covid-19-dimensions-ai.data.publications`\nwhere \nEXTRACT(DATE FROM date_inserted) \u003e= DATE_ADD(CURRENT_DATE(), INTERVAL -30 DAY)\nand altmetrics.score \u003e 1\n```\n\nIf omitted, the default configuration values will be used. These are all the possible configurations and their meaning.\n\n| Option                | Default               | Notes                                                                                                                                                                    |\n|-----------------------|-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| network_types         | concepts, organizations | Currenlty two network types are supported.                                                                                                                               |\n| max_nodes             | 500                   | How many nodes should be displayed, at maximum?                                                                                                                          |\n| min_edge_weight       | 3                     | How many edges should two nodes share before they are linked in the network?                                                                                             |\n| min_concept_relevance | 0.5                   | Each concept tagged to a publication is assigned a relevance score between 0 and 1. What is the threshold that must be cleared before we consider a concept as relevant? |\n| min_concept_frequency | 5                     | How many times should a concept appear in the corpus overall before it's included in the network?                                                                        |\n\n\n\n\n## Output visualizations\n\nGenerated visualizations get added to the folder `build`, which is automatically created after running an extraction. \n\nThe folder contains a static website consisting of HTML, JS and JSON assets. The website uses relative links hence it can be published on web server *as is*. For example, see the `/docs` folder in this repository, which is viewable at https://digital-science.github.io/dimensions-network-gen/. \n\nIn order to browse the output folder locally, run the server utility: `dimensions-networks -s`. That will start a server on http://127.0.0.1:8009/\n\n\n## Screenshots\n\nA concept network:\n![concepts-network](/screenshots/concepts-network.png)\n\nAn organization collaboration network:\n\n![organizations-network.png](/screenshots/organizations-network.png)\n\n\n\n## Useful links\n\n* Vosviewer docs https://app.vosviewer.com/docs/\n* Dimensions BigQuery docs https://docs.dimensions.ai/bigquery/","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigital-science%2Fdimensions-network-gen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdigital-science%2Fdimensions-network-gen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigital-science%2Fdimensions-network-gen/lists"}