{"id":22700076,"url":"https://github.com/ub-mannheim/raisewikibase","last_synced_at":"2025-08-07T08:31:58.428Z","repository":{"id":44635815,"uuid":"344840884","full_name":"UB-Mannheim/RaiseWikibase","owner":"UB-Mannheim","description":"Knowledge graph construction: Fast inserts into a Wikibase instance","archived":false,"fork":false,"pushed_at":"2022-02-03T07:17:10.000Z","size":1465,"stargazers_count":43,"open_issues_count":7,"forks_count":7,"subscribers_count":6,"default_branch":"main","last_synced_at":"2023-12-07T23:00:28.481Z","etag":null,"topics":["knowledge-graph","knowledge-graph-construction","knowledge-graphs","knowledgegraph","wikibase","wikidata"],"latest_commit_sha":null,"homepage":"https://ub-mannheim.github.io/RaiseWikibase/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UB-Mannheim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-05T14:46:45.000Z","updated_at":"2023-05-11T11:14:47.000Z","dependencies_parsed_at":"2022-09-06T10:51:33.210Z","dependency_job_id":null,"html_url":"https://github.com/UB-Mannheim/RaiseWikibase","commit_stats":null,"previous_names":[],"tags_count":2,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UB-Mannheim%2FRaiseWikibase","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UB-Mannheim%2FRaiseWikibase/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UB-Mannheim%2FRaiseWikibase/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UB-Mannheim%2FRaiseWikibase/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UB-Mannheim","download_url":"https://codeload.github.com/UB-Mannheim/RaiseWikibase/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229013256,"owners_count":18006191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["knowledge-graph","knowledge-graph-construction","knowledge-graphs","knowledgegraph","wikibase","wikidata"],"created_at":"2024-12-10T06:09:43.798Z","updated_at":"2024-12-10T06:09:44.334Z","avatar_url":"https://github.com/UB-Mannheim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RaiseWikibase\n\n```\nA tool for speeding up multilingual knowledge graph construction with Wikibase\n```\n[[Camera-ready PDF preprint \"RaiseWikibase: Fast inserts into the BERD instance\" for ESWC 2021 P\u0026D](https://openreview.net/pdf?id=87hp7LJDJE)]\n* Fast inserts into a Wikibase instance: creates up to a million entities and wikitexts per hour.\n* Creates a mini Wikibase instance with Wikidata properties in a few minutes.\n* Creates the [BERD](https://www.berd-bw.de) knowledge graph with millions of entities in a few hours.\n\n:warning: \u0026nbsp; This tool is experimental. The current desire is to move its functionality to the Wikibase API, see the ticket [T287164 \"Improve bulk import via API\"](https://phabricator.wikimedia.org/T287164). If you are interested in improving bulk import into Wikibase, please contribute to that ticket.\n\n## Table of contents\n- [How to use](#how-to-use)\n  * [Installation](#installation)\n  * [Wikibase Docker](#wikibase-docker)\n  * [Wikibase Extensions](#wikibase-extensions)\n  * [Wikibase Data Model and RaiseWikibase functions](#wikibase-data-model-and-raisewikibase-functions)\n  * [Creating entities and texts](#creating-entities-and-texts)\n  * [Testing all datatypes](#testing-all-datatypes)\n  * [Compatibility with WikidataIntegrator and WikibaseIntegrator](#compatibility-with-wikidataintegrator-and-wikibaseintegrator)\n  * [Getting data from Wikidata and filling it into a Wikibase instance](#getting-data-from-wikidata-and-filling-it-into-a-wikibase-instance)\n- [Performance analysis](#performance-analysis)\n- [Creating a mini Wikibase instance with thousands of entities in a few minutes](#creating-a-mini-wikibase-instance-with-thousands-of-entities-in-a-few-minutes)\n- [Creating a mega Wikibase instance with millions of BERD entities in a few hours](#creating-a-mega-wikibase-instance-with-millions-of-berd-entities-in-a-few-hours)\n- [Deployment in production](#deployment-in-production)\n- [Paper](#paper)\n- [Acknowledgments](#acknowledgments)\n- [See also](#see-also)\n\n## How to use\n\n### Installation\n\nClone RaiseWikibase and install it via `pip3`:\n```shell\ngit clone https://github.com/UB-Mannheim/RaiseWikibase\ncd RaiseWikibase/\npip3 install .\n```\n\n### Wikibase Docker\n\n:eyes: \u0026nbsp; [Wikibase Docker](https://github.com/wmde/wikibase-release-pipeline) is distributed under [BSD 3-Clause License](https://github.com/wmde/wikibase-release-pipeline/blob/master/LICENSE). Please fulfill the requirements.\n\nRaiseWikibase is solely based on [Wikibase Docker](https://github.com/wmde/wikibase-release-pipeline) developed by [Wikimedia Germany](https://wikimedia.de). [Wikibase Docker](https://github.com/wmde/wikibase-release-pipeline) significantly simplifies deployment of a [Wikibase](https://github.com/wikimedia/Wikibase) instance.\n\n:warning: \u0026nbsp; Copy [env.tmpl](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/env.tmpl) to `.env` and substitute the default values with your\nown usernames and passwords.\n\nInstall [Docker](https://docs.docker.com/get-docker/).\n\nRun in the main RaiseWikibase folder:\n```shell\ndocker-compose -f docker-compose.yml -f docker-compose.extra.yml up -d --scale wikibase_jobrunner=1\n```\nSee more details at [Wikibase Release Pipeline](https://github.com/wmde/wikibase-release-pipeline/tree/wmde.2/example).\n\nIf it runs first time, it pulls the Wikibase Docker images. Then it builds, creates, starts, and attaches to containers for a service.\nCheck whether it's running using:\n```shell\ndocker ps\n```\n\nIf it's running, the output looks like this:\n```shell\nCONTAINER ID        IMAGE                                COMMAND                   CREATED              STATUS              PORTS                       NAMES\n0cac985f00a5        wikibase/quickstatements:latest      \"/bin/bash /entrypoi…\"    About a minute ago   Up About a minute   0.0.0.0:9191-\u003e80/tcp        raisewikibase_quickstatements_1\n2f277b599ea0        wikibase/wdqs:0.3.40                 \"/entrypoint.sh /run…\"    About a minute ago   Up About a minute                               raisewikibase_wdqs-updater_1\n3d7e6462b290        wikibase/wdqs-frontend:latest        \"/entrypoint.sh ngin…\"    About a minute ago   Up About a minute   0.0.0.0:8282-\u003e80/tcp        raisewikibase_wdqs-frontend_1\nef945d05fc88        wikibase/wikibase:1.35-bundle        \"/bin/bash /entrypoi…\"    About a minute ago   Up About a minute   0.0.0.0:8181-\u003e80/tcp        raisewikibase_wikibase_1\n10df54332657        wikibase/wdqs-proxy                  \"/bin/sh -c \\\"/entryp…\"   About a minute ago   Up About a minute   0.0.0.0:8989-\u003e80/tcp        raisewikibase_wdqs-proxy_1\n37f34328b73f        wikibase/wdqs:0.3.40                 \"/entrypoint.sh /run…\"    About a minute ago   Up About a minute   9999/tcp                    raisewikibase_wdqs_1\n9a1c8ddd8c89        wikibase/elasticsearch:6.5.4-extra   \"/usr/local/bin/dock…\"    About a minute ago   Up About a minute   9200/tcp, 9300/tcp          raisewikibase_elasticsearch_1\nb640eaa556e3        mariadb:10.3                         \"docker-entrypoint.s…\"    About a minute ago   Up About a minute   127.0.0.1:63306-\u003e3306/tcp   raisewikibase_mysql_1\n```\n\nThe logs can be viewed via:\n```shell\ndocker-compose logs -f\n```\n\nUsually in less than a minute from the start you will see the messages from `wdqs-updater_1` in the logs: `INFO  o.w.q.r.t.change.RecentChangesPoller - Got no real changes` and `INFO  org.wikidata.query.rdf.tool.Updater - Sleeping for 10 secs`. The Wikibase front-end (http://localhost:8181) and query service (http://localhost:8282) are already available. Data filling can be started.\n\nIf you want to stop the Wikibase Docker, to remove all your uploaded data and to run a fresh Wikibase instance, use:\n```shell\ndocker-compose down\ndocker volume prune\ndocker-compose up -d\n```\n\nSee also [Wikibase/Docker](https://www.mediawiki.org/wiki/Wikibase/Docker).\n\n### Wikibase Extensions\n\n\"Extensions let you customize how MediaWiki looks and works\" is written in [Manual:Extensions](https://www.mediawiki.org/wiki/Manual:Extensions). Note that [Wikibase](https://wikiba.se) is itself an extension to the [Mediawiki](https://www.mediawiki.org/wiki/MediaWiki) software.\n\nTo add the datatype [Mathematical expression](https://www.wikidata.org/wiki/Help:Data_type#Mathematical_expression) (or simply `Math`) to a Wikibase instance, install the extension [Math](https://www.mediawiki.org/wiki/Extension:Math). An example is the property [defining formula](https://www.wikidata.org/entity/P2534).\n\nSee also [Extending Wikibase](https://wikiba.se/extend).\n\n### Wikibase Data Model and RaiseWikibase functions\n\nThe [Wikibase Data Model](https://www.mediawiki.org/wiki/Wikibase/DataModel) is an ontology describing the structure of the data in Wikibase. A non-technical summary of the Wikibase model is available at [DataModel/Primer](https://www.mediawiki.org/wiki/Wikibase/DataModel/Primer). The initial [conceptual specification](https://www.mediawiki.org/wiki/Wikibase/DataModel)\nfor the Data Model was created by [Markus Krötzsch](http://korrekt.org/)\nand [Denny Vrandečić](http://simia.net/wiki/Denny), with minor contributions by\nDaniel Kinzler and [Jeroen De Dauw](https://www.EntropyWins.wtf). The Wikibase Data Model has been implemented by [Jeroen De Dauw](https://www.EntropyWins.wtf)\nand Thiemo Kreuz as [Wikimedia Germany](https://wikimedia.de) employees for the [Wikidata project](https://wikidata.org/).\n\nRaiseWikibase provides the functions for the [Wikibase Data Model](https://www.mediawiki.org/wiki/Wikibase/DataModel):\n```python\nfrom RaiseWikibase.datamodel import label, alias, description, snak, claim, entity\n```\n\nThe functions `entity`, `claim`, `snak`, `description`, `alias`and `label` return the template dictionaries. So all basic operations with dictionaries in Python can be used. You can merge two dictionaries `X` and `Y` using `X | Y` (since Python 3.9), `{**X, **Y}` (since Python 3.5) and `X.update(Y)`.\n\nLet's check the Wikidata entity [Q43229](https://www.wikidata.org/wiki/Q43229) with an English label 'organization'. You can create both English and German labels for the entity in a local Wikibase instance using RaiseWikibase:\n```python\nlabels = {**label('en', 'organization'), **label('de', 'Organisation')}\n```\n\nMultiple English and German aliases can also be easily created:\n```python\naliases = alias('en', ['organisation', 'org']) | alias('de', ['Org', 'Orga'])\n```\n\nMultilingual descriptions can be added:\n```python\ndescriptions = description('en', 'social entity (not necessarily commercial)')\ndescriptions.update(description('de', 'soziale Struktur mit einem gemeinsamen Ziel'))\n```\n\nTo add statements (claims), qualifiers and references, we need the `snak` function. To create a snak, we have to specify `property`, `datavalue`, `datatype` and `snaktype`. For example, if a Wikibase instance has the property with ID `P1`, a label `Wikidata ID` and datatype `external-id`, we can create a mainsnak with that property and the value 'Q43229':\n```python\nmainsnak = snak(datatype='external-id', value='Q43229', prop='P1', snaktype='value')\n```\n\nJust as an example of creating the qualifiers and references, let's add:\n```python\nqualifiers = [snak(datatype='external-id', value='Q43229', prop='P1', snaktype='value')]\nreferences = [snak(datatype='external-id', value='Q43229', prop='P1', snaktype='value')]\n```\n\nWe have now a mainsnak, qualifiers and references. Let's create a claim for an item:\n```python\nclaims = claim(prop='P1', mainsnak=mainsnak, qualifiers=qualifiers, references=references)\n```\n\nIf you need a claim with multiple values for one property, there are two opportunities. The first one is using the `extend` function on lists:\n```python\nclaims1 = claim(prop='P1', mainsnak=mainsnak1, qualifiers=qualifiers1, references=references1)\nclaims2 = claim(prop='P1', mainsnak=mainsnak2, qualifiers=qualifiers2, references=references2)\nclaims1['P1'].extend(claims2['P1'])\n```\n\nThe second option is using the `mainsnak` and `statement` functions:\n```python\nsnak1 = snak(datatype='external-id', value='Q43229', prop='P1', snaktype='value')\nsnak2 = snak(datatype='external-id', value='Q5', prop='P1', snaktype='value')\nmainsnak1 = mainsnak(prop='P1', snak=snak1, qualifiers=[], references=[])\nmainsnak2 = mainsnak(prop='P1', snak=snak2, qualifiers=[], references=[])\nstatements = statement(prop='P1', mainsnaks=[mainsnak1, mainsnak2])\n```\n\nNote that the `claim` and `statement` functions return the same template dictionaries, but their input parameters are different. The `claim` function is useful when your claims have one value per property. Multiple values per property are easier to create using the `statement` function.\n\nAll ingredients for creating the JSON representation of an item are ready. The `entity` function does the job:\n```python\nitem = entity(labels=labels, aliases=aliases, descriptions=descriptions, claims=claims, etype='item')\n```\nwhere `claims=claims` can be replaced by `claims=statements`.\n\nIf a property is created, the corresponding datatype has to be additionally specified:\n```python\nproperty = entity(labels=labels, aliases=aliases, descriptions=descriptions,\n\t\t  claims=claims, etype='property', datatype='string')\n```\n\nNote that these functions create only the dictionaries for the corresponding elements in the Wikibase Data Model. Writing into the database is performed using the `page` and `batch` functions.\n\n### Creating entities and texts\n\nTo create one thousand items with the already created JSON representation of an item, use:\n```python\nfrom RaiseWikibase.raiser import batch\nbatch(content_model='wikibase-item', texts=[item for i in range(1000)])\n```\n\nLet `wtext` is a Python string representing a wikitext. Then, `wikitexts = [wtext for i in range(1000)]` is a list of wikitexts and `page_titles = ['wikitext' + str(i) for i in range(1000)]` is a list of the corresponding page titles. To create one thousand wikitexts in the main namespace, use:\n\n```python\nbatch(content_model='wikitext', texts=wikitexts, namespace=0, page_title=page_titles)\n```\n\nThe dictionary of [namespaces](https://www.mediawiki.org/wiki/Extension_default_namespaces) can be found here:\n```python\nfrom RaiseWikibase.datamodel import namespaces\n```\n\nThe ID for the main namespace `namespaces['main']` is `0`.\n\nAlternatively, the `page` function can be used directly. First, a connection object is created. The page function executes the necessary inserts, the changes are commited and the connection is closed:\n```python\nfrom RaiseWikibase.dbconnection import DBConnection\nfrom RaiseWikibase.raiser import page\nconnection = DBConnection()\npage(connection=connection, content_model=content_model,\n     namespace=namespace, text=text, page_title=page_title, new=True)\nconnection.conn.commit()\nconnection.conn.close()\n```\n\nThe argument `new` specifies whether the page is created (`new=True`) or edited (`new=False`). The `new` argument can be used in the `batch` function as well.\n\n### Testing all datatypes\n\nThis section is moved to [docs](https://ub-mannheim.github.io/RaiseWikibase/quickstart/testing_datatypes). It describes testing all datatypes in a Wikibase instance and checking what kind of extensions they require.\n\n### Compatibility with WikidataIntegrator and WikibaseIntegrator\n\n[WikidataIntegrator](https://github.com/SuLab/WikidataIntegrator) and [WikibaseIntegrator](https://github.com/LeMyst/WikibaseIntegrator) are the wrappers of the [Wikibase API](https://www.mediawiki.org/wiki/Wikibase/API). A bot account is needed to start data filling with them. RaiseWikibase can create a bot account for a local Wikibase instance, save the login and password to a configuration file and read them back to a `config` dictionary:\n\n```python\nfrom RaiseWikibase.raiser import create_bot\nfrom RaiseWikibase.settings import Settings\ncreate_bot()\nconfig = Settings()\n```\n\nThe `config` dictionary can be used in WikibaseIntegrator for creating a login instance:\n```python\nfrom wikibaseintegrator import wbi_login\nlogin_instance = wbi_login.Login(user=config.username, password=config.password)\n```\nand in WikidataIntegrator:\n```python\nfrom wikidataintegrator import wdi_login\nlogin_instance = wdi_login.WDLogin(user=config.username, pwd=config.password)\n```\n\nYou can also create the JSON representations of entities in WikidataIntegrator or WikibaseIntegrator and then fill them into a Wikibase instance using RaiseWikibase.\nIn WikibaseIntegrator you can create a `Item` object and use the `get_json` function:\n```python\nfrom wikibaseintegrator import WikibaseIntegrator\nwbi = WikibaseIntegrator(login=login_instance)\nitem = wbi.item.get(entity_id='Q1003030')\nijson = item.get_json()\n```\n\nIn WikidataIntegrator a `wdi_core.WDItemEngine` object can be created and the `get_wd_json_representation` function can be used:\n```python\nfrom wikidataintegrator import wdi_core\nitem = wdi_core.WDItemEngine(wd_item_id='Q1003030')\nijson = item.get_wd_json_representation()\n```\n\nThe JSON representation of an entity can be uploaded into a Wikibase instance using the `batch` function in RaiseWikibase:\n```python\nfrom RaiseWikibase.raiser import batch\nbatch('wikibase-item', [ijson])\n```\n\n### Getting data from Wikidata and filling it into a Wikibase instance\n\nThe [Wikidata](https://wikidata.org/) knowledge graph already has millions of items and thousands of properties. For many projects some of these entities can be reused. Let's create the multilingual items [human](https://wikidata.org/entity/Q5), [organization](https://wikidata.org/entity/Q43229) and [location](https://wikidata.org/entity/Q17334923) in a local Wikibase instance using RaiseWikibase.\n\nThe example below defines the function `get_wd_entity`. It takes a Wikidata ID as an input, sends a request to Wikidata, gets the JSON representation of an entity, removes the keys unwanted in a local Wikibase instance, creates a claim and returns the JSON representation of the entity, if an error has not occured. The function `get_wd_entity` is used to get the JSON representations for [human](https://wikidata.org/entity/Q5), [organization](https://wikidata.org/entity/Q43229) and [location](https://wikidata.org/entity/Q17334923). These JSON representations are then filled into a local Wikibase instance using the `batch` function.\n\n```python\nfrom RaiseWikibase.raiser import batch\nfrom RaiseWikibase.datamodel import claim, snak\nimport requests\n\ndef get_wd_entity(wid=''):\n    \"\"\"Returns JSON representation of a Wikidata entity for the given WID\"\"\"\n    # Remove the following keys to avoid a problem with a new Wikibase instance\n    remove_keys = ['lastrevid', 'pageid', 'modified', 'title', 'ns']\n    try:\n        r = requests.get('https://www.wikidata.org/entity/' + wid + '.json')\n        entity = r.json().get('entities').get(wid)\n        for key in remove_keys:\n            entity.pop(key)\n        entity['claims'] = claim(prop='P1',\n                                 mainsnak=snak(datatype='external-id',\n                                               value=wid,\n                                               prop='P1',\n                                               snaktype='value'),\n                                 qualifiers=[],\n                                 references=[])\n    except Exception:\n        entity = None\n    return entity\n\nwids = ['Q5', 'Q43229', 'Q17334923'] # human, organization, location\nitems = [get_wd_entity(wid) for wid in wids]\nbatch('wikibase-item', items)\n```\n\nThe lines, where `entity['claims']` is rewritten, can be commented. Then, the created items contain the claims with the property IDs corresponding to Wikidata. Just try it out.\n\nIf you filled the entities from Wikidata into a fresh Wikibase instance, but you cannot open a page at http://localhost:8181/entity/Q1, run in shell:\n```shell\ndocker exec raisewikibase_wikibase_1 bash \"-c\" \"php maintenance/update.php --quick --force\"\n```\n\nWe used the property with ID 'P1' in the claim. That property with a label 'Wikidata ID' can be created using the script [miniWikibase.py](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/miniWikibase.py). It creates all 9000+ Wikidata properties in two minutes.\n\n## Performance analysis\n\nThe script [performance.py](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/performance.py) runs two performance experiments for creating the wikitexts and items. Run:\n```shell\npython3 performance.py\n```\n\nThe variable [batch_lengths](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/performance.py#L39) is set by default to `[100]`. This means that the length of a batch in each experiment is `100`. Running both experiments in this case takes 80 seconds. You can set it to `[100, 200, 300]` in order to run multiple experiments with different batch lengths. In our experiments we used  `batch_lengths = [10000]`.\n\nThe script saves the CSV files with numeric values of results and creates the pdf files with figures in `./experiments/`.\n\n| (1a) Wikitexts | (1b) Items |\n|:------:|:------:|\n| ![alt text](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/experiments/exp1.png) | ![alt text](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/experiments/exp2.png) |\n\nThe insert rates in pages per second are shown at Figure 1a for wikitexts and at Figure 1b for items. Every data point corresponds to a batch of ten thousands pages. At Figure 1a six different data points correspond to six repeated experiments. At Figure 1b two colors correspond to two repeated experiments and three shapes of a data point correspond to the three cases: 1) circle - each claim without a qualifier and without a reference, 2) x - each claim with one qualifier and without a reference, and 3) square - each claim with one qualifier and one reference.\n\nTo 'reproduce' Figures 1a and 1b, set [batch_lengths](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/performance.py#L39) to `[10000]`. Note that 'reproducibility' in this case does not mean that you will get the same values in the experiments as at Figures 1a and 1b. It means that you can get similar plots with the values specific for your hardware and software. Our analysis was performed using a workstation with 6-core Intel i5-8500T CPU @ 2.10GHz, 16GB RAM, SSD storage and running Debian 10.\n\n## Creating a mini Wikibase instance with thousands of entities in a few minutes\n\nThe script [miniWikibase.py](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/miniWikibase.py) fills a fresh Wikibase instance with some structured and unstructured data in roughly 30 seconds. The data include 8400+ properties from [Wikidata](https://www.wikidata.org), two templates, a page with SPARQL examples, a page with a sidebar and modules. Check the folder `texts` containing unstructured data and add there your own data. Information about the Wikidata properties is queried through the [Wikidata endpoint](https://query.wikidata.org) and it takes a few seconds. Run:\n```shell\npython3 miniWikibase.py\n```\n\n| (2a) Main page | (2b) List of properties |\n|:------:|:------:|\n| ![alt text](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/experiments/mini1.png) | ![alt text](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/experiments/mini2.png) |\n\nFigure 2a shows the main page and Figure 2b shows a list of properties. If you run the script `miniWikibase.py` with the commented line 156, you will see only the property identifiers instead of the labels. You can either uncomment line 156 or run in shell `docker-compose down` and `docker-compose up -d`.\n\n## Creating a mega Wikibase instance with millions of BERD entities in a few hours\n\nThe script [megaWikibase.py](https://github.com/UB-Mannheim/RaiseWikibase/blob/main/megaWikibase.py) creates a knowledge graph with millions of BERD (Business, Economic and Related Data) entities from scratch. Before running it prepare the OpenCorporates dataset.\nDownload https://daten.offeneregister.de/openregister.db.gz. Unzip it and run in shell:\n```shell\nsqlite3 -header -csv handelsregister.db \"select * from company;\" \u003e millions_companies.csv\n```\nPut `millions_companies.csv` to the main RaiseWikibase folder.\n\nRun:\n```shell\npython3 megaWikibase.py\n```\n\n## Deployment in production\n\nThe setting above runs on localhost.\n\nA [setup](https://github.com/needsone/my-wikibase) (and [this](https://stackoverflow.com/a/63397827)) for deployment using Nginx is provided by [Louis Poncet](https://github.com/needsone) ([personaldata.io](https://wiki.personaldata.io)).\n\n## Paper\n\n```\n@inproceedings{RaiseWikibase2021,\nauthor={Shigapov, Renat and Mechnich, J{\\\"o}rg and Schumm, Irene},\ntitle={RaiseWikibase: {F}ast inserts into the {BERD} instance},\nbooktitle={The {S}emantic {W}eb: {ESWC} 2021 {S}atellite {E}vents},\nyear={2021},\npublisher={Springer International Publishing},\npages={60--64},\ndoi={10.1007/978-3-030-80418-3\\_11},\nurl={https://doi.org/10.1007/978-3-030-80418-3\\_11}\n}\n```\n\n[[DOI](https://doi.org/10.1007/978-3-030-80418-3_11)] [[preprint](https://openreview.net/pdf?id=87hp7LJDJE)] [[poster](https://ub-mannheim.github.io/RaiseWikibase/poster)]\n\n## Acknowledgments\n\nThis work was funded by the Ministry of Science, Research and Arts of Baden-Württemberg through the project [Business and Economics Research Data Center Baden-Württemberg (BERD@BW)](https://www.berd-bw.de).\n\nWe thank [Jesper Zedlitz](https://github.com/jze) for his experiments explained at [the FactGrid blog](https://blog.factgrid.de/archives/2013) and for his open source code [wikibase-insert](https://github.com/jze/wikibase-insert).\n\n## See also\n\n[The official Wikibase website](https://wikiba.se), [Wikidata \u0026 Wikibase architecture documentation](https://wmde.github.io/wikidata-wikibase-architecture), [Strategy for the Wikibase Ecosystem](https://upload.wikimedia.org/wikipedia/commons/c/cc/Strategy_for_Wikibase_Ecosystem.pdf), the [posts about Wikibase](https://addshore.com/tag/wikibase) and [Wikidata](https://addshore.com/tag/wikidata/) by [Adam 'addshore' Shorland](https://addshore.com/about), a [Wikibase tutorial](https://stuff.coffeecode.net/2018/wikibase-workshop-swib18.html) by [Dan Scott](https://dscott.ca/#i), [Wikibase Install Basic Tutorial](https://semlab.io/howto/wikibase_basic) and [Wikibase for Research Infrastructure](https://medium.com/@thisismattmiller/wikibase-for-research-infrastructure-part-1-d3f640dfad34) by [Matt Miller](https://thisismattmiller.com/about), [Get your own copy of WikiData](http://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData) by [Wolfgang Fahl](https://github.com/WolfgangFahl), [Transferring Wikibase data between wikis](https://wikibase.consulting/transferring-wikibase-data-between-wikis) by [Jeroen De Dauw](https://www.EntropyWins.wtf), [Putting Data into Wikidata using Software](http://baskauf.blogspot.com/2019/06/putting-data-into-wikidata-using.html) by [Steve Baskauf](https://github.com/baskaufs), [Vanderbilt Heard Library digital scholarship resources on Wikidata and Wikibase](https://heardlibrary.github.io/digital-scholarship/host/wikidata/), [Learning Wikibase](http://learningwikibase.com), [Wikibase Yearly Summary 2020](https://www.lehir.net/wikibase-yearly-summary-2020) and [Wikibase Yearly Summary 2021](https://www.lehir.net/wikibase-yearly-summary-2021).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fub-mannheim%2Fraisewikibase","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fub-mannheim%2Fraisewikibase","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fub-mannheim%2Fraisewikibase/lists"}