{"id":15009042,"url":"https://github.com/wroberts/pygermanet","last_synced_at":"2025-04-09T16:23:19.264Z","repository":{"id":15295191,"uuid":"18024869","full_name":"wroberts/pygermanet","owner":"wroberts","description":"GermaNet API for Python","archived":false,"fork":false,"pushed_at":"2018-03-08T21:03:17.000Z","size":10365,"stargazers_count":53,"open_issues_count":0,"forks_count":12,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-23T18:39:09.698Z","etag":null,"topics":["german","nlp","nltk","python","python-2","python-3","semantic-similarity","wordnet"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wroberts.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-03-23T02:28:33.000Z","updated_at":"2024-05-30T07:34:07.000Z","dependencies_parsed_at":"2022-07-31T04:08:54.290Z","dependency_job_id":null,"html_url":"https://github.com/wroberts/pygermanet","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wroberts%2Fpygermanet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wroberts%2Fpygermanet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wroberts%2Fpygermanet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wroberts%2Fpygermanet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wroberts","download_url":"https://codeload.github.com/wroberts/pygermanet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247755588,"owners_count":20990626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["german","nlp","nltk","python","python-2","python-3","semantic-similarity","wordnet"],"created_at":"2024-09-24T19:22:35.453Z","updated_at":"2025-04-09T16:23:19.236Z","avatar_url":"https://github.com/wroberts.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"============\n pygermanet\n============\n\n.. image:: https://img.shields.io/pypi/v/pygermanet.svg\n    :target: https://pypi.python.org/pypi/pygermanet/\n    :alt: Latest Version\n\nGermaNet API for Python.\n\nCopyright (c) 23 March, 2014 Will Roberts \u003cwildwilhelm@gmail.com\u003e.\n\nGermaNet_ is the German WordNet_, a machine-readable lexical semantic\nresource which lists nouns, verbs and adjectives in German, along with\nlexical relations which connect these words together into a semantic\nnetwork.  This library gives a Python interface to this resource.\n\n.. _GermaNet: http://www.sfs.uni-tuebingen.de/GermaNet/\n.. _WordNet: http://wordnet.princeton.edu/\n\n\nIntroduction\n------------\n\nStart GermaNet by connecting to the MongoDB_ database which contains\nthe lexical information (for setting up the MongoDB database, see the\nsection `Setup`_, below).  On the local machine using the default\nport, this is as simple as::\n\n    \u003e\u003e\u003e from pygermanet import load_germanet\n    \u003e\u003e\u003e gn = load_germanet()\n\nYou can search GermaNet for synsets containing a particular lemmatised\nword form using the ``synsets`` function::\n\n    \u003e\u003e\u003e gn.synsets('gehen')\n    [Synset(auseinandergehen.v.3),\n     Synset(funktionieren.v.1),\n     Synset(funktionieren.v.2),\n     Synset(gehen.v.1),\n     Synset(gehen.v.4),\n     Synset(gehen.v.5),\n     Synset(gehen.v.6),\n     Synset(gehen.v.7),\n     Synset(gehen.v.9),\n     Synset(gehen.v.10),\n     Synset(gehen.v.11),\n     Synset(gehen.v.12),\n     Synset(gehen.v.13),\n     Synset(gehen.v.14),\n     Synset(handeln.v.1)]\n\nTo look up synsets, you must have the canonical form of the word as it\nwould appear in a dictionary (head word); this module calls this word\nform the *lemma*.  The GermaNet instance can perform lemmatisation of\nwords using data derived from the `Projekt deutscher Wortschatz`_::\n\n    \u003e\u003e\u003e gn.lemmatise(u'ginge')\n    [u'gehen']\n\n.. _Projekt deutscher Wortschatz: http://wortschatz.uni-leipzig.de/\n\nEach ``Synset`` is represented by the orthographic form, part of speech,\nand sense number of its first ``Lemma``; this acts as a unique\nidentifier for synsets.  If you know this identifier, you can also\nlook up a synset in GermaNet::\n\n    \u003e\u003e\u003e funktionieren = gn.synset(u'funktionieren.v.2')\n    \u003e\u003e\u003e funktionieren\n    Synset(funktionieren.v.2)\n\n``Synset`` objects have data members and methods::\n\n    \u003e\u003e\u003e funktionieren.hyponyms\n    [Synset(vorgehen.v.1), Synset(leerlaufen.v.2)]\n    \u003e\u003e\u003e gn.synset('Husky.n.1').hypernym_paths\n    [[Synset(GNROOT.n.1),\n      Synset(Entität.n.2),\n      Synset(Objekt.n.4),\n      Synset(Ding.n.2),\n      Synset(Teil.n.2),\n      Synset(Teilmenge.n.2),\n      Synset(Gruppe.n.1),\n      Synset(biologische Gruppe.n.1),\n      Synset(Spezies.n.1),\n      Synset(Rasse.n.1),\n      Synset(Tierrasse.n.1),\n      Synset(Hunderasse.n.1),\n      Synset(Husky.n.1)],\n     [Synset(GNROOT.n.1),\n      Synset(Entität.n.2),\n      Synset(kognitives Objekt.n.1),\n      Synset(Kategorie.n.1),\n      Synset(Art.n.1),\n      Synset(Spezies.n.1),\n      Synset(Rasse.n.1),\n      Synset(Tierrasse.n.1),\n      Synset(Hunderasse.n.1),\n      Synset(Husky.n.1)],\n     [Synset(GNROOT.n.1),\n      Synset(Entität.n.2),\n      Synset(Objekt.n.4),\n      Synset(natürliches Objekt.n.1),\n      Synset(Wesenheit.n.1),\n      Synset(Organismus.n.1),\n      Synset(höheres Lebewesen.n.1),\n      Synset(Tier.n.1),\n      Synset(Gewebetier.n.1),\n      Synset(Chordatier.n.1),\n      Synset(Wirbeltier.n.1),\n      Synset(Säugetier.n.1),\n      Synset(Plazentatier.n.1),\n      Synset(Beutegreifer.n.1),\n      Synset(Landraubtier.n.1),\n      Synset(hundeartiges Landraubtier.n.1),\n      Synset(Hund.n.2),\n      Synset(Husky.n.1)],\n     [Synset(GNROOT.n.1),\n      Synset(Entität.n.2),\n      Synset(Objekt.n.4),\n      Synset(natürliches Objekt.n.1),\n      Synset(Wesenheit.n.1),\n      Synset(Organismus.n.1),\n      Synset(höheres Lebewesen.n.1),\n      Synset(Tier.n.1),\n      Synset(Haustier.n.1),\n      Synset(Hund.n.2),\n      Synset(Husky.n.1)]]\n\nEach ``Synset`` contains one or more ``Lemma`` objects::\n\n    \u003e\u003e\u003e funktionieren.lemmas\n    [Lemma(funktionieren.v.2.funktionieren),\n     Lemma(funktionieren.v.2.funzen),\n     Lemma(funktionieren.v.2.gehen),\n     Lemma(funktionieren.v.2.laufen),\n     Lemma(funktionieren.v.2.arbeiten)]\n\nA given orthographic form may be represented by multiple ``Lemma``\nobjects belonging to different ``Synset`` objects::\n\n    \u003e\u003e\u003e gn.lemmas('brennen')\n    [Lemma(brennen.v.1.brennen),\n     Lemma(verbrennen.v.1.brennen),\n     Lemma(brennen.v.3.brennen),\n     Lemma(brennen.v.4.brennen),\n     Lemma(brennen.v.5.brennen),\n     Lemma(destillieren.v.1.brennen),\n     Lemma(brennen.v.7.brennen),\n     Lemma(brennen.v.8.brennen)]\n\nSemantic Similarity\n-------------------\n\npygermanet includes several functions for calculating semantic\nsimilarity and semantic distance, somewhat like `WN::Similarity`_.\nThese metrics use word frequency information estimated on the SdeWaC_\ncorpus and then automatically lemmatised using the TreeTagger_.\n\nThe probability of encountering an instance of a given synset *s* is\nestimated over SdeWaC using the procedure described by `Resnik (1995)`_.\nBriefly, for each instance of a noun in the corpus, we find the set of\nsynsets *S* containing a sense of that noun; each of these synsets is then\ncredited with a count of 1/*|S|*.  A count added to a synset is\nalso added to all of its hypernyms, so that count observations are\npropagated up the taxonomy.  By dividing by the total number of noun\ninstances in the corpus, each synset is assigned a probability value;\nthese probabilities increase monotonically up the taxonomy, and the\nroot node has *p = 1*.\n\nUsing\nthis interface, we can replicate the results of `(Gurevych, 2005)`_\nand `(Gurevych and Niederlich, 2005)`_, who collected human semantic\nsimilarity judgements on 65 word pairs and then measured the\ncorrelation of these judgements against similarity scores reported by\nvarious automatic similarity metrics.  These two papers reported\nPearson's *r* of 0.715 for (Resnik, 1995), 0.738 for a normalised\ndistance version of `(Jiang and Conrath, 1997)`_, and 0.734 for\n`(Lin, 1998)`_, with inter-annotator agreement of 0.810.\n\nReplication of the two studies, using the gur65_ data set::\n\n    from pygermanet import load_germanet, Synset\n    from scipy.stats.stats import pearsonr\n    import codecs\n    import numpy as np\n\n    GUR65_FILENAME = 'gur65.csv'\n\n    def load_gurevych():\n        gur65 = []\n        with codecs.open(GUR65_FILENAME, 'r', 'latin-1') as input_file:\n            for idx, line in enumerate(input_file):\n                fields = line.strip().split(';')\n                if idx == 0:\n                    header = fields\n                else:\n                    # fix typo in gur65\n                    fields[1] = {'Reis': 'Reise'}.get(fields[1], fields[1])\n                    fields[2] = float(fields[2])\n                    fields[3] = float(fields[3])\n                    gur65.append(fields)\n        gur65 = np.core.records.array(\n            gur65,\n            dtype=np.dtype({'formats': ['U30', 'U30', '\u003cf8', '\u003cf8'],\n                            'names': header}))\n        return gur65\n\n    gur65 = load_gurevych()\n    gn    = load_germanet()\n\n    # select those words which are found in GermaNet; exclude the\n    # adjective \"jung\"\n    pred = lambda w1, w2: bool(gn.synsets(w1) and gn.synsets(w2) and\n                               w1 != 'jung' and w2 != 'jung')\n\n    print 'Semantic similarity computed on {0} of {1} word pairs'.format(\n        sum([1 for word1, word2 in zip(gur65['Word1'], gur65['Word2'])\n             if pred(word1, word2)]),\n        len(gur65))\n\n    sim_funcs = [('lch', Synset.sim_lch,  np.max),\n                 ('res', Synset.sim_res,  np.max),\n                 ('jcn', Synset.dist_jcn, np.min),\n                 ('lin', Synset.sim_lin,  np.max)]\n\n    print\n    print 'metric   r'\n    print '---------------'\n    for sim_name, sim_func, comb_func in sim_funcs:\n        scores = []\n        for word1, word2, human, _hstd in gur65:\n            if not pred(word1, word2):\n                continue\n            score = comb_func(np.array([sim_func(ss1, ss2)\n                                        for ss1 in gn.synsets(word1)\n                                        for ss2 in gn.synsets(word2)]))\n            scores.append([score, human])\n        scores = np.array(scores)\n        r, _p = pearsonr(scores[:,0],scores[:,1])\n        print '{0}      {1:.3f}'.format(sim_name, r)\n\nThis script outputs::\n\n    Semantic similarity computed on 60 of 65 word pairs\n\n    metric   r\n    ---------------\n    lch      0.742\n    res      0.715\n    jcn      -0.770\n    lin      0.737\n\n.. _`WN::Similarity`: http://wn-similarity.sourceforge.net/\n.. _gur65: https://www.ukp.tu-darmstadt.de/data/semantic-relatedness/german-relatedness-datasets/\n.. _TreeTagger: http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/\n.. _SdeWaC: http://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/sdewac.en.html\n.. _`(Gurevych, 2005)`: http://atlas.tk.informatik.tu-darmstadt.de/Publications/2005/ijcnlp05.pdf\n.. _`(Gurevych and Niederlich, 2005)`: http://oldsite.aclweb.org/anthology-new/I/I05/I05-7005.pdf\n.. _`(Jiang and Conrath, 1997)`: http://arxiv.org/pdf/cmp-lg/9709008.pdf\n.. _`(Lin, 1998)`: http://webdocs.cs.ualberta.ca/~lindek/papers/sim.pdf\n.. _`Resnik (1995)`: http://arxiv.org/pdf/cmp-lg/9511007.pdf\n\nRequirements\n------------\n\n- The GermaNet_ resource (available from the University of Tübingen)\n- Python 2.7 or Python 3 (tested with Python 3.4)\n- MongoDB_\n- pymongo_\n- future_ (for Python 2)\n- `repoze.lru`_ (optional)\n\n.. _MongoDB:    https://www.mongodb.org/\n.. _pymongo:    http://api.mongodb.org/python/current/\n.. _future:     http://python-future.org\n.. _repoze.lru: https://pypi.python.org/pypi/repoze.lru/\n\nExample setup::\n\n    sudo apt-get install mongodb\n    sudo pip install repoze.lru pygermanet\n\nSetup\n-----\n\nGermaNet_ is distributed as a set of XML files, or as a PostgreSQL\ndatabase dump, neither of which is a convenient format to handle\nfrom inside Python.  This library delegates responsibility for\nhandling the data to a MongoDB database.  As such, setup happens in\ntwo steps.\n\n1. Start a MongoDB instance running.  For example, the\n   ``start_mongo.sh`` script contains::\n\n       mkdir -p ./mongodb\n       mongod --dbpath ./mongodb\n\n2. Import GermaNet_ into the MongoDB instance.  The ``mongo_import.py``\n   script needs the path to the directory that contains the GermaNet_\n   XML files::\n\n       python -m pygermanet.mongo_import ~/corpora/germanet/GN_V80/GN_V80_XML/\n\n   This step only needs to be performed once, before you use\n   pygermanet for the first time.\n\n3. pygermanet can now be used by connecting to the running MongoDB\n   instance.  Using default settings and connecting to a database on\n   the local machine, this is accomplished with::\n\n       \u003e\u003e\u003e from pygermanet import load_germanet\n       \u003e\u003e\u003e gn = load_germanet()\n\nLicense\n-------\n\nThis README file and the source code in this library are licensed\nunder the MIT License (see source file LICENSE.txt for details).\n\nThe file ``baseforms_by_projekt_deutscher_wortschatz.txt.gz`` contains\ndata derived from the `Projekt deutscher Wortschatz`_; this database\nis free for educational and researching purposes but not for\ncommercial use. For more information visit:\nhttp://wortschatz.uni-leipzig.de/.\n\nThe file ``sdewac-gn-words.tsv.gz`` contains data derived from the\nSdeWaC_ corpus, whose license and copyright status are\nslightly unclear to me.  I'm releasing the file under the\n`Creative Commons Attribution 4.0 International License`_.\n\n.. _`Creative Commons Attribution 4.0 International License`: http://creativecommons.org/licenses/by/4.0/\n\nHistory\n-------\n\nThe NLTK_ project had an API once upon a time for interacting with\nGermaNet, but this has now been removed from the current NLTK\ndistribution.  This API was called GermaNLTK_ and was described in\nsome detail in `NLTK Issue 604`_.  pygermanet shamelessly imitates the\ninterface of this older NLTK code, which was, in turn, based on the\nstandard NLTK interface to WordNet.\n\n.. _NLTK:           http://www.nltk.org/\n.. _GermaNLTK:      https://docs.google.com/document/d/1rdn0hOnJNcOBWEZgipdDfSyjJdnv_sinuAUSDSpiQns/edit?hl=en\n.. _NLTK Issue 604: https://code.google.com/p/nltk/issues/detail?id=604\n\nThe GermaNLTK project had a script to push the content of the XML\nfiles into a sqlite database; I haven't tested this code myself, and\nthe GermaNet database has changed over the years since GermaNLTK was\nwritten.  This ``mongo_import.py`` script included in this library does much the\nsame thing, and could easily be adapted to use sqlite as a back end\ninstead of MongoDB.\n\n\nContributors\n------------\n\nThanks to stefanpernes_ for his work on making a Python 3 port of\npygermanet.\n\n.. _stefanpernes: https://github.com/stefanpernes\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwroberts%2Fpygermanet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwroberts%2Fpygermanet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwroberts%2Fpygermanet/lists"}