{"id":13836582,"url":"https://github.com/inspirehep/magpie","last_synced_at":"2025-04-04T07:08:52.681Z","repository":{"id":67345642,"uuid":"46973796","full_name":"inspirehep/magpie","owner":"inspirehep","description":"Deep neural network framework for multi-label text classification","archived":false,"fork":false,"pushed_at":"2023-01-31T03:56:09.000Z","size":95373,"stargazers_count":683,"open_issues_count":49,"forks_count":192,"subscribers_count":44,"default_branch":"master","last_synced_at":"2025-03-28T06:09:29.704Z","etag":null,"topics":["classification","deep-learning","machine-learning","multi-label-classification","neural-network","nlp","prediction","word2vec"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/inspirehep.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2015-11-27T10:16:06.000Z","updated_at":"2025-03-06T06:28:07.000Z","dependencies_parsed_at":"2023-02-21T08:15:22.526Z","dependency_job_id":null,"html_url":"https://github.com/inspirehep/magpie","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspirehep%2Fmagpie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspirehep%2Fmagpie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspirehep%2Fmagpie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspirehep%2Fmagpie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/inspirehep","download_url":"https://codeload.github.com/inspirehep/magpie/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247135144,"owners_count":20889421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["classification","deep-learning","machine-learning","multi-label-classification","neural-network","nlp","prediction","word2vec"],"created_at":"2024-08-04T15:00:50.303Z","updated_at":"2025-04-04T07:08:52.658Z","avatar_url":"https://github.com/inspirehep.png","language":"Python","funding_links":[],"categories":["Code"],"sub_categories":[],"readme":"![image](docs/img/logo.png)\n\nMagpie is a deep learning tool for multi-label text classification. It learns on the training corpus to assign labels to arbitrary text and can be used to predict those labels on unknown data. It has been developed at CERN to assign subject categories to High Energy Physics abstracts and extract keywords from them.\n\n## Very short introduction\n```\n\u003e\u003e\u003e magpie = Magpie()\n\u003e\u003e\u003e magpie.init_word_vectors('/path/to/corpus', vec_dim=100)\n\u003e\u003e\u003e magpie.train('/path/to/corpus', ['label1', 'label2', 'label3'], epochs=3)\nTraining...\n\u003e\u003e\u003e magpie.predict_from_text('Well, that was quick!')\n[('label1', 0.96), ('label3', 0.65), ('label2', 0.21)]\n```\n\n\n## Short introduction\nTo train the model you need to have a large corpus of labeled data in a text format encoded as UTF-8. An example corpus can be found under data/hep-categories directory. Magpie looks for `.txt` files containing the text to predict on and corresponding `.lab` files with assigned labels in separate lines. A pair of files containing the labels and the text should have the same name and differ only in extensions e.g.\n\n```\n$ ls data/hep-categories\n1000222.lab 1000222.txt 1000362.lab 1000362.txt 1001810.lab 1001810.txt ...\n```\n\nBefore you train the model, you need to build appropriate word vector representations for your corpus. In theory, you can train them on a different corpus or reuse already trained ones ([tutorial](http://rare-technologies.com/word2vec-tutorial/)), however Magpie enables you to do that as well.\n```python\nfrom magpie import Magpie\n\nmagpie = Magpie()\nmagpie.train_word2vec('data/hep-categories', vec_dim=100)\n```\n\nThen you need to fit a scaling matrix to normalize input data, it is specific to the trained word2vec representation. Here's the one liner:\n\n```python\nmagpie.fit_scaler('data/hep-categories')\n```\n\nYou would usually want to combine those two steps, by simply running:\n```python\nmagpie.init_word_vectors('data/hep-categories', vec_dim=100)\n```\n\nIf you plan to reuse the trained word representations, you might want to save them and pass in the constructor to `Magpie` next time. For the training, just type:\n```python\nlabels = ['Gravitation and Cosmology', 'Experiment-HEP', 'Theory-HEP']\nmagpie.train('data/hep-categories', labels, test_ratio=0.2, epochs=30)\n```\nBy providing the `test_ratio` argument, the model splits data into train \u0026 test datasets (in this example into 80/20 ratio) and evaluates itself after every epoch displaying it's current loss and accuracy. The default value of `test_ratio` is 0 meaning that all the data will be used for training.\n\nIf your data doesn't fit into memory, you can also run `magpie.batch_train()` which has a similar API, but is more memory efficient.\n\nTrained models can be used for prediction with methods:\n```python\n\u003e\u003e\u003e magpie.predict_from_file('data/hep-categories/1002413.txt')\n[('Experiment-HEP', 0.47593361),\n ('Gravitation and Cosmology', 0.055745006),\n ('Theory-HEP', 0.02692855)]\n\n\u003e\u003e\u003e magpie.predict_from_text('Stephen Hawking studies black holes')\n[('Gravitation and Cosmology', 0.96627593),\n ('Experiment-HEP', 0.64958507),\n ('Theory-HEP', 0.20917746)]\n```\n\n## Saving \u0026 loading the model\nA `Magpie` object consists of three components - the word2vec mappings, a scaler and a `keras` model. In order to train Magpie you can either provide the word2vec mappings and a scaler in advance or let the program compute them for you on the training data. Usually you would want to train them yourself on a full dataset and reuse them afterwards. You can use the provided functions for that purpose:\n\n```python\nmagpie.save_word2vec_model('/save/my/embeddings/here')\nmagpie.save_scaler('/save/my/scaler/here', overwrite=True)\nmagpie.save_model('/save/my/model/here.h5')\n```\n\nWhen you want to reinitialize your trained model, you can run:\n\n```python\nmagpie = Magpie(\n    keras_model='/save/my/model/here.h5',\n    word2vec_model='/save/my/embeddings/here',\n    scaler='/save/my/scaler/here',\n    labels=['cat', 'dog', 'cow']\n)\n```\nor just pass the objects directly!\n\n## Installation\n\nThe package is not on PyPi, but you can get it directly from GitHub:\n```\n$ pip install git+https://github.com/inspirehep/magpie.git@v2.1.1\n```\nIf you encounter any problems with the installation, make sure to install the correct versions of dependencies listed in `setup.py` file.\n\n## Disclaimer \u0026 citation\nThe neural network models used within Magpie are based on work done by [Yoon Kim](https://arxiv.org/abs/1408.5882) and subsequently [Mark Berger](https://cs224d.stanford.edu/reports/BergerMark.pdf).\n\n## Contact\nIf you have any problems, feel free to open an issue. We'll do our best to help :+1:\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finspirehep%2Fmagpie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finspirehep%2Fmagpie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finspirehep%2Fmagpie/lists"}