{"id":21290477,"url":"https://github.com/saattrupdan/scholarly","last_synced_at":"2025-08-08T08:54:08.718Z","repository":{"id":49914293,"uuid":"183391454","full_name":"saattrupdan/scholarly","owner":"saattrupdan","description":"Classification of scientific papers","archived":true,"fork":false,"pushed_at":"2023-02-15T23:07:51.000Z","size":9822,"stargazers_count":24,"open_issues_count":6,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-07-11T15:30:29.235Z","etag":null,"topics":["arxiv","arxiv-api","classification","multilabel","neural-network","scientific-papers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saattrupdan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-25T08:32:56.000Z","updated_at":"2025-05-28T22:24:00.000Z","dependencies_parsed_at":"2024-11-21T12:51:40.988Z","dependency_job_id":"24f6b59f-f5fe-49a5-a327-56ee09c5c6e9","html_url":"https://github.com/saattrupdan/scholarly","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/saattrupdan/scholarly","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saattrupdan%2Fscholarly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saattrupdan%2Fscholarly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saattrupdan%2Fscholarly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saattrupdan%2Fscholarly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saattrupdan","download_url":"https://codeload.github.com/saattrupdan/scholarly/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saattrupdan%2Fscholarly/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269393638,"owners_count":24409755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arxiv","arxiv-api","classification","multilabel","neural-network","scientific-papers"],"created_at":"2024-11-21T12:51:38.503Z","updated_at":"2025-08-08T08:54:08.671Z","avatar_url":"https://github.com/saattrupdan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scholarly\nCategory classification of scientific papers. \n\nGiven a title and an abstract of a paper, the model will predict a list of categories to which the paper belongs. These categories are the 148 categories used on [arXiv](https://arxiv.org).\n\n## Usage\nA demonstration of the model can be found at [ttilt.dk/scholarly](https://www.ttilt.dk/scholarly). Note that you're also free to write LaTeX equations like $\\frac{1}{5}$.\n\nA REST API is also available at the same endpoint, with arguments `title` and `abstract`. You will then receive a JSON response containing a list of lists, with each inner list containing the category id, category description and the probability. The list will only include results with probabilities at least 50%, and the list is sorted descending by probability. [Here](https://saattrupdan.pythonanywhere.com/scholarly?title=\"test\"\u0026abstract=\"test\") is an example of a query.\n\n## Performance\nThe score that I was using was the *sample-average F1 score*, which means that for every sample I'm computing the F1 score of the predictions of the sample (note that we are in a [multilabel](https://en.wikipedia.org/wiki/Multi-label_classification) setup), and averaging that over all the samples. If this was a [multiclass](https://en.wikipedia.org/wiki/Multiclass_classification) setup (in particular binary classification) then this would simply correspond to accuracy. The difference is that in a multilabel setup the model can be *partially* correct, if it correctly predicts some of the categories.\n\nThe model ended up achieving a ~93% and ~65% validation sample-average F1 score on the master categories and all the categories, respectively. Training the model requires ~17GB memory and it takes roughly a day to train on an Nvidia P100 GPU. This was trained on the [BlueCrystal Phase 4 compute cluster](https://www.acrc.bris.ac.uk/acrc/phase4.htm) at University of Bristol, UK.\n\n## Documentation and data\nThis model was trained on all titles and abstracts from all of [arXiv](https://arxiv.org) up to and including year 2019, which were all scraped from their API. The scraping script can be found in `arxiv_scraper.py`. All the data can be found at\n\n\u003cp align=center\u003e\n  \u003ca href=\"https://filedn.com/lRBwPhPxgV74tO0rDoe8SpH/scholarly_data\"\u003e\n    https://filedn.com/lRBwPhPxgV74tO0rDoe8SpH/scholarly_data\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\nThe main data file is the SQLite file `arxiv_data.db`, which contains 6 tables:\n  - `cats`, containing the 148 arXiv categories\n  - `master_cats`, containing the arXiv \"master categories\", which are 6 aggregates of the categories into things like Mathematics and Physics\n  - `papers`, containing the id, date, title and abstract of papers\n  - `papers_cats`, which links papers to their categories\n  - `authors`, containing author names\n  - `papers_authors`, which links authors to their papers\n\nFrom this database I extracted the dataset `arxiv_data.tsv`, which contains the title and abstract for every paper in the database, along with a binary column for every category, denoting whether a paper belongs to that category. LaTeX equations in titles and abstracts have been replaced by \"-EQN-\" at this stage. Everything related to the database can be found in the `db.py` script.\n\nA preprocessed dataset is also available, `arxiv_data_pp.tsv`, in which the titles and abstracts have been merged as \"-TITLE_START- {title} -TITLE_END- -ABSTRACT_START- {abstract} -ABSTRACT_END-\", and the texts have been tokenised using the SpaCy en_core_web_sm tokeniser. The resulting texts have all tokens separated by spaces, so that simply splitting the texts by whitespace will yield the tokenised versions. The preprocessing is done in the `data.py` script.\n\nTwo JSON files, `cats.json` and `mcat_dict.json` are also available, which are basically the `cats` table from the database and a dictionary to convert from a category to its master category, respectively.\n\nI trained FastText vectors on the entire corpus, and the resulting model can be found as `fasttext_model.bin`, with the vectors themselves belonging to the text file `fasttext`. The script I used to train these can be found in `train_fasttext.py`.\n\nIn case you're like me and are having trouble working with the entire dataset, there's also the `arxiv_data_mini_pp.tsv` dataset, which simply consists of 100,000 randomly samples papers from `arxiv_data_pp.tsv`. You can make your own versions of these using the `make_mini.py` script, which constructs the smaller datasets without loading the larger one into memory.\n\nThe model itself is a simplified version of the new [SHA-RNN model](https://arxiv.org/abs/1911.11423), trained from scratch on the [BlueCrystal Phase 4 compute cluster](https://www.acrc.bris.ac.uk/acrc/phase4.htm) at the University of Bristol, UK. All scripts pertaining to the model are `modules.py`, `training.py` and `inference.py`.\n\n## Contact\nIf you have any questions regarding the data or model, please contact me at saattrupdan at gmail dot com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaattrupdan%2Fscholarly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaattrupdan%2Fscholarly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaattrupdan%2Fscholarly/lists"}