https://github.com/quantaji/sciduet-rebuild
This is the repo for my rebuild of the dataset SciDuet from IBM
https://github.com/quantaji/sciduet-rebuild
Last synced: about 1 year ago
JSON representation
This is the repo for my rebuild of the dataset SciDuet from IBM
- Host: GitHub
- URL: https://github.com/quantaji/sciduet-rebuild
- Owner: quantaji
- Created: 2023-03-18T15:48:14.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-05-11T21:34:50.000Z (about 3 years ago)
- Last Synced: 2025-03-12T13:13:41.715Z (over 1 year ago)
- Language: Python
- Size: 797 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# sciduet-rebuild
This is the rebuild of [SciDuet](https://github.com/IBM/document2slides) dataset. This repo includes paper and slides of NeurIPS {2018, 2019, 2021, 2022}, ICML {2020, 2021, 2022}, ICLR {2021, 2022}. This repo only provide the download urls for paper and slides, and also the preprocessing code similar to original SciDuet dataset.
## Get the data
```shell
# step 1: download pdfs, need webarchieve python package
python core/download_pdfs.py
# step 2: extract xml from paper pdfs, need java8 and GROBID running in the background
python core/paper2xml.py
# step 3: extract figures and their captions, need sbt installed
python core/paper2fig.py
# step 4: remove failure case
python core/remove_img_failure.py
# step 5: merge xml and figures into json file
python core/merge2json.py
```