{"id":16661052,"url":"https://github.com/agrafix/grabcite","last_synced_at":"2025-09-12T02:32:41.075Z","repository":{"id":66320070,"uuid":"91705443","full_name":"agrafix/grabcite","owner":"agrafix","description":"Haskell: Library/Executable to extract citations from scientific papers","archived":false,"fork":false,"pushed_at":"2018-02-13T06:08:57.000Z","size":1186,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-11T20:45:23.490Z","etag":null,"topics":["citation","extraction","haskell","nlp","paper","text"],"latest_commit_sha":null,"homepage":"","language":"TeX","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/agrafix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-18T14:48:57.000Z","updated_at":"2021-06-17T13:17:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"57cfd151-4f26-450f-b5dc-140618930ff0","html_url":"https://github.com/agrafix/grabcite","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/agrafix/grabcite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrafix%2Fgrabcite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrafix%2Fgrabcite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrafix%2Fgrabcite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrafix%2Fgrabcite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/agrafix","download_url":"https://codeload.github.com/agrafix/grabcite/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrafix%2Fgrabcite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274744019,"owners_count":25341136,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["citation","extraction","haskell","nlp","paper","text"],"created_at":"2024-10-12T10:33:15.688Z","updated_at":"2025-09-12T02:32:41.053Z","avatar_url":"https://github.com/agrafix.png","language":"TeX","funding_links":[],"categories":[],"sub_categories":[],"readme":"# grabcite\n\n[![CircleCI](https://circleci.com/gh/agrafix/grabcite.svg?style=svg)](https://circleci.com/gh/agrafix/grabcite)\n\nGrabCite is a tool to generate data sets for tasks like citation recommendation. It supports various input formats such as:\n\n* Plain Text (i.E. pdftotext output)\n* PDF\n* Grobid Tei XML\n* Ice Cite JSON\n* CiteSeerX Databases (experimental)\n* TeX (+ Bib) Files\n\nThe output format is split into 3 files per input file, one containing the individual sentences and global citation markers, one containing meta information for citation markers and one containing citation markers for the paper itself. To use the tool, you can build it from source.\n\n## Building from source\n\n```bash\n# install haskell stack\ncurl -sSL https://get.haskellstack.org/ | sh\n\n# clone the repo\ngit clone https://github.com/agrafix/grabcite \u0026\u0026 cd grabcite\n\n# install dependencies and build\nstack setup\nstack build\n\n# run it\nstack exec -- grabcite-datagen --help\n```\n\n## Command line interface\n\nThe main tool is called `grabcite-datagen`. It can be run from any linux command line. There are two supported use cases:\n\n### Unpack an ArXiv.org dump\n\nTo unpack an arxiv.org dump (i.E. extract all archives and find the correct .tex/.bib file), you can run:\n\n```bash\ngrabcite-datagen --in-mode InPdf --in-dir [DUMP_DIR] --recursive --arxiv-to-tex-mode --arxiv-meta-xml [LOCATION_OF_META_XML] --out-dir [TARGET_DIR] --jobs [JOBS]\n```\n\n### Generating a data set\n\nTo build a data set, first figure out what your input is shaped. As stated above we support:\n\n* `InText`: Plain Text (i.E. pdftotext output)\n* `InPdf`: PDF\n* `InGrobid`: Grobid Tei XML\n* `InIceCite`: Ice Cite JSON\n* `InIceCiteBasic`: Ice Cite JSON, but ignore the roles detected\n* `InCiteSeerX`: CiteSeerX Databases (experimental)\n* `InTex`: TeX (+ Bib) Files\n\nThen, run the command line:\n\n```bash\ngrabcite-datagen --in-mode [MODE_FROM_ABOVE] --in-dir [DATA_DIR] --recursive --out-dir [OUT_DIR] --jobs 4\n```\n\nIf you which to write the output to the database, add the following flags:\n\n```bash\n--out-db [POSTGRESQL_CONNECTION_STRONG] --db-mig-dir [MIGRATION_FILES] --data-source-name [NAME_OF_DATA_SOURCE]\n```\n\n`[MIGRATION_FILES]` is the path to the `database` folder in this repository. You can add `--db-overwrite` if you wish to overwrite previous data with the same `data-source-name`.\n\nExperimental support for `InCiteSeerX` requires to supply `--in-db-conn-str` and point to the CiteSeerX Oracle DB. This has not properly been tested yet.\n\nAdd `--debug` to output debug messages.\n\n### Hints\n\nTo speed up the process between runs, you can copy the `ref_cache.json` from a previous run into your new output directory before launching the task. This will prevent unneeded DBLP-ID lookups.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagrafix%2Fgrabcite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagrafix%2Fgrabcite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagrafix%2Fgrabcite/lists"}