{"id":13585585,"url":"https://github.com/internetarchive/fatcat-scholar","last_synced_at":"2025-05-07T16:43:27.351Z","repository":{"id":38361329,"uuid":"300025829","full_name":"internetarchive/fatcat-scholar","owner":"internetarchive","description":"search interface for scholarly works","archived":false,"fork":false,"pushed_at":"2024-02-20T20:44:16.000Z","size":5538,"stargazers_count":75,"open_issues_count":24,"forks_count":14,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-14T02:21:41.432Z","etag":null,"topics":["digital-library","elasticsearch","python","scholarly-communication"],"latest_commit_sha":null,"homepage":"https://scholar.archive.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/internetarchive.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.AGPLv3.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-30T18:54:41.000Z","updated_at":"2024-05-21T19:02:00.702Z","dependencies_parsed_at":"2024-05-21T19:12:08.649Z","dependency_job_id":null,"html_url":"https://github.com/internetarchive/fatcat-scholar","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Ffatcat-scholar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Ffatcat-scholar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Ffatcat-scholar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Ffatcat-scholar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/internetarchive","download_url":"https://codeload.github.com/internetarchive/fatcat-scholar/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225096641,"owners_count":17420293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["digital-library","elasticsearch","python","scholarly-communication"],"created_at":"2024-08-01T15:05:01.758Z","updated_at":"2024-11-17T22:14:08.194Z","avatar_url":"https://github.com/internetarchive.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"src/scholar/static/scholar-vaporwave-logo.png\"\u003e\n\u003c/div\u003e\n\n# Internet Archive Scholar + Fatcat\n\nIA Scholar is an effort within the [Internet Archive](https://archive.org) to track, preserve, index, and serve scholarly articles.\n\nOur focus is on [Open Access](https://en.wikipedia.org/wiki/Open_access) content that might otherwise disappear from the web, but we also focus on building an open bibliographic database of all scholarly content.\n\nThis is source code for [scholar.archive.org](https://scholar.archive.org), a\nfull-text web search interface over the 25+ million open research papers in the\nInternet Archive, and [https://scholar.archive.org/fatcat](https://scholar.archive.org/fatcat), a web front-end to the [fatcat](https://github.com/internetarchive/fatcat/) bibliographic database.\n\n## Overview\n\nThis repository is fairly small and contains:\n\n- `src/scholar/`: Python code for web service and indexing pipeline\n- `src/scholar/templates/`: HTML template for web interface\n- `src/scholar/fatcat`: web frontend to [fatcat](https://github.com/internetarchive/fatcat)\n- `tests/`: Python test files\n- `proposals/`: design documentation and change proposals\n- `data/`: empty directory for indexing pipeline\n\nA data pipeline converts groups of one or more fatcat \"release\" entities\n(grouped under a single \"work\" entity) into a single search index document.\nElasticsearch is used as the full-text search engine. A simple web interface\nparses search requests and formats Elasticsearch results with highlights and\nfirst-page thumbnails.\n\n## Getting Started for Developers\n\nYou'll need `python3`. We test against 3.11; your mileage may vary with older pythons. Ensure that `pip` and `venv` modules are available (these need to be installed manually via `apt` on Debian).\n\nMost tasks are run using a Makefile; `make help` will show all options.\n\nWorking on the indexing pipeline effectively requires internal access to the\nInternet Archive cluster and services, though some contributions and bugfixes\nare probably possible without staff access.\n\nTo install dependencies for the first time run:\n\n    make dep\n\nthen run the tests (to ensure everything is working):\n\n    make test\n\nTo start the web interface run:\n\n    make serve\n\nWhile developing the web interface, you will almost certainly need an example\ndatabase running locally. A docker-compose file in `extra/docker/` can be used\nto run Elasticsearch 7.x locally. The `make dev-index` command will reset the\nlocal index with the correct schema mapping, and index any intermediate files\nin the `./data/` directory. We don't have an out-of-the-box solution for non-IA\nstaff at this step (yet).\n\nAfter making changes to any user interface strings, the interface translation\nfile (\".pot\") needs to be updated with `make extract-i18n`. When these changes\nare merged to master, the Weblate translation system will be updated\nautomatically.\n\nThis repository uses `ruff` for code formatting and `mypy` for type checking;\nplease run `make fmt` and `make lint` for submitting a pull request.\n\n\n## Contributing\n\nSoftware, copy-editing, translation, and other contributions to this repository\nare welcome!\n\nThe web interface is translated using the Weblate platform, at\n[internetarchive/fatcat-scholar](https://hosted.weblate.org/projects/internetarchive/fatcat-scholar/)\n\nThe software license for this repository is Affero General Public License v3+\n(APGL 3+), as described in the `LICENSE.md` file. We ask that you acknowledge\nthe license terms when making your first contribution.\n\nFor software developers, the \"help wanted\" tag in Github Issues is a way to\ndiscover bugs and tasks that external folks could contribute to.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finternetarchive%2Ffatcat-scholar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finternetarchive%2Ffatcat-scholar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finternetarchive%2Ffatcat-scholar/lists"}