{"id":18518334,"url":"https://github.com/arxiv/arxiv-compiler","last_synced_at":"2025-04-09T08:31:40.015Z","repository":{"id":33261247,"uuid":"146316720","full_name":"arXiv/arxiv-compiler","owner":"arXiv","description":"Service to compile LaTeX source packages into PDF, PostScript, and other formats","archived":true,"fork":false,"pushed_at":"2025-01-28T19:57:16.000Z","size":1170,"stargazers_count":12,"open_issues_count":12,"forks_count":2,"subscribers_count":14,"default_branch":"develop","last_synced_at":"2025-03-28T17:18:00.686Z","etag":null,"topics":["arxiv-ce","flask","latex","pdf","python"],"latest_commit_sha":null,"homepage":"https://arxiv.github.io/arxiv-compiler/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arXiv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-27T15:24:46.000Z","updated_at":"2025-01-28T19:59:26.000Z","dependencies_parsed_at":"2025-01-28T20:38:41.177Z","dependency_job_id":null,"html_url":"https://github.com/arXiv/arxiv-compiler","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arXiv%2Farxiv-compiler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arXiv%2Farxiv-compiler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arXiv%2Farxiv-compiler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arXiv%2Farxiv-compiler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arXiv","download_url":"https://codeload.github.com/arXiv/arxiv-compiler/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248003155,"owners_count":21031742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arxiv-ce","flask","latex","pdf","python"],"created_at":"2024-11-06T17:12:56.563Z","updated_at":"2025-04-09T08:31:40.005Z","avatar_url":"https://github.com/arXiv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# arXiv compiler service\n\n*NOTE:* This service is not under active development, and should not be used for attempting to build a \"local arXiv-like\" \ncompiler. For such a service, please see the [arXiv/submission-tools](https://github.com/arXiv/submission-tools) project. \n\n__This project has been archived, as of 2025-01-28__\n\nThe build service compiles LaTeX source packages into PDF, PostScript,\nand other formats. This service encompasses the arXiv TeX tree. Compilation\nlogs are also made available, for example to provide submitters feedback about\ncompilation failures or warnings.\n\nThere are three moving parts:\n- The compiler service API, a Flask web app that handles new requests for  \n  compilation and makes the results available (or not).\n- The compiler worker, a Celery app that grabs sources, dispatches compilation,\n  and stores the results.\n- The converter Docker image, which does the actual work of compilation and is\n  executed by the compiler worker. This contains the arXiv TeX tree.\n\nIn addition, the following infrastructure parts are required:\n- Redis, used as a task queue between the API and the worker.\n- S3, used to store the result of compilation tasks.\n\n## TODO\n\n- [ ] Update the ``schema/``, and implement resource URLs in the response data.\n\n## Running the compiler service locally\n\nThe easiest way to get up and running is to launch the whole service group\nusing Docker Compose. You will need to pull the\n[converter](https://github.com/arXiv/arxiv-converter/tree/develop) image ahead\nof time, or it will be  pulled on the first compilation request. Note that you\nwill need to provide valid AWS credentials with ECR read access as env vars\nwhen starting docker-compose.\n\nIf you do not have an instance of the file manager service running, you can\ntry compiling published sources on the public arXiv.org site.\n\nYou will also need a directory that the worker can use as /tmp space.\n\nFor example:\n\n```bash\n$ mkdir /tmp/compilestuff     # Docker needs access to this.\n$ export DIND_SOURCE_ROOT=/tmp/compilestuff\n$ export CONVERTER_DOCKER_IMAGE=[name (including transport) of converter image]\n$ export FILEMANAGER_ENDPOINT=https://arxiv.org  # Get public sources.\n$ export FILEMANAGER_CONTENT_PATH=/src/{source_id}\n$ export AWS_ACCESS_KEY_ID={access key with ECR access}\n$ export AWS_SECRET_ACCESS_KEY={secret key with ECR access}\n```\n\nAnd then run with:\n\n```bash\n$ docker-compose build    # Build the local images.\n$ docker-compose up       # Start the service group.\n```\n\nGive it a few seconds; Localstack needs to come up (provides a local S3), and\na bucket will be created.\n\n### Authentication + Authorization\n\nTo use the API you will need an auth token with scopes ``compile:read`` and\n``compile:create``. The easiest way to generate one of these is to use the\nhelper script\n[here](https://github.com/arXiv/arxiv-auth/blob/develop/generate_token.py).\nMake sure that you use the same ``JWT_SECRET`` that is used in\n``docker-compose.yml``.\n\n\n```bash\nJWT_SECRET=foosecret pipenv run python generate_token.py\n```\n\nYou should pass this token as the value of the ``Authorization`` header in\nall requests to the API. For example:\n\n```bash\ncurl -XPOST -H \"Authorization: [auth token]\" http://127.0.0.1:8000/...\n```\n\nFor requests in your browser, you can use something like\n[requestly](https://chrome.google.com/webstore/detail/requestly-redirect-url-mo/mdnleldcmiljblolnjhpnblkcekpdkpa?hl=en)\nto automatically add the auth header to your requests.\n\n[Advanced Rest Client](https://install.advancedrestclient.com/install) is also\nhandy for this purpose.\n\n### Request compilation\n\n```bash\n$ curl -XPOST -i -H 'Authorization: {JWT}' -d '{\"source_id\":\"1602.00123\",\"checksum\":\"\\\"Tue, 02 Feb 2016 01:04:33 GMT\\\"\",\"format\":\"pdf\"}' http://localhost:8000/\nHTTP/1.0 202 ACCEPTED\nContent-Type: application/json\nContent-Length: 3\nLocation: http://localhost:8000/task/53cccb2e-faf7-4dfa-b8de-63854bd08b0a\nServer: Werkzeug/0.14.1 Python/3.6.4\nDate: Mon, 12 Nov 2018 10:41:42 GMT\n\n{}\n```\n\nYou should get ``202 Accepted`` with headers that look like this:\n\n```bash\ncontent-type: application/json\ncontent-length: 3\nlocation: http://127.0.0.1:8000/1901.00123/%22Thu%2C%2003%20Jan%202019%2001:04:33%20GMT%22/pdf\nserver: Werkzeug/0.15.2 Python/3.6.5\ndate: Wed, 08 May 2019 20:49:26 GMT\n```\n\nThe ``location`` is the status resource for the compilation task.\n\n#### What's with the ``checksum`` entry in the request payload?\n\nNote that the compiler service assumes that the checksum of the source package\nis included in the ``ETag`` header (this is the behavior of the file manager\nservice). If you are pulling sources from the core arXiv site (as above), this\nwill be a datestamp instead of a checksum. You can get the current ETag for a\nsource package like this:\n\n```bash\n$ curl -I https://arxiv.org/src/1901.00123\nHTTP/1.1 200 OK\nDate: Wed, 08 May 2019 20:29:32 GMT\nServer: Apache\nETag: \"Thu, 03 Jan 2019 01:04:33 GMT\"\nExpires: Thu, 09 May 2019 00:00:00 GMT\nContent-Encoding: x-gzip\nContent-Disposition: attachment; filename=\"arXiv-1901-00123v1.tar.gz\"\nStrict-Transport-Security: max-age=31536000\nSet-Cookie: browser=128.84.116.178.1557347372434122; path=/; max-age=946080000; domain=.arxiv.org\nLast-Modified: Thu, 03 Jan 2019 01:04:33 GMT\nContent-Length: 111118\nVary: User-Agent\nContent-Type: application/x-eprint-tar\n```\n\n**Note that the quotation marks are included as part of the value of the ETag/checksum\nfield in the request.**\n\n### Checking the compilation status endpoint\n\nYou can request the status endpoint like this:\n\n```bash\n$ curl -H 'Authorization: {JWT}' http://127.0.0.1:8000/1901.00123/%22Thu%2C%2003%20Jan%202019%2001:04:33%20GMT%22/pdf\n{\n    \"checksum\": \"\\\"Thu, 03 Jan 2019 01:04:33 GMT\\\"\",\n    \"description\": \"\",\n    \"output_format\": \"pdf\",\n    \"owner\": null,\n    \"reason\": null,\n    \"size_bytes\": 598859,\n    \"source_id\": \"1901.00123\",\n    \"status\": \"completed\",\n    \"task_id\": \"1901.00123/\\\"Thu, 03 Jan 2019 01:04:33 GMT\\\"/pdf\"\n}\n```\n\n### Getting the content\n\nYou can get the content at:\nhttp://127.0.0.1:8000/1901.00123/%22Thu%2C%2003%20Jan%202019%2001:04:33%20GMT%22/pdf/content\n\n\n## Documentation\n\nThe latest documentation can be found at\nhttps://arxiv.github.io/arxiv-compiler.\n\n### Building\n\n```bash\nsphinx-apidoc -o docs/source/api/ -e -f -M compiler *test*/*\ncd docs/\nmake html SPHINXBUILD=$(pipenv --venv)/bin/sphinx-build\n```\n\n\n## License\n\nSee [LICENSE](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farxiv%2Farxiv-compiler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farxiv%2Farxiv-compiler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farxiv%2Farxiv-compiler/lists"}