{"id":18518345,"url":"https://github.com/arxiv/arxiv-filemanager","last_synced_at":"2025-04-09T08:31:39.220Z","repository":{"id":34058421,"uuid":"113456994","full_name":"arXiv/arxiv-filemanager","owner":"arXiv","description":"File management and sanitization service","archived":false,"fork":false,"pushed_at":"2022-12-08T05:50:07.000Z","size":43458,"stargazers_count":2,"open_issues_count":12,"forks_count":8,"subscribers_count":13,"default_branch":"develop","last_synced_at":"2025-03-24T03:53:09.097Z","etag":null,"topics":["arxiv-ng","flask","part-of-ng-submit","python"],"latest_commit_sha":null,"homepage":"","language":"PostScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arXiv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-07T13:55:34.000Z","updated_at":"2024-09-09T20:38:50.000Z","dependencies_parsed_at":"2022-08-24T13:30:46.383Z","dependency_job_id":null,"html_url":"https://github.com/arXiv/arxiv-filemanager","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arXiv%2Farxiv-filemanager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arXiv%2Farxiv-filemanager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arXiv%2Farxiv-filemanager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arXiv%2Farxiv-filemanager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arXiv","download_url":"https://codeload.github.com/arXiv/arxiv-filemanager/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248003147,"owners_count":21031742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arxiv-ng","flask","part-of-ng-submit","python"],"created_at":"2024-11-06T17:13:05.119Z","updated_at":"2025-04-09T08:31:36.958Z","avatar_url":"https://github.com/arXiv.png","language":"PostScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# arXiv filemanager service\n\nFile management and sanitization service\n\n### Quick Start Guide\n\nThere are multiple ways to run the file management server: docker,\ndocker-compose, and local flask development server.\n\n### Authentication + authorization\n\nThis app uses ``arxiv.users`` to implement authentication and authorization\nrequirements. You will need to generate and use an auth token for development\nand manual testiing. Be sure to use the same value for ``JWT_SECRET`` when\ngenerating the auth token and when running the application.\n\n```bash\nJWT_SECRET=foosecret generate-token\n```\n\nThe following scopes are used in this application:\n\n| Scope                           | Description                               |\n|---------------------------------|-------------------------------------------|\n| ``upload:read``                 | View the content of an upload workspace.  |\n| ``upload:create``               | Create a new workspace.                   |\n| ``upload:update``               | Upload files to to a workspace.           |\n| ``upload:release``              | Release a workspace (pre-delete).         |\n| ``upload:delete``               | Can delete files from a workspace.        |\n| ``upload:delete_workspace``     | Delete an entire workspace.               |\n| ``upload:read_logs``            | Can read logs for a workspace.            |\n| ``upload:read_service_logs``    | Can read service logs.                    |\n| ``upload:create_checkpoint``    | Create a checkpoint.                      |\n| ``upload:delete_checkpoint``    | Delete a checkpoint.                      |\n| ``upload:read_checkpoints``     | Read from checkpoints.                    |\n| ``upload:restore_checkpoint``   | Restore workspace to a checkpoint.        |\n\n\n\n### Docker\n\nPrerequisites: (Docker application, arxiv-base)\n\n1. Setup [Docker CE using the instructions for your\n   OS](https://docs.docker.com/engine/installation/)\n\nBuild/Run/Test FileManager Docker image:\n\n2. Build the Docker image with:\n\n```bash\ndocker build . -t arxiv-filemanager \\\n    --build-args=BASE_VERSION=[target base version]\n```\n\nwhere ``[target base version]`` is the tag on ``arxiv/base`` against which you\nwish to build. The default should be the most recent version of ``arxiv/base``\nagainst which this service has been tested.\n\n3. Run the application with:\n\n```bash\ndocker run -p 8000:8000 \\\n    --name=filemanager \\\n    -e JWT_SECRET=foosecret \\\n    arxiv-filemanager\n```\n\nNote: (add a `-d` flag to run in daemon mode)\n\n4. Test that the container is working:\n   http://localhost:8000/filemanager/api/status\n5. To shut down the container, press ``ctrl-c`` (or `docker stop filemanager`\n   if in daemon mode).\n6. Each time you change a file, you will need to rebuild the Docker image\n   in order to import the updated files. Alternatively, volume-mount selected\n   parts of your home directory such has .ssh and .gitconfig in your `docker\n   run` command if you wish to be able to push modifications to github.\n\nNote: Local flask development server (described below) detects changed files\nand reloads them into running development server (debug mode).\n\n#### Docker Cleanup\n\nTo purge your container run  `docker rmi filemanager`.\n\nIf you receive the following error:\n\n```\n$ docker rmi c196c3ef21c7\nError response from daemon: conflict: unable to delete c196c3ef21c7 (must be\nforced) - image is being used by stopped container 75bb481b5857\n```\n\nYou will need to issue a remove command for each container that depends on the\nimage you are trying to delete. Run `docker rm CONTAINER_ID` for each stopped\ncontainer until the above error clears\n\nNote: There are commands that will remove images and containers en masse. For\nnow I'll refer you to the Docker documentation.\n\n### Local Flask Deployment\n\nThis section describes launching flask development server and running script to\nload test database.\n\nSometimes Docker adds more overhead than you want, especially when making quick\nchanges. We assume your developer machine already has a version of Python 3.6\nwith `pip`.\n\n1. `pipenv install --dev`\n2. `FLASK_APP=app.py pipenv run python populate_test_database.py`\n3. `JWT_SECRET=foosecret FLASK_APP=app.py FLASK_DEBUG=1 pipenv run flask run`\n4. Test that the app is working: http://localhost:5000/filemanager/api/status\n\n#### Notes on the development server\n\nFlask provides a single-threaded dev server for your enjoyment.\n\nThe entrypoint for this dev server is [``app.py``](app.py) (in the root of the\nproject). Flask expects the path to this entrypoint in the environment variable\n``FLASK_APP``. To run the dev server, try (from the project root):\n\n```bash\n$ JWT_SECRET=foosecret FLASK_APP=app.py FLASK_DEBUG=1 pipenv run flask run\n```\n\n``FLASK_DEBUG=1`` enables a slew of lovely development and debugging features.\nFor example, the dev server automatically restarts when you make changes to the\napplication code.\n\nNote that neither the dev server or the ``app.py`` entrypoint are acceptable\nfor use in production.\n\n#### Load test database\n\nA convenience script [``populate_test_database.py``](populate_test_database.py)\nis provided to set up an on-disk SQLite database and some sample data. You can\nuse this as a starting point for more complex set-up operations (or not). Be\nsure to run this with the ``FLASK_APP`` variable set, e.g.\n\n```bash\n$ FLASK_APP=app.py pipenv run python populate_test_database.py\n```\n\n\n\n### Authorization token\n\nUse the ``generate_token.py`` script to generate an authentication JWT. Be sure\nthat you are using the same secret when running this script as when you run\nthe app.\n\n```bash\n$ JWT_SECRET=foosecret FLASK_APP=app.py pipenv run python generate_token.py\nNumeric user ID: 4\nEmail address: erick@foo.com\nUsername: erick\nFirst name [Jane]:\nLast name [Doe]:\nName suffix [IV]:\nAffiliation [Cornell University]:\nNumeric rank [3]:\nAlpha-2 country code [us]:\nDefault category [astro-ph.GA]:\nSubmission groups (comma delim) [grp_physics]:\nEndorsement categories (comma delim) [astro-ph.CO,astro-ph.GA]:\nAuthorization scope (comma delim) [upload:read,upload:write,upload:admin]:\neyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzZXNzaW9uX2lkIjoiZTljMGQwMDUtMTk1My00YWRiLWE0YzEtYzdmNWY1OGM5YTk4Iiwic3RhcnRfdGltZSI6IjIwMTgtMDgtMDlUMTQ6NDg6MDguNzY2NjUzLTA0OjAwIiwidXNlciI6eyJ1c2VybmFtZSI6ImVyaWNrIiwiZW1haWwiOiJlcmlja0Bmb28uY29tIiwidXNlcl9pZCI6IjQiLCJuYW1lIjp7ImZvcmVuYW1lIjoiSmFuZSIsInN1cm5hbWUiOiJEb2UiLCJzdWZmaXgiOiJJViJ9LCJwcm9maWxlIjp7ImFmZmlsaWF0aW9uIjoiQ29ybmVsbCBVbml2ZXJzaXR5IiwiY291bnRyeSI6InVzIiwicmFuayI6Mywic3VibWlzc2lvbl9ncm91cHMiOlsiZ3JwX3BoeXNpY3MiXSwiZGVmYXVsdF9jYXRlZ29yeSI6eyJhcmNoaXZlIjoiYXN0cm8tcGgiLCJzdWJqZWN0IjoiR0EifSwiaG9tZXBhZ2VfdXJsIjoiIiwicmVtZW1iZXJfbWUiOnRydWV9fSwiY2xpZW50IjpudWxsLCJlbmRfdGltZSI6IjIwMTgtMDgtMTBUMDA6NDg6MDguNzY2NjUzLTA0OjAwIiwiYXV0aG9yaXphdGlvbnMiOnsiY2xhc3NpYyI6MCwiZW5kb3JzZW1lbnRzIjpbW1siYXN0cm8tcGgiLCJDTyJdLG51bGxdLFtbImFzdHJvLXBoIiwiR0EiXSxudWxsXV0sInNjb3BlcyI6W1sidXBsb2FkOnJlYWQiLCJ1cGxvYWQ6d3JpdGUiLCJ1cGxvYWQ6YWRtaW4iXV19LCJpcF9hZGRyZXNzIjpudWxsLCJyZW1vdGVfaG9zdCI6bnVsbCwibm9uY2UiOm51bGx9.aOgRj73TT-zsRvF7gnPPjplJzcnXkKzYzEvMB61jEsY\n```\n\nStart the dev server with:\n\n```\n$ JWT_SECRET=foosecret FLASK_APP=app.py FLASK_DEBUG=1 pipenv run flask run\n```\n\nUse the (rather long) token in your requests to authorized endpoints. Set the\nheader ``Authorization: [token]``.\n\n### Push File Manager Image to Docker Hub.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farxiv%2Farxiv-filemanager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farxiv%2Farxiv-filemanager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farxiv%2Farxiv-filemanager/lists"}