{"id":27317338,"url":"https://github.com/rewong03/xtract_file_service","last_synced_at":"2026-05-05T22:43:06.451Z","repository":{"id":233259503,"uuid":"202750900","full_name":"rewong03/xtract_file_service","owner":"rewong03","description":"Code for XtractHub server that processes individual files.","archived":false,"fork":false,"pushed_at":"2024-06-17T22:45:03.000Z","size":136,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-12T08:55:44.290Z","etag":null,"topics":["extractor","flask","globus","process-metadata","redis","xtracthub"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rewong03.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-08-16T15:17:06.000Z","updated_at":"2019-09-08T18:33:30.000Z","dependencies_parsed_at":"2024-04-15T08:35:18.447Z","dependency_job_id":"b5042861-c891-4a9b-8d65-4e1a99ac05c5","html_url":"https://github.com/rewong03/xtract_file_service","commit_stats":null,"previous_names":["rewong03/xtract_file_service"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rewong03%2Fxtract_file_service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rewong03%2Fxtract_file_service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rewong03%2Fxtract_file_service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rewong03%2Fxtract_file_service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rewong03","download_url":"https://codeload.github.com/rewong03/xtract_file_service/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248543880,"owners_count":21121838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["extractor","flask","globus","process-metadata","redis","xtracthub"],"created_at":"2025-04-12T08:55:47.887Z","updated_at":"2026-05-05T22:43:06.423Z","avatar_url":"https://github.com/rewong03.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# XtractHub File Service\nXtractHub File Service is a user metadata file service based on [XtractHub](https://github.com/xtracthub)\nfrom Globus Labs. XtractHub File Service allows users to manage files, process metadata from uploaded files using \nXtractHub, and view extracted file metadata using a REST API.\n\n## Getting Started\nThese instructions will get a copy of XtractHub File Service running on your local machine for development and testing \npurposes.\n\n### Prerequisites\n- Redis (available [here](https://redis.io/download))\n- Docker (available [here](https://docs.docker.com/install/))\n\n### Installation\nFirst, clone this repository and activate a virtual environment:\n```\ngit clone https://github.com/rewong03/xtract_file_service\ncd xtract_file_service\npython3 -m venv venv\nsource venv/bin/activate\n```\nNext, install the requirements:\n```\npip install -r requirements.txt\ndeactivate\n```\n\n### Running XtractHub File Service\nFirst, open a terminal and start a redis server:\n```\ncd /path/to/redis/\nsrc/redis-server\n```\nThen in a second terminal, start a celery worker:\n```\ncd /path/to/xtract_file_service/\nsource venv/bin/activate\nvenv/bin/celery -A app.celery_app worker -Q celery,priority\n```\nIn a third terminal, start the flask app:\n```\ncd /path/to/xtract_file_service\nsource venv/bin/activate\nvenv/bin/flask run\n```\n\n## Interacting with the server\nThe server is a REST API with no GUI or HTML. All interactions are done using `curl` or the xtract_file_service command\nline interface. This section is documentation of how to interact with all of the server's features with `curl`.  \n\n\n**While you *can* use `curl` to interact with the server, it is recommended that you use the xtract_file_service CLI \ninstead. To learn how to use the CLI, click [here](xfs_cli/README.md).**\n\n### Creating a user:\nTo create a new user, run:\n```\ncurl -X POST -d '{\"Username\": \"your_username\", \"Email\": \"your_email\", \"Password\": \"your_password\"}' http://localhost:5000\n```\n- This will either return a success message, or a failure message if your input wasn't formatted correctly or if the \nusername has already been taken.\n\n### Logging in:\nTo login, run:\n```\ncurl -X GET -d '{\"Username\": \"your_username\", \"Password\": \"your_password\"}' http://localhost:5000/login\n```\n- This will return a user ID that will become your `\"Authentication\"` for other server interactions. Save or copy this \nso you won't have to login again.\n\n### Deleting an account:\nTo delete an account, run:\n```\ncurl -X DELETE -H \"Authentication: your_authentication\" -d '{\"Username\": \"your_username\", \"Password\": \"your_password\"}' http://localhost:5000/delete_user\n```\n\n### Viewing, uploading, and deleting files:\nTo upload files for automatic metadata processing, run:\n```\ncurl -X POST -H \"Authentication: your_authentication\" -H \"Extractor: extractor_name\" -F \"file=@/local/file/path.txt\" http://localhost:5000/files\n```\n- **Note: See the available extractors section below.**  \n- This will return a task ID for the metadata processing job, which can be used to view the status of your job. Omitting\n`extractor_name` will still return a task ID but will not result in any metadata being processed.\n- Compressed files will be automatically be decompressed. \n\nTo view uploaded files, run:\n```\ncurl -X GET -H \"Authentication: your_authentication\" http://localhost:5000/files\n```\n- This will return a string containing the names of uploaded files as well as their size.\n\nTo delete files, run:\n```\ncurl -X DELETE -H \"Authentication: your_authentication\" -d filename http://localhost:5000/files\n```\n- This will return a success message or an error message if the file doesn't exist.\n\n### Viewing, processing, and deleting metadata:\nTo view processed metadata, run:\n```\ncurl -X GET -H \"Authentication: your_authentication\" -d filename http://localhost:5000/metadata\n```\n- This will return all metadata extracted for a given file.\n\nTo process metadata for an uploaded file, run:\n```\ncurl -X POST -H \"Authentication: your_authentication\" -d '{\"Filename\": \"filename\", \"Extractor\": \"extractor_name\"}' http://localhost:5000/metadata\n```\n- **Note: See the available extractors section below.**  \n- **Note: You cannot process metadata for a file and extractor if you have already done so for that file and extractor\ncombination.**\n- This will return a task ID for the metadata processing job, which can be used to view the status of your job. \nOmitting `extractor_name` will still return a task ID but will not result in any metadata being processed.\n\nTo delete all metadata for a file, run:\n```\ncurl -X DELETE -H \"Authentication: your_authentication\" -d filename http://localhost:5000/metadata\n```\n- An additional `-H \"Extractor: extractor` header can be passed to specify extractor metadata to delete for `filename`. \nIf omitted, all metadata for `filename` will be deleted.\n\n### Viewing task status:\nTo view a task status, run:\n```\ncurl -X GET -d task_id http://localhost:5000/tasks\n```\n- This returns a status message for the given task ID. \n\n## Available Extractors:\n`tabular`:\n- Can process tabular/columnar files (.tsv, .csv, etc.).\n- Returns preamble, headers, and means, medians, modes, max, min for each column. \n    - Tabular files containing preamble will automatically process the preamble using the\n    `keyword` extractor. \n\n`keyword`:\n- Can process text files.\n- Returns a list of keywords as well as scores for those keywords (calculated using (degree)/(frequency)).\n\n`jsonxml`:\n- Can process .json/.xml style files.\n- Returns depth, headers, columns, and all text found in the file.\n    - Json/xml files containing text will automatically process the text using the `keyword`\n    extractor. \n\n`netcdf`:\n- Can process netcdf style files.\n- Returns attributes, dimensions, size, and variables.\n\n`image`:\n- Can process any image file.\n- Returns a classification of plot, map, map-plot, graphic, or photograph.\n\n`map`:\n- Can process a map image (works better for maps with text or coordinates).\n- Returns any text found from the image as well as locations on the map.\n\n`matio`:\n- Can process material science files.\n- `matio` was built using the [MaterialsIO](https://github.com/materials-data-facility/MaterialsIO) library. For a more\nextensive list of accepted file types and outputs, please visit their [documentation](https://materialsio.readthedocs.io/en/latest/parsers.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frewong03%2Fxtract_file_service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frewong03%2Fxtract_file_service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frewong03%2Fxtract_file_service/lists"}