{"id":41980166,"url":"https://github.com/globus/searchable-files-demo","last_synced_at":"2026-01-25T23:40:49.912Z","repository":{"id":37953355,"uuid":"363257589","full_name":"globus/searchable-files-demo","owner":"globus","description":"Searchable Files; a demo application using Globus Search","archived":false,"fork":false,"pushed_at":"2026-01-19T18:59:47.000Z","size":652,"stargazers_count":5,"open_issues_count":2,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-20T00:54:27.530Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/globus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-04-30T20:59:19.000Z","updated_at":"2025-04-15T15:27:05.000Z","dependencies_parsed_at":"2024-02-19T23:28:45.135Z","dependency_job_id":"0f5eef4e-0d92-46b9-89da-6e42f545ffa1","html_url":"https://github.com/globus/searchable-files-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/globus/searchable-files-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/globus%2Fsearchable-files-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/globus%2Fsearchable-files-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/globus%2Fsearchable-files-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/globus%2Fsearchable-files-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/globus","download_url":"https://codeload.github.com/globus/searchable-files-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/globus%2Fsearchable-files-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28761816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-25T23:06:19.311Z","status":"ssl_error","status_checked_at":"2026-01-25T23:03:50.555Z","response_time":113,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-25T23:40:49.850Z","updated_at":"2026-01-25T23:40:49.905Z","avatar_url":"https://github.com/globus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Searchable Files (demo)\n\nThis demo application shows how Globus Search can be used to build an index of\nfile metadata. Similar to the unix `find` command, it lets you search for files\nin a directory.\nUnlike `find`, however, the user searching the files does not need shell access\nto the server where files are stored.\n\n## Companion Doc\n\nGlobus provides a\n[companion doc](https://docs.globus.org/api/search/guides/searchable_files/)\nwhich expands upon the content in this demo.\n\nThe doc covers some of the motivation and background for this app, as well as\nsome ideas about ways in which the demo can be adapted or extended.\n\n## Architecture\n\nThe demo app is broken up into four main components:\n\n- the **Extractor** (`src/searchable_files/extractor.py`)\n\nParses file metadata and contents into chunks, and formats that data into JSON\nfiles.\n\nBy default, this parses content in `data/files` and outputs to\n`output/extracted/`.\n\n- the **Assembler** (`src/searchable_files/assembler.py`)\n\nCombines the output of the Extractor with visibility information\nto produce ingest documents for Globus Search. An ingest document is data\nformatted for submission to Globus Search, containing searchable data and\nvisibility information for who is allowed to search on and view different parts\nof the data.\n\nThe visibility information and additional annotations used to augment the data from\nthe Extractor is loaded from configuration, located by default in\n`data/config/assembler.yaml`.\n\nBy default, the Assembler reads data from `output/extracted/` and outputs to\n`output/assembled/`.\n\n- the **Submitter** (`src/searchable_files/submit.py`)\n\nThe Submitter sends ingest documents to the Search service.\n\nBy default, the Submitter reads data from `output/assembled/` and writes\ninformation to `output/task_submit/`.\n\n- the **Watcher** (`src/searchable_files/watcher.py`)\n\nThe Watcher monitors tasks in Globus Search and waits for completion or failure.\n\nBy default, the Watcher reads task IDs from `output/task_submit/` and outputs to\nstandard output the number of passing and failing\ntasks (or only success if no tasks fail).\n\n## Prerequisites\n\nThe following software is required in order to install and run the\nSearchable Files app:\n\n- python3.6+\n- virtualenv\n- pip\n- make\n\n## Download\n\nTo grab the latest version of the app, clone this repo or download it from\nGitHub.\n\n- [download searchable-files-demo-main.zip](https://github.com/globus/searchable-files-demo/archive/refs/heads/main.zip)\n- [download searchable-files-demo-main.tar.gz](https://github.com/globus/searchable-files-demo/archive/refs/heads/main.tar.gz)\n\n## Installation\n\nRun\n\n    make install\n\nThis will create a virtualenv and install the necessary dependencies.\n\nIt will also create a script named `searchable-files`.\n\n\u003e **WARNING**: Always run `searchable-files` from the top level of the\n\u003e repository, unless you pass additional options. The script's defaults are all\n\u003e written as relative paths with respect to this directory.\n\n### Setup\n\nBefore running other steps, run\n\n    ./searchable-files create-index\n\nThis will create a new index for you to use with the Searchable Files demo app.\nIts index ID will be stored by default in `data/index_info.json`.\n\nTo retrieve the index ID and other index information, run\n\n    ./searchable-files show-index\n\n### Running the Workflow\n\nEach component of the Searchable Files app is run with a separate\nsubcommand. Each supports a `--help` option for full details on its\nusage.\n\n    ./searchable-files extract --help\n    ./searchable-files assemble --help\n    ./searchable-files submit --help\n    ./searchable-files watch --help\n\nThe order of these commands matters, as each command's output is the input to\nthe next command.\n\nThe entire workflow can run in one line by simply running each command\nback-to-back, thusly:\n\n    ./searchable-files extract \u0026\u0026 ./searchable-files assemble \u0026\u0026 ./searchable-files submit \u0026\u0026 ./searchable-files watch\n\n### Querying Results\n\nThe Searchable Files demo app includes a query command which you can use to\nsearch your files. Search results will be output in the JSON format produced by\nthe Globus Search service.\n\nSee\n\n    ./searchable-files query --help\n\nfor more details.\n\nYou can filter your search results. For example\n\n    ./searchable-files query \"foo\" --types-or=text,png\n\nwill submit a query which matches `\"foo\"` and requires that the files matched\nhave either the `txt` or `png` type.\n\n\n    ./searchable-files query \"foo\" --types=text,non-executable\n\nwill filter results to text files that are not executable.\n\n#### Making Unauthenticated Queries\n\nBy default, queries are submitted as the logged-in user.\nThe query command supports a `--no-auth` flag, which will submit the\nquery without any credentials.\n\nUnauthenticated queries return only results for which which the `visible_to`\nfield is set to `public`.\n\nUsing the example data, you should see a different result set between\n\n    ./searchable-files query '*' --extensions=sh\n\nand\n\n    ./searchable-files query '*' --extensions=sh --no-auth\n\n#### Dumping the Query\n\nIf you want to inspect the query which the `searchable-files` command\nis generating instead of submitting the query, you can use\n`--dump-query` to write the query to standard out, as in\n\n    ./searchable-files query \"foo\" --types=tar --dump-query\n\n### Logout\n\nWhen you are done with the demo, you can log out with\n\n    ./searchable-files logout\n\nPlease note that this will not delete your index. The index will still be available\nand searchable.\n\n## Next Steps\n\nFor a fully featured Globus Search client, you may want to install and explore the\n[`globus-search-cli`](https://globus-search-cli.readthedocs.io/en/latest/overview.html).\n\nYou can also write your own python clients against the Search service by\nusing the\n[`SearchClient` class from the Globus SDK](https://globus-sdk-python.readthedocs.io/en/stable/clients/search.html).\n\nThe full [Globus Search documentation](https://docs.globus.org/api/search/) offers a\ngreat deal more detail about the service and reference documentation for all of\nits supported methods and features.\n\n### Customizing and Extending Searchable Files\n\nThis demo application is intentionally segmented into parts which you can\ncustomize or replace to meet your needs.\n\n#### Setup\n\nThe `login` and `create-index` commands create and store data in local files.\nIf you wish to use an alternative index ID, use the `set-index` command.\n\n\u003e [!NOTE]\n\u003e Explicitly running `login` is optional. The application will automatically\n\u003e log you in if you are not logged in.\n\n#### Extractor\n\nThe Extractor examines a source for raw metadata and pulls out features which\nit recognizes. In the simplest case, just pass the `--directory` option to read\na different source directory with the existing extractor. This can be run on\nany directory without any special considerations.\n\n#### Assembler\n\nThe Assembler takes the raw data from the Extractor and annotates it with\nadditional information from a secondary source. In the provided Assembler,\nannotations are stored in some simple YAML files, but annotations could just as\neasily come from a database, external API calls, or any other source.\n\nA replacement Assembler still needs to incorporate that information with the\nExtractor's data to produce valid documents for Globus Search. Note that the\nAssembler included in the Searchable Files demo has special handling for the\nstring `\"{current_user}\"` in order to resolve this to the logged-in user's\nprimary identity ID. A custom Assembler could replicate this functionality\n(requiring login) or omit support for this usage.\n\n#### Submitter and Watcher\n\nThe Submitter and Watcher can be applied to any directory full of Globus Search\ningest documents (the format of data produced by the Assembler). Although you\nmay want to modify them to alter their outputs, combine them into a single\ncommand, or make other minor changes, their main logic should probably be\nleft unmodified.\n\nThe only special consideration when modifying these components is that these\ncommands use the Index ID retrieved from `create-index`. If modifying or\nreplacing these commands, it may be necessary to replace the logic that\nloads the `index_id` from storage.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglobus%2Fsearchable-files-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fglobus%2Fsearchable-files-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglobus%2Fsearchable-files-demo/lists"}