{"id":25902333,"url":"https://github.com/dapplets/swarm-search","last_synced_at":"2025-07-17T05:39:22.083Z","repository":{"id":98800027,"uuid":"392251188","full_name":"dapplets/swarm-search","owner":"dapplets","description":"The set of modules for the Swarm Search dapplet","archived":false,"fork":false,"pushed_at":"2023-04-06T11:58:02.000Z","size":39117,"stargazers_count":2,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-13T02:05:54.493Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dapplets.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-08-03T08:49:27.000Z","updated_at":"2021-12-23T12:49:28.000Z","dependencies_parsed_at":"2025-06-13T02:16:06.113Z","dependency_job_id":null,"html_url":"https://github.com/dapplets/swarm-search","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dapplets/swarm-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapplets%2Fswarm-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapplets%2Fswarm-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapplets%2Fswarm-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapplets%2Fswarm-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dapplets","download_url":"https://codeload.github.com/dapplets/swarm-search/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapplets%2Fswarm-search/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265569457,"owners_count":23789732,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-03T03:15:53.908Z","updated_at":"2025-07-17T05:39:22.077Z","avatar_url":"https://github.com/dapplets.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg width=\"1245\" alt=\"swarm-search\" src=\"https://user-images.githubusercontent.com/43613968/230372432-8c075cee-c638-4006-a1fb-e836f9ce3687.png\"\u003e\n\n# Swarm Search\n\nThe set of modules augmenting Google and YouTube search pages with data from OpenSearch-compatible engines.\n\n## Getting Started\n\n### Google Search\n\nThe dapplet works at Google search pages. In the tab \"all\" any content type will be shown, in the tab \"video\" - only videos.\n\n[![Watch the video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-google-search-preview.png)](https://drive.google.com/file/d/1IFLuD0vqVHKlKuHBm71if-7zLKL1oaXL/view?usp=sharing)\n\n[Download video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-google-search.mp4)\n\n### YouTube Search\n\nAt the YouTube only video search results are available.\n\n[![Watch the video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-youtube-search-preview.png)](https://drive.google.com/file/d/1f0L2-0GNffUzWzMSqUA-pZr5ZeXz3xyf/view?usp=sharing)\n\n[Download video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-youtube-search.mp4)\n\n### Indexing via Swarm Gateway\n\n[**Swarm Gateway**](https://gateway.ethswarm.org/) is the website allowing any user to upload a small file to Swarm free. We've created the dapplet \"Swarm Indexer\" to augment this webpage for collecting metadata from user.\n\n[![Watch the video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-gateway-preview.png)](https://drive.google.com/file/d/1sJvzoBVoQReYJBU2Ry0PsUSaye6oMST5/view?usp=sharing)\n\n[Download video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-gateway.mp4)\n\n### Indexing via Media Downloader\n\n[**Media Downloader**](https://github.com/dapplets/media-downloader) is a dapplet created at Liberate Data Week Hackathon. We added the feature which allows you to add video to the index and make it available via Swarm Search dapplet.\n\n[![Watch the video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-media-downloader-preview.png)](https://drive.google.com/file/d/1FNeJVZ3KMPEj2WX6KgJYfzDXZUxR1scm/view?usp=sharing)\n\n[Download video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-media-downloader.mp4)\n\n### Change Search Engine\n\nTwo search engines were verified at the development:\n\nDevianArt's backend containing a huge collection of media content.\n\n```\nhttps://backend.deviantart.com/rss.xml?q={searchTerms}\u0026offset={startIndex}\u0026limit={count}\n```\n\nSwarm Search server written as a mock of non-developed yet search engine which allows to add files uploaded to Swarm via dapplets working on Swarm Gateway and Media Downloader.\n\n```\nhttps://swarm-search-server.herokuapp.com/rss?q={searchTerms}\u0026count={count}\u0026offset={startIndex}\u0026type={type?}\n```\n\nAny [OpenSearch-compatible](https://github.com/dewitt/opensearch/blob/main/opensearch-1-1-draft-6.md) search engine can be specified in the dapplet's settings. \n\nThe instruction about how to change a search engine in the dapplet's settings is in following video.\n\n[![Watch the video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-change-search-engine-preview.png)](https://drive.google.com/file/d/1QFvSjFNGrramH0qNvv82AzURAnMehBG4/view?usp=sharing)\n\n[Download video](https://github.com/dapplets/swarm-search/raw/main/docs/demo-change-search-engine.mp4)\n\n## Project Architecture\n\n![Communication diagram](https://github.com/dapplets/swarm-search/raw/main/docs/communication.png)\n\n### Actors\n\n* **Uploader** - an user uploading files to the Swarm.\n* **Searcher** - an user searching something with activated Search Dapplet.\n\n### Components\n\n* [**Indexer Dapplet**](https://github.com/dapplets/swarm-search/tree/main/packages/swarm-indexer-dapplet) - augments Swarm Gateway to collect metadata from Uploader.\n* [**Search Dapplet**](https://github.com/dapplets/swarm-search/tree/main/packages/search-dapplet) - injects search results in third party websites.\n* [**Search API**](https://github.com/dapplets/swarm-search/tree/main/packages/search-server) - a server which proxies the Elasticsearch engine and transforms data to OpenSearch compatible format.\n* **Elasticsearch** - an engine implementing full-text search.\n* [**Swarm Gateway**](https://gateway.ethswarm.org/) - a website for free files uploading to the Swarm.\n* **Bee Nodes** - Swarm network storing data in a decentralized way.\n\nBee Nodes\n\n### A: File uploading (indexing)\n\nA1: An Uploader (user) attaches a file to the Swarm Gateway and fills out the manifest form for indexing.\n\nA2: Swarm Gateway sends a file to Bee node.\n\nA3: Bee node returns a swarm reference hash.\n\nA4: The Indexer Dapplet intercepts the uploaded file and swarm reference.\n\nA5: The Indexer Dapplet sends file, reference and metadata to the Search Backend.\n\nA6: Search API retranslates the query to Elasticsearch.\n\n### B: File searching\n\nB1: Searcher opens the website and sends a query.\n\nB2: Search Dapplet intercepts the entered query from the website.\n\nB3: Search Dapplet sends [OpenSearch-compatible query](#custom-opensearch-query) to fetch search results.\n\nB4: Search API receives OpenSearch query and transforms it to ElasticSearch request.\n\nB5: Elasticsearch returns search results in JSON format.\n\nB6: Search API transforms JSON to OpenSearch's XML and returns to the dapplet.\n\nB7: Search Dapplet injects search results to the website.\n\nB8: Searcher can see external search results and open them.\n\n### Custom OpenSearch Query\n\nThe Search Dapplet uses additional `type` parameter to filter search results by content type.\n\nThis parameter is not specified by OpenSearch specification and must be implemented by a search server if you want to have content type specific search.\n\nValid value of this parameter is `video`.\n\n```\n/rss?q={searchTerms}\u0026count={count}\u0026offset={startIndex}\u0026type={type?}\n```\n\n## Development\n\n### Build Project\n\nThis project is designed as monorepo, so NPM Workspaces feature is required to install dependencies.\n\n```\nnpm install\n```\n\nTo start the development server use command:\n\n```\nnpm start\n```\n\n### Elasticsearch Installation\n\n1. Install Elasticsearch by following [this official guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html)\n\n2. Install [Ingest Attachment Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.html) which allows to search by files content.\n\n3. Create the piplene and add processors that allows searching by file content and removes unused sorces fields.\n\n```\nPUT http://localhost:9200/_ingest/pipeline/attachment\n{\n    \"description\": \"Extract attachment information\",\n    \"processors\": [\n        {\n            \"attachment\": {\n                \"field\": \"data\",\n                \"target_field\": \"attachment\"\n            }\n        },\n        {\n            \"remove\": {\n                \"field\": \"data\"\n            }\n        }\n    ]\n}\n```\n\n4. Create the index\n\n```\nPUT http://localhost:9200/fs_index\n```\n\n5. Create `/packages/search-server/.env` file with URL to the Elasticsearch HTTP API and start development!\n\nThe URL must ending at slash `/` symbol.\n\n```\nELASTICSEARCH_URL=http://localhost:9200/\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdapplets%2Fswarm-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdapplets%2Fswarm-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdapplets%2Fswarm-search/lists"}