{"id":13422909,"url":"https://github.com/ipfs-search/ipfs-search","last_synced_at":"2025-04-09T08:06:53.290Z","repository":{"id":12717558,"uuid":"65658906","full_name":"ipfs-search/ipfs-search","owner":"ipfs-search","description":"Search engine for the Interplanetary Filesystem.","archived":false,"fork":false,"pushed_at":"2023-10-27T07:11:40.000Z","size":12572,"stargazers_count":875,"open_issues_count":43,"forks_count":109,"subscribers_count":41,"default_branch":"master","last_synced_at":"2025-04-02T06:43:25.995Z","etag":null,"topics":["elasticsearch","golang","ipfs","ipfs-search","rabbitmq","search-engine"],"latest_commit_sha":null,"homepage":"http://ipfs-search.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ipfs-search.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null},"funding":{"github":"dokterbob","open_collective":"ipfs-search","custom":["https://commerce.coinbase.com/checkout/f6b4fcad-3f09-41ac-8520-81aba321d4d7"]}},"created_at":"2016-08-14T09:21:02.000Z","updated_at":"2025-03-12T01:48:23.000Z","dependencies_parsed_at":"2023-12-20T14:35:06.408Z","dependency_job_id":"4ff41717-71dd-4dda-a2e5-b697aab6d2c2","html_url":"https://github.com/ipfs-search/ipfs-search","commit_stats":{"total_commits":982,"total_committers":11,"mean_commits":89.27272727272727,"dds":"0.031568228105906315","last_synced_commit":"ada256fd70142cd0fac500d631a21ca65c07859a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs-search%2Fipfs-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs-search%2Fipfs-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs-search%2Fipfs-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs-search%2Fipfs-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ipfs-search","download_url":"https://codeload.github.com/ipfs-search/ipfs-search/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247675614,"owners_count":20977379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","golang","ipfs","ipfs-search","rabbitmq","search-engine"],"created_at":"2024-07-30T23:00:59.708Z","updated_at":"2025-04-09T08:06:53.268Z","avatar_url":"https://github.com/ipfs-search.png","language":"Go","funding_links":["https://github.com/sponsors/dokterbob","https://opencollective.com/ipfs-search","https://commerce.coinbase.com/checkout/f6b4fcad-3f09-41ac-8520-81aba321d4d7"],"categories":["Apps","Go"],"sub_categories":[],"readme":"# [ipfs-search.com](https://ipfs-search.com)\n[![pipeline status](https://gitlab.com/ipfs-search.com/ipfs-search/badges/master/pipeline.svg)](https://gitlab.com/ipfs-search.com/ipfs-search/-/commits/master)\n[![Maintainability](https://api.codeclimate.com/v1/badges/1c25261992991d72137c/maintainability)](https://codeclimate.com/github/ipfs-search/ipfs-search/maintainability)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/1c25261992991d72137c/test_coverage)](https://codeclimate.com/github/ipfs-search/ipfs-search/test_coverage)\n[![Documentation Status](https://readthedocs.org/projects/ipfs-search/badge/?version=latest)](https://ipfs-search.readthedocs.io/en/latest/?badge=latest)\n[![Go Reference](https://pkg.go.dev/badge/github.com/ipfs-search/ipfs-search.svg)](https://pkg.go.dev/github.com/ipfs-search/ipfs-search)\n[![Backers on Open Collective](https://opencollective.com/ipfs-search/backers/badge.svg)](#backers)\n[![Sponsors on Open Collective](https://opencollective.com/ipfs-search/sponsors/badge.svg)](#sponsors)\n\nSearch engine for the [Interplanetary Filesystem](https://ipfs.io). Sniffs the DHT gossip and indexes file and directory hashes.\n\nMetadata and contents are extracted using [ipfs-tika](https://github.com/ipfs-search/ipfs-tika), searching is done using OpenSearch, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.\n\nThe ipfs-search command consists of two components: the crawler and the sniffer. The sniffer extracts hashes from the gossip between nodes. The crawler extracts data from the hashes and indexes them.\n\n## Docs\nDocumentation is hosted on [Read the Docs](https://ipfs-search.readthedocs.io/en/latest/), based on files contained in the [docs](https://github.com/ipfs-search/ipfs-search/tree/master/docs) folder. In addition, there's extensive [Go docs](https://pkg.go.dev/github.com/ipfs-search/ipfs-search) for the internal API as well as [SwaggerHub OpenAPI documentation](https://app.swaggerhub.com/apis-docs/ipfs-search/ipfs-search/) for the REST API.\n\n## Contact\nPlease find us on our Freenode/[Riot/Matrix](https://riot.im/app/#/room/#ipfs-search:matrix.org) channel [#ipfs-search:matrix.org](https://matrix.to/#/#ipfs-search:matrix.org).\n\n## Snapshots\nipfs-search provides the daily snapshot for all of the indexed data using\n[snapshots](https://opensearch.org/docs/latest/opensearch/rest-api/snapshots/index/).\nTo learn more about downloading and restoring snapshots please refer to the [relevant section](https://ipfs-search.readthedocs.io/en/latest/snapshots.html) in our documentation.\n\n## Related repo's\n* [frontend](https://github.com/ipfs-search/ipfs-search-frontend)\n* [search API](https://github.com/ipfs-search/ipfs-search-api)\n* [deployment](https://github.com/ipfs-search/ipfs-search-deployment)\n* [nsfw-server](https://github.com/ipfs-search/nsfw-server)\n* [ipfs-tika](https://github.com/ipfs-search/ipfs-tika)\n\n## Contributors wanted\nBuilding a search engine like this takes a considerable amount of resources (money _and_ TLC).\nIf you are able to help out with either of them, do reach out (see the contact section in this file).\n\nPlease read the Contributing.md file before contributing.\n\n## Roadmap\nFor discussing and suggesting features, look at the [issues](https://github.com/ipfs-search/ipfs-search/issues).\n\n## External dependencies\n\n* Go 1.19\n* OpenSearch 2.3.x\n* RabbitMQ / AMQP server\n* NodeJS 9.x\n* IPFS 0.7\n* Redis\n\n## Internal dependencies\n\n* [nsfw-server](https://github.com/ipfs-search/nsfw-server)\n* [ipfs-tika](https://github.com/ipfs-search/ipfs-tika)\n\n## Building\n```bash\n$ go get ./...\n$ make\n```\n\n## Running\n\n### Docker\nThe most convenient way to run the crawler is through Docker. Simply run:\n\n```bash\ndocker-compose up\n```\n\nThis will start the crawler, the sniffer and all its dependencies. Hashes can also be queued for crawling manually by running `ipfs-search a \u003chash\u003e` from within the running container. For example:\n\n```bash\ndocker-compose exec ipfs-crawler ipfs-search add QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv\n```\n\n### Ansible deployment\nAutomated deployment can be done on any (virtual) Ubuntu 16.04 machine. The full production stack is automated and can be found in it's own [repository](https://github.com/ipfs-search/ipfs-search-deployment).\n\n## Contributors\n\nThis project exists thanks to all the people who contribute.\n\u003ca href=\"https://github.com/ipfs-search/ipfs-search/graphs/contributors\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/contributors.svg?width=890\u0026button=false\" /\u003e\u003c/a\u003e\n\n\n## Backers\n\nThank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/ipfs-search#backer)]\n\n\u003ca href=\"https://opencollective.com/ipfs-search#backers\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/backers.svg?width=890\"\u003e\u003c/a\u003e\n\n\n## Sponsors\n\n\u003ca href=\"https://nlnet.nl/project/IPFS-search/\"\u003e\u003cimg width=\"200pt\" src=\"https://nlnet.nl/logo/banner.png\"\u003e\u003c/a\u003e \u003ca href=\"https://nlnet.nl/project/IPFS-search/\"\u003e\u003cimg width=\"200pt\" src=\"https://nlnet.nl/image/logos/NGI0_tag.png\"\u003e\u003c/a\u003e\n\u003cbr\u003e\nipfs-search is supported by NLNet through the EU's Next Generation Internet (NGI0) programme.\n\n\u003ca href=\"https://redpencil.io/projects/\"\u003e\u003cimg width=\"270pt\" src=\"https://raw.githubusercontent.com/redpencilio/frontend-redpencil.io/327318b84ffb396d8af6776f19b9f36212596082/public/assets/vector/rpio-logo.svg\"\u003e \u003c/a\u003e\u003cbr\u003e\nRedPencil is supporting the hosting of ipfs-search.com.\n\nSupport this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/ipfs-search#sponsor)]\n\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/0/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/0/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/1/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/1/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/2/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/2/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/3/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/3/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/4/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/4/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/5/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/5/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/6/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/6/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/7/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/7/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/8/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/8/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/ipfs-search/sponsor/9/website\" target=\"_blank\"\u003e\u003cimg src=\"https://opencollective.com/ipfs-search/sponsor/9/avatar.svg\"\u003e\u003c/a\u003e\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fipfs-search%2Fipfs-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fipfs-search%2Fipfs-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fipfs-search%2Fipfs-search/lists"}