{"id":13579557,"url":"https://github.com/mwmbl/mwmbl","last_synced_at":"2025-05-14T13:06:13.029Z","repository":{"id":37247409,"uuid":"441849572","full_name":"mwmbl/mwmbl","owner":"mwmbl","description":"An open source, non-profit web search engine","archived":false,"fork":false,"pushed_at":"2025-05-01T11:56:02.000Z","size":193011,"stargazers_count":1614,"open_issues_count":39,"forks_count":74,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-05-06T22:32:41.118Z","etag":null,"topics":["non-profit","search-engine"],"latest_commit_sha":null,"homepage":"https://mwmbl.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mwmbl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-12-26T09:03:27.000Z","updated_at":"2025-05-06T19:13:51.000Z","dependencies_parsed_at":"2024-04-16T18:23:51.524Z","dependency_job_id":"d9a0e904-e7a6-4518-be54-cde793bb4b2f","html_url":"https://github.com/mwmbl/mwmbl","commit_stats":{"total_commits":719,"total_committers":13,"mean_commits":55.30769230769231,"dds":0.04311543810848406,"last_synced_commit":"1191c8e0a250a1ad0bba05011a8b5d932e3c1f14"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwmbl%2Fmwmbl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwmbl%2Fmwmbl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwmbl%2Fmwmbl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwmbl%2Fmwmbl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mwmbl","download_url":"https://codeload.github.com/mwmbl/mwmbl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253536691,"owners_count":21923879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["non-profit","search-engine"],"created_at":"2024-08-01T15:01:40.522Z","updated_at":"2025-05-14T13:06:12.996Z","avatar_url":"https://github.com/mwmbl.png","language":"Python","funding_links":["https://opencollective.com/mwmbl"],"categories":["Python"],"sub_categories":[],"readme":"![banner](docs/assets/images/banner_mwmbl.svg)\n\n# Mwmbl - the Open Source Web Search Engine\n\n[![Matrix](https://img.shields.io/matrix/mwmbl:matrix.org)](https://matrix.to/#/#mwmbl:matrix.org)\n\n**No ads, no tracking, no profit**\n\n[Mwmbl](https://mwmbl.org) is a non-profit, open source search engine\nwhere the community determines the rankings. We aim to be a\nreplacement for commercial search engines such as Google and\nBing.\n\n![mwmbl](https://user-images.githubusercontent.com/1283077/218265959-be4220b4-dcf0-47ab-acd3-f06df0883b52.gif)\n\nWe have our own index powered by our community. Our index is currently\nmuch smaller than those of commercial search engines, with around 500\nmillion unique URLs ([more stats](https://mwmbl.org/stats/)). The\nquality is a long way's off from matching the commercial engines at the moment, but\nyou can help change that by joining us! We aim to have 1 billion\nunique URLs indexed by the end of 2024, 10 billion by the end of 2025\nand 100 billion by the end of 2026 by which point we should be\ncomparable with the commercial search engines.\n\n\nCommunity\n=========\n\nOur main community is on\n[Matrix](https://matrix.to/#/#mwmbl:matrix.org) but we also have a\n[Discord server](https://discord.gg/2BGSUYFdkD) for non-development\nrelated discussion.\n\nThe community is responsible for crawling the web (see below) and\n[curating search results](https://book.mwmbl.org/page/curating/). We are\nfriendly and welcoming. Join us!\n\n\nDocumentation\n=============\n\nAll documentation is at [https://book.mwmbl.org](https://book.mwmbl.org).\n\n\nCrawling\n========\n\nCrawling is distributed across the community, while indexing is\ncentralised on the main server.\n\nIf you have spare computer power and bandwidth, the best way you can help is\nby running our [command line\ncrawler](https://github.com/mwmbl/crawler-script) with as many threads\nas you can spare.\n\nIf you have Firefox you can help out by [installing our\nextension](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/). This\nwill crawl the web in the background. It does not use or access any of\nyour personal data. Instead it crawls a set of URLs sent from our\ncentral server. After extracting a summary of each page, it batches\nthese up and sends the data to the central server to be stored and\nindexed.\n\nWhy a non-profit search engine?\n===============================\n\nThe motives of ad-funded search engines are at odds with providing an\noptimal user experience. These sites are optimised for ad revenue,\nwith user experience taking second place. This means that pages are\nloaded with ads which are often not clearly distinguished from search\nresults. Also, eitland on Hacker News\n[comments](https://news.ycombinator.com/item?id=29427442):\n\n\u003e Thinking about it it seems logical that for a search engine that\n\u003e practically speaking has monopoly both on users and as mattgb points\n\u003e out - [to some] degree also on indexing - serving the correct answer\n\u003e first is just dumb: if they can keep me going between their search\n\u003e results and tech blogs with their ads embedded one, two or five\n\u003e times extra that means one, two or five times more ad impressions.\n\nBut what about...?\n==================\n\nThe space of alternative search engines has expanded rapidly in recent\nyears. Here's a very incomplete list of some that have interested me:\n\n - [search.marginalia.nu](https://search.marginalia.nu/) - a search\n   engine favouring text-heavy websites\n - [SearXNG](https://github.com/searxng/searxng) - an open source meta\n   search engine\n - [YaCy](https://yacy.net/) - an open source distributed search engine\n - [Stract](https://stract.com/) - an open source, private search engine with a focus on privacy and customizability\n - [Brave](https://search.brave.com/)\n - [DuckDuckGo](https://duckduckgo.com/)\n - [Kagi](https://kagi.com/)\n\nOf these, YaCy is the closest in spirit to the idea of a non-profit\nsearch engine. The index is distributed across a peer-to-peer\nnetwork. Unfortunately this design decision slows the fetching of search results.\n\nMarginalia Search is fantastic, but our goals are different: we aim to\nbe a replacement for commercial search engines whereas Marginalia aims to\nprovide a different type of search.\n\nAll other search engines that I've come across are for-profit. Please\nlet me know if I've missed one!\n\nDesigning for non-profit\n========================\n\nTo be a good search engine, we need to store many items, but the cost\nof running the engine is at least proportional to the number of items\nstored. Our main consideration is thus to reduce the cost per item\nstored.\n\nThe design is founded on the observation that most items rank for a\nsmall set of terms. In the extreme version of this, where each item\nranks for a single term, the usual inverted index design is grossly\ninefficient, since we have to store each term at least twice: once in\nthe index and once in the item data itself.\n\nOur design is a giant hash map. We have a single store consisting of a\nfixed number N of pages. Each page is of a fixed size (currently 4096\nbytes to match a page of memory), and consists of a compressed list of\nitems. Given a term for which we want an item to rank, we compute a\nhash of the term, a value between 0 and N - 1. The item is then stored\nin the corresponding page.\n\nTo retrieve pages, we simply compute the hash of the terms in the user\nquery and load the corresponding pages, filter the items to those\ncontaining the term and rank the items. Since each page is small, this\ncan be done very quickly.\n\nBecause we compress the list of items, we can rank for more than a\nsingle term and maintain an index smaller than the inverted index\ndesign. At least, that's the theory. This idea has yet to be tested on\na large scale.\n\nHow to contribute\n=================\n\nThere are multiple ways to help:\n - [Help us crawl the\n   web](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/)\n - [Donate some money](https://opencollective.com/mwmbl) towards\n   hosting costs and supporting our volunteers\n - Give feedback/suggestions\n - Assist in development of the engine itself\n\nIf you would like to help in any of these or other ways, thank you!\nPlease join our [Matrix chat\nserver](https://matrix.to/#/#mwmbl:matrix.org) or email the main\nauthor (email address is in the git commit history).\n\nDevelopment\n===========\n\n### Local Testing\n\nFor trying out the service locally see the section in the Mwmbl [book](https://book.mwmbl.org/page/developers/).\n\n### Using Dokku\n\nNote: this method is not recommended as it is more involved, and your index will not include any data unless you \nset up a crawler to crawl to your server. You will need to set up your own Backblaze or S3 equivalent storage, or \nhave access to the production keys, which we probably won't give you.\n\nFollow the [deployment instructions](https://github.com/mwmbl/mwmbl/wiki/Deployment)\n\n\nFrequently Asked Question\n=========================\n\n### How do you pronounce \"mwmbl\"?\n\nLike \"mumble\". I live in\n[Mumbles](https://en.wikipedia.org/wiki/Mumbles), which is spelt\n\"Mwmbwls\" in Welsh. But the intended meaning is \"to mumble\", as in\n\"don't search, just mwmbl!\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmwmbl%2Fmwmbl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmwmbl%2Fmwmbl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmwmbl%2Fmwmbl/lists"}