{"id":16607129,"url":"https://github.com/kbumsik/blogsearch","last_synced_at":"2025-09-01T01:04:42.704Z","repository":{"id":41660632,"uuid":"228577626","full_name":"kbumsik/blogsearch","owner":"kbumsik","description":"A pure client-side full-text search engine for static websites, using WebAssembly","archived":false,"fork":false,"pushed_at":"2023-01-06T03:55:58.000Z","size":29118,"stargazers_count":12,"open_issues_count":25,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-27T14:32:20.828Z","etag":null,"topics":["full-text-search","search-engine","sqlite","static-website","webassembly"],"latest_commit_sha":null,"homepage":"https://kbumsik.io/blogsearch/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kbumsik.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-17T09:13:03.000Z","updated_at":"2024-10-18T08:28:58.000Z","dependencies_parsed_at":"2023-02-05T06:31:35.661Z","dependency_job_id":null,"html_url":"https://github.com/kbumsik/blogsearch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kbumsik%2Fblogsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kbumsik%2Fblogsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kbumsik%2Fblogsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kbumsik%2Fblogsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kbumsik","download_url":"https://codeload.github.com/kbumsik/blogsearch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243830949,"owners_count":20354854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["full-text-search","search-engine","sqlite","static-website","webassembly"],"created_at":"2024-10-12T01:11:44.106Z","updated_at":"2025-03-16T21:30:59.036Z","avatar_url":"https://github.com/kbumsik.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"= BlogSearch\n\n// Asciidoc references\n// Documentation: https://asciidoctor.org/docs/user-manual/\n// Quick reference: https://asciidoctor.org/docs/asciidoc-syntax-quick-reference/\n// Asciidoc vs Markdown: https://asciidoctor.org/docs/user-manual/#comparison-by-example\n// GitHub Flavored Asciidoc (GFA): https://gist.github.com/dcode/0cfbf2699a1fe9b46ff04c41721dda74\n\n:project-version: 0.0.3\n:rootdir: https://github.com/kbumsik/blogsearch\n\nifdef::env-github[]\n// Emoji\n:tip-caption: :bulb:\n:note-caption: :information_source:\n:important-caption: :heavy_exclamation_mark:\n:caution-caption: :fire:\n:warning-caption: :warning:\n// URL\n:imagesdir: https://raw.githubusercontent.com/kbumsik/blogsearch/master\nendif::[]\n\nimage::https://github.com/kbumsik/blogsearch/workflows/Build%20and%20Test%20CI/badge.svg[link=\"https://github.com/kbumsik/blogsearch/actions?query=workflow%3A%22Build+and+Test+CI%22\"]\n\nimage::docs/demo.gif[link=https://kbumsik.io/blogsearch]\nhttps://kbumsik.io/blogsearch[Try it!]\n\n[.lead]\nBlogSearch is a blogging tool that enables a search engine without any external\nservices.\n\nThis is like https://docsearch.algolia.com/[DocSearch] but for blogs.\n\nMore technically, BlogSearch is a pure client-side, full-text search engine for static websites, powered by SQLite compiled to WebAssembly.\n\n.Features:\n* Purely client-side search\n* No server to maintain. No service cost.\n* Easy. It's built for blogs and static websites in mind.\n* Supports popular blog frameworks:\n** https://jekyllrb.com/[Jekyll]\n** https://www.gatsbyjs.org/[Gatsby]\n** https://gohugo.io/[Hugo]\n** ...and any static websites!\n\n.Sister project:\n* https://github.com/kbumsik/sqlite-wasm[sqlite-wasm]: Run SQLite on the web, using WebAssembly. This project is made for blogsearch's needs.\n\n== Concepts\n\n=== Workflow overview\n\n[cols=\"50%,50%\", options=\"noheader\"]\n|===\n2+| The workflow is consist of two steps: **1.** You build an index file `.db.wasm`, then copy it to the public directory. **2.** The engine in the webpage will read the index file and enables the search.\n\n| 1. Build an index file | 2. Enable the search\n\na|\nThe index file `.db.wasm` is a small database file that contains the contents of your website. You can use easy-to-use index building tools: \n\n* Jekyll (link:./jekyll-blogsearch[jekyll-blogsearch])\n* Gatsby (link:./gatsby-plugin-blogsearch[gatsby-plugin-blogsearch])\n* Hugo (link:./blogsearch-crawler[blogsearch-crawler])\n* Generic crawler (link:./blogsearch-crawler[blogsearch-crawler])\n\nThen you copy the generated `.db.wasm` to the public directory (where index.html located) of website.\n\na|\nYour webpage should load the blogsearch engine. There is only one engine available:\n\n* BlogSearch Engine (link:./blogsearch[blogsearch])\n\nLoad the engine using \u003cscript\u003e tag or in JavaScript file.\nOnce the engine fetch the `.db.wasm` file correctly, now you have a fully working searchable webpage! \n\n|===\n\nNOTE: Throughout the project, the terms \"index\" and \"database\" are often mixed, but they mean same SQLite `.db.wasm` file in the most of the case.\n\n[#building-index]\n=== 1. Building a search index file\n==== Installing an index building tool\n\n* Jekyll (link:./jekyll-blogsearch[jekyll-blogsearch])\n* Gatsby (link:./gatsby-plugin-blogsearch[gatsby-plugin-blogsearch])\n* Hugo (link:./blogsearch-crawler[blogsearch-crawler])\n* Generic crawler (link:./blogsearch-crawler[blogsearch-crawler])\n\n[#whats-in-the-index]\n==== What's in the index file\nUsers should configure an index building tool to collect the value of fields\nin order to work the search engine properly.\n\nThe index building tool should collect the following default fields for each posts:\n\nfields::\n* `title`: The title of the post.\n* `body`: The content of the post.\n* `url`: The URL link to the post.\n* `categories`: A comma-separated (`,`) list of categories that the post belongs to.\n* `tags`: A comma-separated (`,`) list of tags that the post has.\n\nUsers can configure every fields using the following properties:\n\n[cols=\"50%,50%\", options=\"header,unbreakable\", stripes=even]\n.Common options for the field\n|===\n\n| Example | Result\n\n2+| `disabled`: If set `true`, completely disable the field.\n\n2+a|\n[source,diff]\n----\n{\n  ...other field options...\n  categories: {\n+    disabled: true,\n  },\n}\n----\n\n\n\n\n2+| `hasContent`: If set `false`, the index building tool won't store the value of the field, but still indexes its value. This can be used to reduce the size of a generated index file by the tool. This is useful especially when the size of `body` field contents is big.\n\nIn the following example, the size of the index file `.db.wasm` is decreased.\n\n2+a| \n[source,diff]\n----\n{\n  ...other field options...\n  body: {\n+    hasContent: false,\n  },\n}\n----\n\n2+| `indexed`: If set `false`, disable indexing for the field. Its value will still appears in the search result. It is especially useful for `url` field, whose value is not meaningful for search.\n\n2+a|\n[source,diff]\n----\n{\n  ...other field options...\n  url: {\n+    indexed: false,\n  },\n}\n----\n\n\n|===\n\nNOTE: Your index building tool may has tool-specific options for the field (e.g. `parser` option for link:./blogsearch-crawler[blogsearch-crawler]). See the documentation of your index building tool for details.\n\n=== 2. Enabling the search engine on the web\n\n.It's as simple as:\n[source,html,options=\"nowrap\",subs=\"verbatim,attributes\"]\n----\n\u003clink rel=\"stylesheet\" href=\"https://cdn.jsdelivr.net/npm/blogsearch@{project-version}/dist/basic.css\" /\u003e\n\n\u003cscript src=\"https://cdn.jsdelivr.net/npm/blogsearch@{project-version}/dist/blogsearch.umd.js\"\u003e\u003c/script\u003e\n\u003cscript src=\"https://cdn.jsdelivr.net/npm/blogsearch@{project-version}/dist/worker.umd.js\"\u003e\u003c/script\u003e\n\n\u003cinput id=\"blogsearch_input_element\" type=\"search\" placeholder=\"Search Text\" class=\"form-control\" /\u003e\n\n\u003cscript\u003e\n  blogsearch({\n    dbPath: 'your_index_file.db.wasm',\n    inputSelector: '#blogsearch_input_element',\n  });\n\u003c/script\u003e\n----\n\nFor the further details and options, go to link:./blogsearch[the subdirectory of blogsearch].\n\n== QnA\n=== Which search engine technology used in this project?\nThe search engine basically is SQLite with https://www.sqlite.org/fts5.html[the FTS5 extension], compiled to WebAssembly. The SQLite FTS5 offers https://www.sqlite.org/fts5.html#the_bm25_function[the built-in BM25 ranking algorithm] for the search functionality. As SQLite is the most portable database engine, you can open any SQLite database files on the web too! Thanks to SQLite, we can easily write plugins for BlogSearch with link:./blogsearch-crawler/database.ts[just a few SQL queries] in different programming languages.\n\n=== Why `.db.wasm` is recommended file extension index? It's not a WebAssembly binary file. Why not just `.db`?\nI tried to make it `.db` but there is a big problem: the index file is not\ngzip-compressed by the web server.\nPopular blog web services (especially GitHub Pages) usually serve a `.db` file as\n`application/octet-stream` and do not compress the file. By lying that it is\na WebAssembly binary file `.wasm`, the servers recognize it as `application/wasm` and ship it compressed.\n\nCompression is important because it significantly reduces the file size.\nI saw the size is reduced up to 1/3.\n\n== Building from source\n\n=== Workflow\n\nTo avoid \"`But it works on my machine`\" problem, it is strongly recommended to use Docker for building tasks.\n\nAlthough this repository is a monorepo where each subprojects has own build scripts, you can easily run tasks in the root directory. \n\nTIP: If you want to build a specific subproject only, go to the subdirectory and run https://classic.yarnpkg.com/en/docs/usage[yarn] commands.\n\nThe required tools are the following:\n\n* GNU Make (https://stackoverflow.com/a/48873146/4661625[v4.2 or higher is recommended], be warned for macOS users!)\n* docker\n* docker-compose\n* yarn\n\nAlthough it is a JS project Makefile is used because it is much more configuratble and supports building in parallel.\n\nFor specific NodeJS versions used in the project, please look at link:./Dockerfile[the Dockerfile].\n\n==== Prepare\n[source,bash]\n----\n# Or yarn install, without docker\nmake install-in-docker\n----\n\n==== Build libraries\n[source,bash]\n----\n# Or yarn install, without docker\nmake lib-in-docker\n----\n\n==== Run a demo server\n[source,bash]\n----\nmake start-in-docker\n\n# You can access the demo page via 0.0.0.0:9000\n----\n\n==== Testing\n[source,bash]\n----\n# Or make test, without docker\nmake test-in-docker\n\n# Run it in parallel\nmake test-in-docker -j4 --output-sync=target\n----\n\n==== Rebuild example index files\n\nWARNING: This will take a lot of time! (~30 mintues)\n\n[source,bash]\n----\n# It is highly recommended to use docker here\nmake examples-in-docker \u0026\u0026 make demo-in-docker\n----\n\n==== Build everything\n\nWARNING: This will take a lot of time! (~30 mintues)\n\n[source,bash]\n----\n# Or make all, without docker\nmake all-in-docker\n\n# Or\n\n# Parallel builds. This reduces the build time almost an half on my machine. \nmake all-in-docker -j4 --output-sync=target\n----\n\n==== Rebuild everything\n[source,bash]\n----\nmake clean\n\n# Then run any commands above\n----\n\n==== Get into a bash session in the container\n[source,bash]\n----\nmake bash-in-docker\n----\n\n== Credits \u0026 License\n\nThis project is inspired by https://docsearch.algolia.com/[DocSearch] and has\na reimplementation of it in TypeScript.\n\nOther than that, the project is MIT License. See link:./LICENSE[LICENSE]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkbumsik%2Fblogsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkbumsik%2Fblogsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkbumsik%2Fblogsearch/lists"}