{"id":31816197,"url":"https://github.com/openzim/javascript-libzim","last_synced_at":"2025-10-11T09:44:45.813Z","repository":{"id":46334446,"uuid":"149647459","full_name":"openzim/javascript-libzim","owner":"openzim","description":"Source and utilities for compiling libzim binaries to WASM and ASM with JavaScript wrapper","archived":false,"fork":false,"pushed_at":"2025-09-10T14:51:20.000Z","size":115119,"stargazers_count":3,"open_issues_count":14,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-10T18:44:17.035Z","etag":null,"topics":["backend","implementation","javascript","library","reader","zim"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"dattaz/libzim_wasm","license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openzim.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":"kiwix","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2018-09-20T17:42:09.000Z","updated_at":"2025-09-10T14:50:20.000Z","dependencies_parsed_at":"2023-10-12T18:14:08.907Z","dependency_job_id":null,"html_url":"https://github.com/openzim/javascript-libzim","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/openzim/javascript-libzim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fjavascript-libzim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fjavascript-libzim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fjavascript-libzim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fjavascript-libzim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openzim","download_url":"https://codeload.github.com/openzim/javascript-libzim/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openzim%2Fjavascript-libzim/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006758,"owners_count":26084184,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","implementation","javascript","library","reader","zim"],"created_at":"2025-10-11T09:44:41.501Z","updated_at":"2025-10-11T09:44:45.805Z","avatar_url":"https://github.com/openzim.png","language":"JavaScript","funding_links":["https://github.com/sponsors/kiwix"],"categories":[],"sub_categories":[],"readme":"# Prototype of libzim in WebAssembly (WASM)\n\nThis Repository provides the source code and utilities for compiling the [ZIM File](https://wiki.openzim.org/wiki/ZIM_file_format) reader\n[lbizim](https://wiki.openzim.org/wiki/Libzim) from C++ to [WebAssembly](https://developer.mozilla.org/en-US/docs/WebAssembly)\n(and [ASM.js](https://developer.mozilla.org/en-US/docs/Games/Tools/asm.js)).\n\nA prototype in HTML/JS, for testing the WASM version, is provided at https://openzim.github.io/javascript-libzim/tests/prototype/. This\nprototype uses WORKERFS as the Emscripten File System and runs in a Web Worker. The file object is mounted before run, and the name is\npassed as argument.\n\nThere is also an HTML/JS utility for testing the ability of Emscripten File Systems to read large files (muliti-gigabyte) at\nhttps://openzim.github.io/javascript-libzim/tests/test_large_file_access/.\n\n![GitHub Workflow Status (branch)](https://img.shields.io/github/actions/workflow/status/openzim/javascript-libzim/build_libzim_wasm.yml?branch=main) [![CodeFactor](https://www.codefactor.io/repository/github/openzim/javascript-libzim/badge)](https://www.codefactor.io/repository/github/openzim/javascript-libzim)\n[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n\n## Nightly and Release versions\n\nWASM and ASM versions are built nightly from the binaries provided (nightly) by [kiwix-build](https://github.com/kiwix/kiwix-build). The artefacts are\nmade available at https://download.openzim.org/nightly/ (if tests pass). Artefacts for PRs and pushes are attached to the respective workflow run. **Please note that currently, versions built form precompiled binaries lack the snippets support, because this support relies on a patch to the source code to override exceptions-based programme flow which cannot be handled well in WASM.** Therefore, to use the full functionality, it is currently necessary to compile from source using, e.g. `docker run --rm -v $(pwd):/src -u $(id -u):$(id -g) docker-emscripten-libzim:v3 make`.\n\nReleased versions are published both in [Releases](https://github.com/openzim/javascript-libzim/releases) and at https://download.openzim.org/release/javascript-libzim/.\n\nThese versions are built with both the WORKERFS and the NODEFS [Emscripten File Systems](https://emscripten.org/docs/api_reference/Filesystem-API.html).\nPlease note that WORKERFS must be run in a Web Worker, and so the JavaScript glue (interface to the C++ code) is provided as a Worker. Messages are sent\nto and received from the Worker via [`window.postMessage()`](https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage).\n\nYou can change the File Systems and other parameters in the provided [Makefile](https://github.com/openzim/javascript-libzim/blob/main/Makefile) in\nthis Repository. This recipe needs to be run in an Emscripten-configured system or a customized Emscripten container (see below).\n\n## JavaScript API Bindings\n\n\u003e **⚠️ API Stability Warning**\n\u003e \n\u003e The JavaScript API documented below is considered **unstable** until the release of version 1.0 of javascript-libzim (currently on v0.x). Breaking changes may occur between minor versions.\n\u003e \n\u003e Additionally, the built W/ASM packages in `tests/prototype/` may be ahead of official releases and may contain experimental alterations to the API that are not yet documented or finalized.\n\u003e \n\u003e **Web Worker API:** For the messaging-based Web Worker API (used in the prototype), refer to [`prejs_file_api.js`](prejs_file_api.js) which serves as the Web Worker's header and documents the available actions and message formats.\n\nThis section documents the JavaScript API bindings that are available after loading the compiled W/ASM module. The bindings provide access to libzim's core functionality including archive loading, content access, and search capabilities.\n\n### Archive Management\n\n#### `Module.loadArchive(filename: string): void`\nLoads a ZIM archive for subsequent operations.\n```javascript\nModule.loadArchive(\"path/to/archive.zim\");\n```\n\n#### `Module.getArticleCount(): number`\nReturns the total number of articles in the loaded archive.\n```javascript\nconst count = Module.getArticleCount();\n```\n\n### Content Access\n\n#### `Module.getEntryByPath(path: string): EntryWrapper | null`\nRetrieves a specific entry by its path in the ZIM archive.\n```javascript\nconst entry = Module.getEntryByPath(\"A/Wikipedia\");\nif (entry) {\n    console.log(entry.getTitle());\n}\n```\n\n### Entry Wrapper Class\n\nThe `EntryWrapper` class provides access to ZIM entries (articles, redirects, etc.):\n\n- `getPath(): string` - Returns the entry's path\n- `getTitle(): string` - Returns the entry's title\n- `isRedirect(): boolean` - Returns true if the entry is a redirect\n- `getRedirectEntry(): EntryWrapper` - Returns the target entry for redirects\n- `getItem(follow: boolean): ItemWrapper` - Returns the item content\n\n### Item Wrapper Class\n\nThe `ItemWrapper` class provides access to the actual content of entries:\n\n- `getData(): BlobWrapper` - Returns the content as binary data\n- `getMimetype(): string` - Returns the MIME type of the content\n\n### Blob Wrapper Class\n\nThe `BlobWrapper` class handles binary content:\n\n- `getContent(): Uint8Array` - Returns the content as a typed array\n\n### Search Functionality\n\n#### Basic Full-Text Search\n\n#### `Module.search(query: string, maxResults: number): vector\u003cEntryWrapper\u003e`\nPerforms basic full-text search returning entry paths.\n```javascript\nconst results = Module.search(\"quantum physics\", 20);\nfor (let i = 0; i \u003c results.size(); i++) {\n    const entry = results.get(i);\n    console.log(entry.getTitle(), entry.getPath());\n}\n```\n\n**Usage Example:** See [javascript_search_usage_example.js](javascript_search_usage_example.js) for comprehensive examples.\n\n#### Enhanced Search with Snippets\n\n#### `Module.searchWithSnippets(query: string, maxResults: number): vector\u003cSearchIteratorWrapper\u003e`\nPerforms full-text search with content snippets and metadata.\n```javascript\nconst results = Module.searchWithSnippets(\"quantum physics\", 20);\nfor (let i = 0; i \u003c results.size(); i++) {\n    const result = results.get(i);\n    console.log(result.getTitle());\n    console.log(result.getSnippet()); // Content excerpt with highlighted terms\n    console.log(\"Score:\", result.getScore());\n}\n```\n\n**Implementation Details:** See [SEARCH_SNIPPETS_IMPLEMENTATION.md](SEARCH_SNIPPETS_IMPLEMENTATION.md) for technical details about snippet generation.\n\n#### Search Iterator Wrapper Class\n\nThe `SearchIteratorWrapper` class provides rich search results with content snippets:\n\n- `getPath(): string` - Returns the entry's path\n- `getTitle(): string` - Returns the entry's title\n- `getSnippet(): string` - Returns content excerpt with search term highlighting\n- `getScore(): number` - Returns search relevance score\n- `getWordCount(): number` - Returns word count of the article\n- `getEntry(): EntryWrapper` - Returns the full entry object\n\n#### Language-Aware Search\n\n#### `Module.searchWithLanguage(query: string, maxResults: number, language?: string): vector\u003cEntryWrapper\u003e`\nPerforms search with optional language specification.\n```javascript\nconst results = Module.searchWithLanguage(\"bonjour\", 10, \"fr\");\n```\n\n### Suggestion/Autocomplete Functionality\n\n#### Simple Suggestion Function\n\n#### `Module.suggest(query: string, maxResults: number): vector\u003cEntryWrapper\u003e`\nQuick title-based suggestions for autocomplete functionality.\n```javascript\nconst suggestions = Module.suggest(\"wik\", 8);\nfor (let i = 0; i \u003c suggestions.size(); i++) {\n    const entry = suggestions.get(i);\n    console.log(entry.getTitle());\n}\n```\n\n#### Advanced Suggestion Classes\n\n#### `Module.SuggestionSearcher` Class\nAdvanced suggestion functionality with more control:\n\n```javascript\nconst searcher = new Module.SuggestionSearcher();\nconst search = searcher.suggest(\"query\");\nconst matchCount = search.getEstimatedMatches();\nconst results = search.getResults(0, 10);\n```\n\n**SuggestionSearcher Methods:**\n- `suggest(query: string): SuggestionSearchWrapper` - Creates a suggestion search\n\n**SuggestionSearchWrapper Methods:**\n- `getEstimatedMatches(): number` - Returns estimated total matches\n- `getResults(start: number, count: number): vector\u003cEntryWrapper\u003e` - Returns paginated results\n\n**Usage Example:** See [javascript_suggestions_usage_example.js](javascript_suggestions_usage_example.js) for comprehensive examples.\n\n### Vector Operations\n\nAll search and suggestion functions return Emscripten vectors with these methods:\n\n- `size(): number` - Returns the number of results\n- `get(index: number): T` - Returns the item at the specified index\n\n### Error Handling\n\nAll functions include proper error handling. Failed operations typically return:\n- `null` for single object returns (e.g., `getEntryByPath`)\n- Empty vectors for collection returns (e.g., `search`, `suggest`)\n- Empty strings for string returns (e.g., `getSnippet`)\n\n### Complete Usage Examples\n\nFor comprehensive usage examples and patterns:\n- **Search functionality:** [javascript_search_usage_example.js](javascript_search_usage_example.js)\n- **Suggestion functionality:** [javascript_suggestions_usage_example.js](javascript_suggestions_usage_example.js)\n- **Search with snippets implementation:** [SEARCH_SNIPPETS_IMPLEMENTATION.md](SEARCH_SNIPPETS_IMPLEMENTATION.md)\n\n## Steps to recompile from source with Docker\n\nThis is the easiest (and recommended) compilation method, because all required tools are configured in the Docker image. Ensure you have docker\ninstalled. (This also works in WSL with Docker Desktop installed and configured as per default to work with a WSL VM.)\n\n* Open a terminal at the root of this repository;\n* Build the Docker image with the provided Dockerfile (based on https://hub.docker.com/r/emscripten/emsdk, which is based on Debian), adapting the VERSION number of the Emscripten SDK as required:\n\n```\ndocker build -t \"docker-emscripten-libzim:v3\" ./docker --build-arg VERSION='3.1.41'\n```\n\n* Run the build with:\n\n```\ndocker run --rm -v $(pwd):/src -v /tmp/emscripten_cache/:/home/emscripten/.emscripten_cache -u $(id -u):$(id -g) -it docker-emscripten-libzim:v3 make\n```\n\nIf you get failures and wish to make adjustments, you can clean all downloaded and intermediate compiled files with the command `make clean`.\n\n## Steps to recompile manually\n\n* Install Emscripten : https://kripken.github.io/emscripten-site/docs/getting_started/downloads.html\n* Install dependencies necessary for compilation. On ubuntu 18.04, you need to activate universe repository and:\n\n```\nsudo apt install ninja-build meson pkg-config python3 autopoint libtool autoconf\nsudo apt install zlib1g-dev libicu-dev libxapian-dev liblzma-dev\n```\n\n* Activate emscripten environment variables with something like `source ./emsdk_env.sh`\n* Run `make`.\n\n## Tests\n\nBasic Unit tests are run on each automated build before publishing on the ASM and WASM builds (e.g., `libim-wasm.dev.js` and `libzim-wasm.dev.wasm`).\nThe units tested are the same as those tested in the prototype (see above) and run on two test ZIMs. The specific tests are:\n\n* mounting a test archive in each of the four libzim builds;\n* checking the reported article count;\n* loading an article;\n* searching.\n\nTests are run in Chromium browser context (needed in order to test WORKERFS) rather than purely in Node, so they are based on automation of the\nprototype, and are available in `/tests/prototype`.\n\nTo run tests manually, replace the six `libzim-[w]asm.*.*` files in `tests/prototype` with the versions you wish to test (this is done automatically\nif you build using the provided Makefile) and then run the following commands from the root of this Repository:\n\n```\nnpm install\nnpm test\n```\n\nIf you want to test certain build files you can start the server via `npx http-server --port 8080` and then visit `http://127.0.0.1:8080/tests/prototype/index.html?worker=libzim-[w]asm.*.*`.\n\nTo run tests in a different browser, copy and adapt the test runner `chromium.e2e.runner.js`. Run it manually like so:\n\n`npx start-server-and-test 'http-server --silent' 8080 'npx mocha ./tests/prototype/chromium.e2e.runner.js'`\n\n## Licence\n\n[GPLv3](https://www.gnu.org/licenses/gpl-3.0) or later, see\n[LICENCE](LICENSE) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenzim%2Fjavascript-libzim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenzim%2Fjavascript-libzim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenzim%2Fjavascript-libzim/lists"}