{"id":26284585,"url":"https://github.com/true-real-michael/tg-message-search","last_synced_at":"2025-05-07T12:25:38.221Z","repository":{"id":278804898,"uuid":"932464022","full_name":"true-real-michael/tg-message-search","owner":"true-real-michael","description":"Search your telegram chat for message threads all in your browser (+ russian lemmatization) :shipit:","archived":false,"fork":false,"pushed_at":"2025-03-19T12:18:09.000Z","size":496,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-31T10:04:28.268Z","etag":null,"topics":["lemmatization","search","telegram","wasm"],"latest_commit_sha":null,"homepage":"https://true-real-michael.github.io/tg-message-search/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/true-real-michael.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-14T00:29:32.000Z","updated_at":"2025-03-19T12:18:13.000Z","dependencies_parsed_at":"2025-03-13T14:24:32.453Z","dependency_job_id":null,"html_url":"https://github.com/true-real-michael/tg-message-search","commit_stats":null,"previous_names":["true-real-michael/wasm-chat-searcher","true-real-michael/tg-message-search"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-real-michael%2Ftg-message-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-real-michael%2Ftg-message-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-real-michael%2Ftg-message-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-real-michael%2Ftg-message-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/true-real-michael","download_url":"https://codeload.github.com/true-real-michael/tg-message-search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252875251,"owners_count":21817990,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lemmatization","search","telegram","wasm"],"created_at":"2025-03-14T18:36:10.757Z","updated_at":"2025-05-07T12:25:38.189Z","avatar_url":"https://github.com/true-real-michael.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Telegram Message Search\n## Usage\n\nThe app is currenty hosted here: [https://true-real-michael.github.io/tg-message-search](https://true-real-michael.github.io/tg-message-search)\n\n1. [Export](https://telegram.org/blog/export-and-more) a telegram **chat** in a JSON format\n2. Upload the `result.json` file (the website runs in the browser and no data is saved)\n3. Search and browse threads and messages\n\n---\n\n1. [Экспортируйте](https://telegram.org/blog/export-and-more) телеграм-чат в формате JSON\n2. Загрузите файл `result.json` (сайт работает полностью в браузере, и никакие данные не сохраняются)\n3. Производите поиск по тредам и сообщениям\n\n\nUI looks like this:\n![ui example](assets/example.jpg)\n\n![wasm offer](assets/wasm-offer.jpg)\n\n\n## Building\n\n1. Install Trunk, add wasm32 target\n```sh\ncargo install trunk\nrustup target add wasm32-unknown-unknown\n```\n2. Download the `lemmatization-ru.tsv.gz` file from releases and place it under the `/data` directory. Alternatively, download the morphological dictionary from [OpenCorpora's website](https://opencorpora.org/?page=downloads), place it under `/data`, run the `scripts/preprocess_opcorpora.py` script, and gzip the result\n3. Run the project\n```sh\ntrunk serve --port 3000 --release\n```\n4. The project will be available at localhost:3000/tg-message-search\n\nIf you want to use this project for a different language, you should replace the lemmatization dictionary with the one for your language.\nIf you want more complex lemmatization/stemming/embedding logic, you should take a look at the `Lemmatizer` struct in `src/analysis/lemmatizer.rs` and modify it accordingly.\n\n## Why?\n\n- The Telegram native message search was not convenient for me, especially:\n  - When I wanted to search for synonyms.\n  - When I wanted to search for combinations of words.\n  - When the info is scattered across multiple messages, which form a reply chain.\n  - When there are many results, it is inconvenient to scroll through them in a tiny search results bar.\n\n## Design choices\n\n- Why WASM?\n  - To maintain privacy by keeping all data client-side.\n  - To avoid round-trips for queries and data upload.\n  - I didn't want to spend money on a backend.\n  - Because it is a cool technology and I wanted to try it out.\n- Why no embedded db?\n  - Because I wanted bespoke lemmatization logic.\n  - I also wanted to keep the app lightweight and minimalistic.\n- Why Leptos?\n  - No reason at all, just wanted to try it out.\n  - This project used to use `wasm-bindgen` + vanilla JS + HTML, but I tried doing reactive UI with Leptos and it worked well.\n  - Language unification was a nice bonus.\n- Why dictionary-based lemmatization?\n  - I initially considered using word embeddings, but I could not find a suitable model for Russian.\n  - Dictionary gets the work done and does not take too much space (arguably): ≈9MB compressed, ≈300MB uncompressed.\n\n\n## What I learned\n\n- How to use WASM in a web application.\n- How to use Leptos for building a reactive UI.\n- Refreshed memories on parsing.\n\n## Potential improvements\n\n- Web Workers for background initialization. Currently it is blocking the main thread.\n- Revise the code because it contains a lot of clones and unwraps.\n\n## License\n\nAll the code is licensed under MIT License\n\nThe file `lemmatization-ru.tsv.gz` in this repository's GitHub releases is a derivative of [OpenCorpora](https://opencorpora.org/?page=downloads)'s Russian language morphologic dictionary and is licenced under [Creative Commons Attribution-ShareAlike 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en)\n\n---\n\nВесь код находится под лицензией MIT\n\nФайл `lemmatization-ru.tsv.gz` в GitHub-релизах этого репозитория является производной от морфологического словаря [OpenCorpora](https://opencorpora.org/?page=downloads) и находится под лицензией [Creative Commons Attribution-ShareAlike 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.ru)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrue-real-michael%2Ftg-message-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrue-real-michael%2Ftg-message-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrue-real-michael%2Ftg-message-search/lists"}