{"id":13420589,"url":"https://github.com/varnamproject/libvarnam","last_synced_at":"2025-03-03T01:27:23.347Z","repository":{"id":53673283,"uuid":"2542518","full_name":"varnamproject/libvarnam","owner":"varnamproject","description":"Deprecated. See https://github.com/varnamproject/govarnam","archived":false,"fork":false,"pushed_at":"2021-05-08T13:03:22.000Z","size":10509,"stargazers_count":101,"open_issues_count":43,"forks_count":21,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-07-31T22:55:45.150Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://www.varnamproject.com","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/varnamproject.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-10-09T12:46:16.000Z","updated_at":"2024-03-24T12:18:02.000Z","dependencies_parsed_at":"2022-09-22T06:01:56.660Z","dependency_job_id":null,"html_url":"https://github.com/varnamproject/libvarnam","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varnamproject%2Flibvarnam","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varnamproject%2Flibvarnam/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varnamproject%2Flibvarnam/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varnamproject%2Flibvarnam/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/varnamproject","download_url":"https://codeload.github.com/varnamproject/libvarnam/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241593901,"owners_count":19987664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T22:01:36.888Z","updated_at":"2025-03-03T01:27:23.327Z","avatar_url":"https://github.com/varnamproject.png","language":"C","funding_links":[],"categories":["TODO scan for Android support in followings"],"sub_categories":[],"readme":"Introduction\n============\n\n`libvarnam` is a cross platform, self learning, open source library which support transliteration and reverse transliteration for Indian languages. At the core is a C shared library providing algorithms and patterns for transliteration. `libvarnam` has a simple learning module built-in which can learn words to improve the transliteration experience.\n\nInstalling libvarnam\n====================\n\n```shell\nwget http://download.savannah.gnu.org/releases/varnamproject/libvarnam/source/libvarnam-$VERSION.tar.gz\ntar -xvf libvarnam-$VERSION.tar.gz\ncd libvarnam-$VERSION\ncmake . \u0026\u0026 make\nsudo make install\n```\n\nThis will install `libvarnam` shared libraries and `varnamc` command line utility. `varnamc` can be used to quickly try out varnam.\n\n### Installation on Windows\n\nIn Windows, you can compile `libvarnam` using Visual Studio. Use the following `cmake` command to generate the project files.\n\n```shell\ncmake -DBUILD_TESTS=false -DBUILD_VST=false -DRUN_TESTS=false .\n```\n\nUsage\n=====\n\n### Transliterate\n\nUsage: varnamc -s lang_code -t word\n\n```shell\nvarnamc -s ml -t varnam\n വർണം\n വർണമേറിയത്\n```\n\n### Reverse Transliterate\n\nUsage: varnamc -s lang_code -r word\n\n```shell\nvarnamc -s ml -r വർണം\n varnam\n```\n\nWord corpus\n===========\n\n`libvarnam` is a learning system. It works better with a word corpus. You can obtain the word corpus and make varnam learn all the words. This will enable `libvarnam` to provide intelligent suggestions.\n\nHere is an example of loading Malayalam word corpus:\n\n```shell\nmkdir words\ncd words\nwget http://download.savannah.gnu.org/releases/varnamproject/words/ml/ml.tar.gz\ntar -xvf ml.tar.gz\nvarnamc  -s ml --learn-from .\n```\n\nThis will take some time depends on how much words you are loading.\n\n[Here are some more word corpus](http://mirror.rackdc.com/savannah/varnamproject/words/)\n\nThere is a `--import-learnings-from` option to import files which already has the learnt parameter. Importing these files don't take too much time as the word corpus.\n\nWhat next?\n==========\n\nIf you just wanted to use varnam for input, you have the following options\n\n-\t[Varnam on iBUS](https://github.com/varnamproject/libvarnam-ibus) - For Linux\n-\t[Varnam online editor](https://www.varnamproject.com/editor) - Platform agnostic\n\nIf you are a programmer, you will be interested in `libvarnam`. You can use it to provide Indian language support in your applications. `libvarnam` can be used from different programming languages.\n\nHow Varnam works\n================\n\n1. Scheme files and symbol tables\n2. Transliteration\n3. Learning\n\n## Scheme files and symbol tables\n\nScheme file maps English letters to phonetic equivalent indic letters. In this, all vowels, consonants and consonant clusters are mapped to the indic equivalent. Varnam uses the scheme file mapping to perform transliteration.\n\nScheme files are plain text but uses a custom DSL to make the mapping easier. This DSL is implemented using Ruby and it can contain any valid Ruby code. It also provides many helper functions to make the mapping easier.\n\n`schemes/` directory contains all the scheme files for the supported languages. Each language is represented with it's ISO language code.\n\n### Symbol tables\n\nCompiled version of Scheme file is called as *Varnam Symbol Table* (vst). This compilation is done using `varnamc` command line utility\n\n```\nvarnamc --compile schemes/ml\n```\n\nSymbol tables are binary representation of the plain text scheme files. It also contains other metadata items to make the lookup easier.\n\nlibvarnam understand only the symbol table format. Because of this, every scheme file should be compiled into *vst* format before it can be used with varnam.\n\n```\nmake vst\n```\n\ncan be used to compile all scheme files present in the *schemes* directory.\n\n#### Symbol table lookup\n\nVarnam can be initialized with just the ISO language code. When this happens, varnam will scan the following directories and tries to find a matching symbol table file. If one is found, it will be loaded and used for all operations.\n\n* \"/usr/local/share/varnam/vst\"\n* \"/usr/share/varnam/vst\"\n* \"schemes\"\n\n## Transliteration\n\n```\nvarnam_transliterate(varnam *handle, const char *input, varray **output);\n```\n\nIs the entry point for transliteration. Transliteration converts *input* to the phonetic equivalent indic text. It also provides a set of matches which are possible for the given input.\n\nTransliteration does the following steps under the hood:\n\nPerforms tokenization on the *input*. Varnam uses a greedy tokenizer which processes *input* from left to right. Tokenizer tries all possible to combinations to generate the longest possible tokens for the given input. This token will be generated by utilizing the symbol table which is provided to varnam\n\nGenerated tokens is assembled and varnam computes all possibilities of these tokens. Assume the input is *malayalam*, varnam generates tokens like, *മ, ല, യാ, ളം ([ma], [la], [ya], [lam])* and many others. Once these tokens are generated, they are combined and tested against the learning model to get rid of garbage values and come up with most used words. Words are sorted according to the frequency value and returned to the caller function.\n\n### Renderer\n\nAll of the processing is varnam is mostly language agnostic. It should work fine for all Indian languages. However, sometimes language specific fixes might be required. Varnam handles this using *Renderers*. Any language can register renderers and varnam will invoke the renderers just before rendering the final output. This can have language specific rules which can't be generalized otherwise.\n\n## Learning\n\n```\nvarnam_learn(varnam *handle, const char *word);\n```\n\nVarnam can learn new words. The more words it learns, the better it performs. Learning process learns the words and it's patterns.\n\nLearning process persists the following data:\n\n1. Patterns: All english combinations which can be used to input the given indic text\n2. Words: Indic text itself\n3. Prefixes: Prefixes of patterns and words\n\nWhen an indic word is learned, varnam tokenizes the word using the symbol table and tries to learn all possible patterns that can be used to input the word. Internally, varnam keeps a prefix tree and frequencies of all patterns. This storage structure allows varnam to retrieve matching words efficiently when a pattern is presented. Basic stemming is also performed while learning words.\n\nWhen the same word/pattern combination is learned, varnam computes frequency at which it has seen this pattern. This frequency is used to sort and pick the best candidate while performing transliteration.\n\nLearning can be initiated by calling Varnam APIs directly or using *varnamc*.\n\nInput tools like ibus-engine will automatically learn the words that you are typing.\n\nLearned data is kept in one of the following locations:\n\n* APPDATA\\varnam\\suggestions (Windows)\n* XDG_DATA_HOME/varnam/suggestions\n* HOME/.local/share/varnam/suggestions\n\n\nMozilla Public License\n======================\n\nCopyright (c) 2016 Navaneeth.K.N\n\nThis Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvarnamproject%2Flibvarnam","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvarnamproject%2Flibvarnam","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvarnamproject%2Flibvarnam/lists"}