Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fabiospampinato/lande
A tiny neural network for natural language detection.
https://github.com/fabiospampinato/lande
detection language natural network neural tiny
Last synced: 25 days ago
JSON representation
A tiny neural network for natural language detection.
- Host: GitHub
- URL: https://github.com/fabiospampinato/lande
- Owner: fabiospampinato
- License: mit
- Created: 2022-12-12T03:06:07.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2023-09-24T16:41:09.000Z (over 1 year ago)
- Last Synced: 2024-11-28T14:48:50.209Z (29 days ago)
- Topics: detection, language, natural, network, neural, tiny
- Language: TypeScript
- Homepage:
- Size: 1.91 MB
- Stars: 68
- Watchers: 4
- Forks: 6
- Open Issues: 2
-
Metadata Files:
- Readme: readme.md
- License: license
Awesome Lists containing this project
README
# Lande
A tiny neural network for natural language detection.
This model is trained with data from the [Tatoeba](https://tatoeba.org/) project, so it works well on short inputs too.
## Model
Architecture:
- A very simple fully connected neural network with 1 hidden layer is trained, with character ngrams as input, and supported languages as output.
Details:
1. Character ngrams, at length 1, 2, 3, and 4, are extracted from the input string, with the assumption that with this data it should be possible to tell apart natural languages fairly reliably, if the input string is long enough.
2. Only a fixed number of the most frequent ngrams of supported languages are considered, which are extracted and stored during training, as they carry the most amount of signal overall, and the rest are simply discarded.
3. Compared to `CLD3` ngrams are not hashed, as I think that kinda unpredictably confuses the model, as now different ngrams can have the same hash, so unimportant ngrams could be confused for important ones. Also that requires allocating a somewhat large key space for the hashes, while instead we want to be parsimonious with that, if we only need to remember 100 1-grams that's what we should be allocating memory for, but a small key space would get quickly crowded with colliding hashes, making it somewhat useless.
4. Compared to `CLD3` no embedding layer is used here, as I think it's not needed with this approach because the model is being given clearer signals, since each input always corresponds to a specific ngram.
5. The value of each input neuron is the normalized frequency of the ngram corresponding to that neuron in the input string. Basically we tell the network if each supported ngram is present and how frequent it is.
6. To make a prediction the frequency of each supported input ngram is calculated, and a forward pass is performed.
## Languages
The following 50 languages are supported.
If you want to experiment you can tweak some internal constants and add more languages, remove languages (which necessarily improves accuracy for the remaining ones), tweak hyper-parameters for the neural network etc.
| ISO 639-3 | Name |
| --------- | ----------- |
| `afr` | Afrikaans |
| `ara` | Arabic |
| `aze` | Azerbaijani |
| `bel` | Belarusian |
| `ben` | Bengali |
| `bul` | Bulgarian |
| `cat` | Catalan |
| `ces` | Czech |
| `ckb` | Kurdish |
| `cmn` | Mandarin |
| `dan` | Danish |
| `deu` | German |
| `ell` | Greek |
| `eng` | English |
| `est` | Estonian |
| `eus` | Basque |
| `fin` | Finnish |
| `fra` | French |
| `hau` | Hausa |
| `heb` | Hebrew |
| `hin` | Hindi |
| `hrv` | Croatian |
| `hun` | Hungarian |
| `hye` | Armenian |
| `ind` | Indonesian |
| `isl` | Icelandic |
| `ita` | Italian |
| `jpn` | Japanese |
| `kat` | Georgian |
| `kaz` | Kazakh |
| `kor` | Korean |
| `lit` | Lithuanian |
| `mar` | Marathi |
| `mkd` | Macedonian |
| `nld` | Dutch |
| `nob` | Norwegian |
| `pes` | Persian |
| `pol` | Polish |
| `por` | Portuguese |
| `ron` | Romanian |
| `run` | Rundi |
| `rus` | Russian |
| `slk` | Slovak |
| `spa` | Spanish |
| `srp` | Serbian |
| `swe` | Swedish |
| `tgl` | Tagalog |
| `tur` | Turkish |
| `ukr` | Ukrainian |
| `vie` | Vietnamese |
## Install
```sh
npm install --save lande
```
## Usage
The library exports a single function which gives you a sorted list of detected languages and their probabilities.
```ts
import lande from 'lande';
lande ( 'What language is this sentence written in?' ); // => [['eng', 0.9999921321868896], ['deu', 0.000002357382982154377], ['heb', 0.000001461773877053929], ...]
lande ( 'In che lingua è scritta questa frase?' ); // => [['ita', 0.9999935626983643], ['ell', 0.0000025603442281862954], ['vie', 0.000002114558583343751], ...]
```
## License
MIT © Fabio Spampinato