Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pemistahl/lingua-rs
The most accurate natural language detection library for Rust, suitable for short text and mixed-language text
https://github.com/pemistahl/lingua-rs
language-classification language-detection language-identification language-processing language-recognition natural-language-processing nlp nlp-machine-learning rust rust-crate rust-library
Last synced: 26 days ago
JSON representation
The most accurate natural language detection library for Rust, suitable for short text and mixed-language text
- Host: GitHub
- URL: https://github.com/pemistahl/lingua-rs
- Owner: pemistahl
- License: apache-2.0
- Created: 2020-06-17T10:47:30.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-04-18T12:19:44.000Z (7 months ago)
- Last Synced: 2024-04-19T19:07:57.842Z (7 months ago)
- Topics: language-classification, language-detection, language-identification, language-processing, language-recognition, natural-language-processing, nlp, nlp-machine-learning, rust, rust-crate, rust-library
- Language: Rust
- Homepage:
- Size: 215 MB
- Stars: 816
- Watchers: 8
- Forks: 33
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/pemistahl/lingua-rs/actions/workflows/rust-build.yml)
[](https://github.com/pemistahl/lingua-rs/actions/workflows/python-build.yml)
[](https://docs.rs/lingua)
[](https://codecov.io/gh/pemistahl/lingua-rs)
[](#3-which-languages-are-supported)
[](https://deps.rs/crate/lingua/1.6.2)
[](https://crates.io/crates/lingua)
[](https://crates.io/crates/lingua)
[](https://lib.rs/crates/lingua)
[](https://www.apache.org/licenses/LICENSE-2.0)
## 1. What does this library do?
Its task is simple: It tells you which language some text is written in.
This is very useful as a preprocessing step for linguistic data in natural language
processing applications such as text classification and spell checking.
Other use cases, for instance, might include routing e-mails to the right geographically
located customer service department, based on the e-mails' languages.
## 2. Why does this library exist?
Language detection is often done as part of large machine learning frameworks or natural
language processing applications. In cases where you don't need the full-fledged
functionality of those systems or don't want to learn the ropes of those,
a small flexible library comes in handy.
So far, other comprehensive open source libraries in the Rust ecosystem for
this task are [*CLD2*](https://github.com/emk/rust-cld2),
[*Whatlang*](https://github.com/greyblake/whatlang-rs) and
[*Whichlang*](https://github.com/quickwit-oss/whichlang).
Unfortunately, most of them have two major drawbacks:
1. Detection only works with quite lengthy text fragments. For very short text snippets
such as Twitter messages, they do not provide adequate results.
2. The more languages take part in the decision process, the less accurate are the
detection results.
*Lingua* aims at eliminating these problems. She nearly does not need any configuration and
yields pretty accurate results on both long and short text, even on single words and phrases.
She draws on both rule-based and statistical methods but does not use any dictionaries of words.
She does not need a connection to any external API or service either.
Once the library has been downloaded, it can be used completely offline.
## 3. Which languages are supported?
Compared to other language detection libraries, *Lingua's* focus is on *quality over quantity*, that is,
getting detection right for a small set of languages first before adding new ones.
Currently, the following 75 languages are supported:
- A
- Afrikaans
- Albanian
- Arabic
- Armenian
- Azerbaijani
- B
- Basque
- Belarusian
- Bengali
- Norwegian Bokmal
- Bosnian
- Bulgarian
- C
- Catalan
- Chinese
- Croatian
- Czech
- D
- Danish
- Dutch
- E
- English
- Esperanto
- Estonian
- F
- Finnish
- French
- G
- Ganda
- Georgian
- German
- Greek
- Gujarati
- H
- Hebrew
- Hindi
- Hungarian
- I
- Icelandic
- Indonesian
- Irish
- Italian
- J
- Japanese
- K
- Kazakh
- Korean
- L
- Latin
- Latvian
- Lithuanian
- M
- Macedonian
- Malay
- Maori
- Marathi
- Mongolian
- N
- Norwegian Nynorsk
- P
- Persian
- Polish
- Portuguese
- Punjabi
- R
- Romanian
- Russian
- S
- Serbian
- Shona
- Slovak
- Slovene
- Somali
- Sotho
- Spanish
- Swahili
- Swedish
- T
- Tagalog
- Tamil
- Telugu
- Thai
- Tsonga
- Tswana
- Turkish
- U
- Ukrainian
- Urdu
- V
- Vietnamese
- W
- Welsh
- X
- Xhosa
- Y
- Yoruba
- Z
- Zulu
## 4. How accurate is it?
*Lingua* is able to report accuracy statistics for some bundled test data available for each
supported language. The test data for each language is split into three parts:
1. a list of single words with a minimum length of 5 characters
2. a list of word pairs with a minimum length of 10 characters
3. a list of complete grammatical sentences of various lengths
Both the language models and the test data have been created from separate documents of the
[Wortschatz corpora](https://wortschatz.uni-leipzig.de) offered by Leipzig University, Germany.
Data crawled from various news websites have been used for training, each corpus comprising one
million sentences. For testing, corpora made of arbitrarily chosen websites have been used,
each comprising ten thousand sentences. From each test corpus, a random unsorted subset of
1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.
Given the generated test data, I have compared the detection results of *Lingua*, *CLD2*, *Whatlang*
and *Whichlang* running over the data of *Lingua's* supported 75 languages. Languages that are not supported
by the other libraries are simply ignored for the respective library during the detection process.
Each of the following sections contains four plots. The bar plots show the detailed accuracy
results for each supported language. The box plots illustrate the distributions of the
accuracy values for each classifier. The boxes themselves represent the areas which the
middle 50 % of data lie within. Within the colored boxes, the horizontal lines mark the
median of the distributions.
The first two plots in each section show the results for all supported languages
in each classifier, respectively. The last two plots are restricted to the common subset
of currently 16 languages that is supported by all compared classifiers. This distinction
makes sense because the first box plot creates the impression that Whichlang is the most
accurate classifier, but it is not. Whichlang supports only 16 languages whereas Lingua
supports 75 languages. For the second box plot, the supported languages in Whatlang and
Lingua have been restricted to those 16 languages supported by Whichlang. This provides
for a more accurate comparison and shows that overall, Lingua is the most accurate
language detection library in this comparison.
### 4.1 Single word detection
#### 4.1.1 All languages
Bar plot
#### 4.1.2 Common languages
Bar plot
### 4.2 Word pair detection
#### 4.2.1 All languages
Bar plot
#### 4.2.2 Common languages
Bar plot
### 4.3 Sentence detection
#### 4.3.1 All languages
Bar plot
#### 4.3.2 Common languages
Bar plot
### 4.4 Average detection
#### 4.4.1 All languages
Bar plot
#### 4.4.2 Common languages
Bar plot
### 4.5 Mean, median and standard deviation
The tables below show detailed statistics for each language and classifier
including mean, median and standard deviation.
#### 4.5.1 All languages
Open table
Language
Average
Single Words
Word Pairs
Sentences
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
Whichlang
Whatlang
CLD2
Afrikaans
78
64
-
51
55
58
37
-
22
12
80
62
-
41
56
96
93
-
92
96
Albanian
87
79
-
-
64
68
54
-
-
18
94
86
-
-
76
99
98
-
-
99
Arabic
98
94
100
93
66
96
88
100
88
19
99
95
100
92
81
99
99
100
99
99
Armenian
100
100
-
-
100
100
100
-
-
100
100
100
-
-
100
100
100
-
-
100
Azerbaijani
89
82
-
79
71
77
71
-
61
34
92
78
-
81
81
99
96
-
97
99
Basque
83
74
-
-
61
71
55
-
-
22
87
76
-
-
69
92
91
-
-
91
Belarusian
96
91
-
91
75
91
80
-
81
42
99
95
-
94
87
99
99
-
99
98
Bengali
100
100
-
99
62
100
100
-
100
19
100
100
-
100
69
100
100
-
99
99
Bokmal
58
49
-
49
-
38
27
-
24
-
58
47
-
44
-
76
74
-
80
-
Bosnian
34
29
-
-
18
29
22
-
-
4
34
28
-
-
15
40
36
-
-
36
Bulgarian
86
77
-
69
65
70
56
-
44
31
91
80
-
67
72
99
96
-
96
92
Catalan
70
58
-
51
37
50
33
-
29
4
73
60
-
45
29
86
81
-
80
79
Chinese
100
100
97
100
33
100
100
93
100
-
100
100
98
100
2
100
100
100
100
98
Croatian
72
59
-
61
51
53
36
-
34
33
74
57
-
54
46
90
85
-
94
72
Czech
80
70
-
63
73
65
54
-
42
50
84
71
-
66
79
91
87
-
82
90
Danish
81
70
-
53
59
61
45
-
31
26
83
70
-
45
56
97
95
-
84
94
Dutch
77
63
73
47
47
55
35
53
22
10
80
61
70
37
41
96
94
98
82
90
English
80
62
66
49
55
54
29
34
17
11
88
62
65
35
55
99
96
98
94
99
Esperanto
83
65
-
58
50
67
43
-
32
7
85
60
-
53
45
98
92
-
90
97
Estonian
91
83
-
67
65
79
62
-
43
23
96
87
-
62
73
99
99
-
96
99
Finnish
96
90
-
76
77
90
77
-
53
43
98
94
-
78
89
99
99
-
98
99
French
89
77
68
63
51
74
51
36
35
12
94
82
70
59
47
99
97
99
97
93
Ganda
91
83
-
-
61
79
64
-
-
23
95
86
-
-
62
100
99
-
-
98
Georgian
99
99
-
99
100
100
100
-
100
100
100
100
-
100
100
99
99
-
99
100
German
89
79
79
69
63
73
56
55
44
26
94
83
82
66
66
99
99
99
98
98
Greek
99
99
-
99
100
100
100
-
100
100
100
100
-
100
100
99
99
-
99
100
Gujarati
99
99
-
99
99
99
99
-
100
99
100
100
-
100
100
100
100
-
99
100
Hebrew
99
99
-
95
-
100
100
-
92
-
100
100
-
95
-
99
99
-
99
-
Hindi
73
32
100
68
76
60
11
100
54
55
64
19
100
54
75
94
67
100
96
99
Hungarian
94
90
-
72
75
86
77
-
50
40
97
93
-
70
85
100
99
-
98
99
Icelandic
93
87
-
-
65
82
71
-
-
25
96
92
-
-
73
99
99
-
-
99
Indonesian
60
47
-
69
62
39
25
-
42
36
60
45
-
69
62
82
71
-
95
88
Irish
90
85
-
-
66
81
70
-
-
29
94
90
-
-
77
95
94
-
-
92
Italian
86
71
85
56
44
69
41
68
25
6
91
73
87
48
32
99
98
99
96
93
Japanese
100
100
99
99
33
100
100
98
98
-
100
100
100
100
-
100
100
99
100
100
Kazakh
95
94
-
-
76
89
87
-
-
43
97
94
-
-
87
99
99
-
-
99
Korean
99
99
99
99
99
100
100
100
100
100
100
100
100
100
100
99
99
99
99
99
Latin
87
73
-
55
46
72
49
-
36
8
92
75
-
51
42
97
94
-
79
88
Latvian
93
87
-
85
72
84
74
-
70
33
96
90
-
86
84
98
96
-
97
98
Lithuanian
94
87
-
79
70
86
75
-
59
29
97
88
-
81
81
99
98
-
97
99
Macedonian
83
72
-
66
60
65
51
-
43
27
86
70
-
60
69
98
95
-
96
83
Malay
30
30
-
-
17
26
21
-
-
9
38
35
-
-
21
28
35
-
-
23
Maori
92
83
-
-
60
84
63
-
-
12
92
87
-
-
71
99
97
-
-
98
Marathi
84
38
-
78
83
73
15
-
59
65
84
29
-
80
86
96
71
-
94
98
Mongolian
96
95
-
-
78
92
87
-
-
43
98
98
-
-
92
99
99
-
-
99
Nynorsk
65
51
-
-
54
40
25
-
-
18
65
49
-
-
50
90
81
-
-
93
Persian
90
80
-
72
61
77
61
-
49
12
93
80
-
68
72
99
97
-
99
99
Polish
94
89
-
81
74
85
76
-
63
37
98
93
-
81
86
99
99
-
99
99
Portuguese
81
68
70
60
53
59
41
42
30
19
85
69
70
52
47
98
95
99
97
94
Punjabi
99
99
-
99
100
100
100
-
100
100
100
100
-
100
100
99
99
-
99
100
Romanian
86
72
-
66
53
68
49
-
45
11
91
73
-
62
53
99
94
-
90
96
Russian
89
78
100
67
60
76
59
100
51
26
94
83
100
66
67
97
92
100
84
86
Serbian
87
77
-
66
68
73
61
-
46
29
90
79
-
63
77
99
91
-
89
99
Shona
91
80
-
76
64
77
56
-
55
23
95
86
-
76
71
100
99
-
98
99
Slovak
84
75
-
64
71
64
49
-
40
37
90
78
-
60
75
98
97
-
93
99
Slovene
82
66
-
52
47
61
39
-
29
8
86
67
-
43
42
98
93
-
83
92
Somali
92
84
-
-
69
81
64
-
-
26
95
90
-
-
83
99
99
-
-
99
Sotho
85
71
-
-
53
66
42
-
-
12
90
74
-
-
54
99
97
-
-
95
Spanish
69
56
62
47
43
43
25
35
18
12
68
48
54
32
33
97
93
98
91
84
Swahili
80
69
-
-
57
60
43
-
-
15
84
68
-
-
58
98
96
-
-
97
Swedish
83
72
71
61
53
64
45
47
34
14
88
75
70
57
52
98
94
97
92
93
Tagalog
77
66
-
-
50
52
35
-
-
9
83
66
-
-
44
98
96
-
-
96
Tamil
100
100
-
100
100
100
100
-
100
100
100
100
-
100
100
100
100
-
100
100
Telugu
100
100
-
99
100
100
100
-
100
100
100
100
-
100
100
100
100
-
99
100
Thai
99
99
-
99
99
100
100
-
100
100
100
100
-
100
100
99
99
-
99
99
Tsonga
84
72
-
-
61
66
46
-
-
19
89
73
-
-
67
98
96
-
-
97
Tswana
84
71
-
-
56
65
44
-
-
17
88
73
-
-
57
98
96
-
-
94
Turkish
93
87
92
63
66
83
70
83
39
30
97
91
94
59
71
99
99
99
91
97
Ukrainian
92
86
-
85
77
84
74
-
69
45
97
91
-
88
87
95
92
-
98
98
Urdu
90
78
-
68
61
80
64
-
45
8
94
78
-
63
75
95
94
-
95
99
Vietnamese
90
87
85
91
63
78
75
64
75
-
94
87
91
97
89
99
98
99
99
99
Welsh
91
82
-
-
72
78
60
-
-
34
95
87
-
-
85
99
98
-
-
98
Xhosa
82
68
-
-
71
63
44
-
-
45
85
67
-
-
70
98
93
-
-
97
Yoruba
74
62
-
-
36
50
33
-
-
1
76
60
-
-
22
96
92
-
-
87
Zulu
80
70
-
76
54
62
45
-
53
18
83
71
-
77
51
97
94
-
99
93
Mean
86
77
84
74
64
74
60
69
57
34
89
78
84
71
68
96
93
99
94
94
Median
89.0
78.0
85.0
69.0
63.0
74.0
56.0
66.0
49.5
26.0
93.0
80.0
89.0
67.5
71.0
99.0
96.0
99.0
97.0
98.0
Standard Deviation
13.17
17.39
14.21
17.27
18.54
18.54
25.09
26.6
27.23
28.81
13.25
19.12
15.66
21.07
22.74
11.06
11.84
0.85
6.04
12.16
#### 4.5.2 Common languages
Open table
Language
Average
Single Words
Word Pairs
Sentences
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
Whichlang
Whatlang
CLD2
Arabic
100
100
100
100
66
100
100
100
100
19
100
100
100
100
81
100
100
100
100
99
Chinese
100
100
97
100
33
100
100
93
100
-
100
100
98
100
2
100
100
100
100
98
Dutch
87
80
73
63
47
73
60
53
39
10
89
81
70
54
41
99
98
98
96
90
English
87
76
66
57
55
70
52
34
29
11
93
77
65
47
55
99
98
98
95
99
French
92
84
68
74
51
82
65
36
50
12
96
88
70
73
47
99
98
99
98
93
German
92
86
79
76
63
82
70
55
54
26
95
89
82
75
66
99
99
99
98
98
Hindi
99
99
100
99
76
100
100
100
100
55
100
100
100
100
75
99
99
100
99
99
Italian
92
82
85
68
44
82
63
68
45
6
95
84
87
63
32
99
99
99
97
93
Japanese
100
100
99
99
33
100
100
98
98
-
100
100
100
100
-
100
100
99
100
100
Korean
99
99
99
99
99
100
100
100
100
100
100
100
100
100
100
99
99
99
99
99
Portuguese
85
76
70
67
53
69
55
42
42
19
88
77
70
62
47
99
97
99
98
94
Russian
100
100
100
100
60
100
100
100
100
26
100
100
100
100
67
100
100
100
100
86
Spanish
78
69
62
54
43
59
46
35
29
12
78
65
54
41
33
98
95
98
93
84
Swedish
93
86
71
74
53
83
69
47
51
14
96
90
70
74
52
99
98
97
97
93
Turkish
97
95
92
81
66
94
89
83
66
30
99
97
94
78
71
100
99
99
98
97
Vietnamese
94
91
85
92
63
86
83
64
79
-
96
92
91
98
89
99
99
99
99
99
Mean
93
89
84
81
57
86
78
69
68
26
95
90
84
79
57
99
99
99
98
95
Median
93.5
88.5
85.0
78.5
54.0
84.5
76.5
66.0
60.0
19.0
96.0
91.0
89.0
76.5
55.0
99.0
99.0
99.0
98.0
97.5
Standard Deviation
6.56
10.44
14.21
16.91
16.44
13.51
20.23
26.6
28.33
25.57
6.03
10.76
15.66
21.17
25.16
0.58
1.31
0.85
1.98
4.95
## 5. How fast is it?
Various benchmarks for all classifiers can be run via:
cargo bench --features benchmark
The benchmarks measure the processing time for classifying 2,000 sentences available
in the common 16 languages that are supported by each language detector. The results
below have been produced on an iMac 3.6 Ghz 8-Core Intel Core i9 with 40 GB RAM.
Whichlang has the shortest processing time, Lingua the longest.
| Detector | Single Thread | Multiple Threads |
|-----------------------------------------------|--------------:|-----------------:|
| Whichlang | 2.046 ms | 351.03 µs |
| CLD 2 | 8.923 ms | 2.05 ms |
| Whatlang (common languages) | 47.74 ms | 5.61 ms |
| Whatlang (all languages) | 113.08 ms | 12.99 ms |
| Lingua (low accuracy mode, common languages) | 180.54 ms | 24.55 ms |
| Lingua (high accuracy mode, common languages) | 333.31 ms | 37.35 ms |
| Lingua (low accuracy mode, all languages) | 373.15 ms | 48.18 ms |
| Lingua (high accuracy mode, all languages) | 622.00 ms | 96.65 ms |
The accuracy reporter script measures the time each language detector needs
to classify 3000 input texts for each of the supported 75 languages.
| Detector | Time |
|----------------------------------------------|-----------------:|
| Whichlang | 0.13 sec |
| CLD 2 | 1.67 sec |
| Lingua (low accuracy mode, multi-threaded) | 3.11 sec |
| Whatlang | 9.82 sec |
| Lingua (high accuracy mode, multi-threaded) | 10.36 sec |
| Lingua (low accuracy mode, single-threaded) | 21.06 sec |
| Lingua (high accuracy mode, single-threaded) | 57.15 sec |
## 6. Why is it better than other libraries?
Every language detector uses a probabilistic [n-gram](https://en.wikipedia.org/wiki/N-gram) model trained on the
character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is
satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short
phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are
available. The probabilities estimated from such few n-grams are not reliable. This is why *Lingua* makes use
of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.
A second important difference is that *Lingua* does not only use such a statistical model, but also a rule-based
engine. This engine first determines the alphabet of the input text and searches for characters which are unique
in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not
necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions
of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration.
This makes sense because loading less language models means less memory consumption and better runtime performance.
In general, it is always a good idea to restrict the set of languages to be considered in the classification process
using the respective api methods. If you know beforehand that certain languages are
never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism
of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.
## 7. How to reproduce the accuracy results?
If you want to reproduce the accuracy results above, you can generate the test reports yourself for all classifiers
and all languages by doing:
cargo run --release --bin accuracy_reports --features accuracy-reports
It is important to use the `--release` flag here because loading the language models in debug mode takes too much time.
For each detector and language, a test report file is then written into
[`/accuracy-reports`](https://github.com/pemistahl/lingua-rs/tree/main/accuracy-reports),
to be found next to the `src` directory. As an example, here is the current output of the *Lingua* German report:
```
##### German #####
>>> Accuracy on average: 89.23%
>> Detection of 1000 single words (average length: 9 chars)
Accuracy: 73.9%
Erroneously classified as Dutch: 2.3%, Danish: 2.1%, English: 2%, Latin: 1.9%, Bokmal: 1.6%, Basque: 1.2%, Esperanto: 1.2%, French: 1.2%, Italian: 1.2%, Swedish: 1%, Afrikaans: 0.8%, Tsonga: 0.7%, Nynorsk: 0.6%, Spanish: 0.6%, Yoruba: 0.6%, Finnish: 0.5%, Sotho: 0.5%, Welsh: 0.5%, Estonian: 0.4%, Irish: 0.4%, Polish: 0.4%, Swahili: 0.4%, Tagalog: 0.4%, Tswana: 0.4%, Bosnian: 0.3%, Icelandic: 0.3%, Romanian: 0.3%, Albanian: 0.2%, Catalan: 0.2%, Croatian: 0.2%, Indonesian: 0.2%, Lithuanian: 0.2%, Maori: 0.2%, Turkish: 0.2%, Xhosa: 0.2%, Zulu: 0.2%, Latvian: 0.1%, Malay: 0.1%, Slovak: 0.1%, Slovene: 0.1%, Somali: 0.1%
>> Detection of 1000 word pairs (average length: 18 chars)
Accuracy: 94.1%
Erroneously classified as Dutch: 0.9%, Latin: 0.8%, English: 0.7%, Swedish: 0.6%, Danish: 0.5%, French: 0.4%, Bokmal: 0.3%, Irish: 0.2%, Tagalog: 0.2%, Afrikaans: 0.1%, Esperanto: 0.1%, Estonian: 0.1%, Finnish: 0.1%, Italian: 0.1%, Maori: 0.1%, Nynorsk: 0.1%, Somali: 0.1%, Swahili: 0.1%, Tsonga: 0.1%, Turkish: 0.1%, Welsh: 0.1%, Zulu: 0.1%
>> Detection of 1000 sentences (average length: 111 chars)
Accuracy: 99.7%
Erroneously classified as Dutch: 0.2%, Latin: 0.1%
```
## 8. How to add it to your project?
Add *Lingua* to your `Cargo.toml` file like so:
```toml
[dependencies]
lingua = "1.6.2"
```
By default, this will download the language model dependencies for all 75 supported languages,
a total of approximately 90 MB. If your bandwidth or hard drive space is limited, or you simply
do not need all languages, you can specify a subset of the language models to be downloaded as
separate features in your `Cargo.toml`:
```toml
[dependencies]
lingua = { version = "1.6.2", default-features = false, features = ["french", "italian", "spanish"] }
```
## 9. How to build?
In order to build the source code yourself, you need the
[stable Rust toolchain](https://www.rust-lang.org/tools/install) installed on your machine
so that [*cargo*](https://doc.rust-lang.org/cargo/), the Rust package manager is available.
```
git clone https://github.com/pemistahl/lingua-rs.git
cd lingua-rs
cargo build
```
The source code is accompanied by an extensive unit test suite. To run them, simply say:
cargo test
With the help of [PyO3](https://github.com/PyO3/pyo3) and
[Maturin](https://github.com/PyO3/maturin), the library has been compiled to a
Python extension module so that it can be used within any Python software as well.
It is available in the [Python Package Index](https://pypi.org/project/lingua-language-detector)
and can be installed with:
```shell
pip install lingua-language-detector
```
To build the Python extension module yourself, create a virtual environment and install
[Maturin](https://github.com/PyO3/maturin).
```shell
python -m venv .venv
source .venv/bin/activate
pip install maturin
maturin build
```
## 10. How to use?
### 10.1 Basic usage
```rust
use lingua::{Language, LanguageDetector, LanguageDetectorBuilder};
use lingua::Language::{English, French, German, Spanish};
let languages = vec![English, French, German, Spanish];
let detector: LanguageDetector = LanguageDetectorBuilder::from_languages(&languages).build();
let detected_language: Option = detector.detect_language_of("languages are awesome");
assert_eq!(detected_language, Some(English));
```
### 10.2 Minimum relative distance
By default, *Lingua* returns the most likely language for a given input text. However, there are
certain words that are spelled the same in more than one language. The word *prologue*, for
instance, is both a valid English and French word. *Lingua* would output either English or
French which might be wrong in the given context. For cases like that, it is possible to
specify a minimum relative distance that the logarithmized and summed up probabilities for
each possible language have to satisfy. It can be stated in the following way:
```rust
use lingua::LanguageDetectorBuilder;
use lingua::Language::{English, French, German, Spanish};
let detector = LanguageDetectorBuilder::from_languages(&[English, French, German, Spanish])
.with_minimum_relative_distance(0.9)
.build();
let detected_language = detector.detect_language_of("languages are awesome");
assert_eq!(detected_language, None);
```
Be aware that the distance between the language probabilities is dependent on the length of the
input text. The longer the input text, the larger the distance between the languages. So if you
want to classify very short text phrases, do not set the minimum relative distance too high.
Otherwise [`None`](https://doc.rust-lang.org/std/option/enum.Option.html#variant.None) will be
returned most of the time as in the example above. This is the return value for cases where
language detection is not reliably possible.
### 10.3 Confidence values
Knowing about the most likely language is nice but how reliable is the computed likelihood?
And how less likely are the other examined languages in comparison to the most likely one?
These questions can be answered as well:
```rust
use lingua::Language::{English, French, German, Spanish};
use lingua::LanguageDetectorBuilder;
fn main() {
let languages = vec![English, French, German, Spanish];
let detector = LanguageDetectorBuilder::from_languages(&languages).build();
let confidence_values = detector
.compute_language_confidence_values("languages are awesome")
.into_iter()
.map(|(language, confidence)| (language, (confidence * 100.0).round() / 100.0))
.collect::>();
assert_eq!(
confidence_values,
vec![
(English, 0.93),
(French, 0.04),
(German, 0.02),
(Spanish, 0.01)
]
);
}
```
In the example above, a vector of two-element tuples is returned containing all possible languages
sorted by their confidence value in descending order. Each value is a probability between 0.0 and 1.0. The probabilities of
all languages will sum to 1.0. If the language is unambiguously identified by the rule engine, the
value 1.0 will always be returned for this language. The other languages will receive a value of 0.0.
There is also a method for returning the confidence value for one specific language only:
```rust
use lingua::Language::{English, French, German, Spanish};
use lingua::LanguageDetectorBuilder;
fn main() {
let languages = vec![English, French, German, Spanish];
let detector = LanguageDetectorBuilder::from_languages(&languages).build();
let confidence = detector.compute_language_confidence("languages are awesome", French);
let rounded_confidence = (confidence * 100.0).round() / 100.0;
assert_eq!(rounded_confidence, 0.04);
}
```
The value that this method computes is a number between 0.0 and 1.0.
If the language is unambiguously identified by the rule engine, the value
1.0 will always be returned. If the given language is not supported by
this detector instance, the value 0.0 will always be returned.
### 10.4 Eager loading versus lazy loading
By default, *Lingua* uses lazy-loading to load only those language models on demand which are
considered relevant by the rule-based filter engine. For web services, for instance, it is
rather beneficial to preload all language models into memory to avoid unexpected latency while
waiting for the service response. If you want to enable the eager-loading mode, you can do it
like this:
```rust
LanguageDetectorBuilder::from_all_languages().with_preloaded_language_models().build();
```
Multiple instances of `LanguageDetector` share the same language models in memory which are
accessed asynchronously by the instances.
### 10.5 Low accuracy mode versus high accuracy mode
*Lingua's* high detection accuracy comes at the cost of being noticeably slower
than other language detectors. The large language models also consume significant
amounts of memory. These requirements might not be feasible for systems running low
on resources. If you want to classify mostly long texts or need to save resources,
you can enable a *low accuracy mode* that loads only a small subset of the language
models into memory:
```rust
LanguageDetectorBuilder::from_all_languages().with_low_accuracy_mode().build();
```
The downside of this approach is that detection accuracy for short texts consisting
of less than 120 characters will drop significantly. However, detection accuracy for
texts which are longer than 120 characters will remain mostly unaffected.
In high accuracy mode (the default), the language detector consumes approximately
970 MB of memory if all language models are loaded. In low accuracy mode, memory
consumption is reduced to approximately 72 MB. The goal is to further reduce memory
consumption in later releases.
An alternative for a smaller memory footprint and faster performance is to reduce the set
of languages when building the language detector. In most cases, it is not advisable to
build the detector from all supported languages. When you have knowledge about
the texts you want to classify you can almost always rule out certain languages as impossible
or unlikely to occur.
### 10.6 Detection of multiple languages in mixed-language texts
In contrast to most other language detectors, *Lingua* is able to detect multiple languages
in mixed-language texts. This feature can yield quite reasonable results, but it is still
in an experimental state and therefore the detection result is highly dependent on the input
text. It works best in high-accuracy mode with multiple long words for each language.
The shorter the phrases and their words are, the less accurate are the results. Reducing the
set of languages when building the language detector can also improve accuracy for this task
if the languages occurring in the text are equal to the languages supported by the respective
language detector instance.
```rust
use lingua::DetectionResult;
use lingua::Language::{English, French, German};
use lingua::LanguageDetectorBuilder;
fn main() {
let languages = vec![English, French, German];
let detector = LanguageDetectorBuilder::from_languages(&languages).build();
let sentence = "Parlez-vous français? \
Ich spreche Französisch nur ein bisschen. \
A little bit is better than nothing.";
let results: Vec = detector.detect_multiple_languages_of(sentence);
if let [first, second, third] = &results[..] {
assert_eq!(first.language(), French);
assert_eq!(
&sentence[first.start_index()..first.end_index()],
"Parlez-vous français? "
);
assert_eq!(second.language(), German);
assert_eq!(
&sentence[second.start_index()..second.end_index()],
"Ich spreche Französisch nur ein bisschen. "
);
assert_eq!(third.language(), English);
assert_eq!(
&sentence[third.start_index()..third.end_index()],
"A little bit is better than nothing."
);
}
}
```
In the example above, a vector of [`DetectionResult`](https://github.com/pemistahl/lingua-rs/blob/main/src/result.rs#L21)
is returned. Each entry in the vector describes a contiguous single-language text section,
providing start and end indices of the respective substring.
### 10.7 Single-threaded versus multi-threaded language detection
The `LanguageDetector` methods explained above all operate in a single thread.
If you want to classify a very large set of texts, you will probably want to
use all available CPU cores efficiently in multiple threads for maximum performance.
Every single-threaded method has a multi-threaded equivalent that accepts a list of texts
and returns a list of results.
| Single-threaded | Multi-threaded |
|--------------------------------------|--------------------------------------------------|
| `detect_language_of` | `detect_languages_in_parallel_of` |
| `detect_multiple_languages_of` | `detect_multiple_languages_in_parallel_of` |
| `compute_language_confidence_values` | `compute_language_confidence_values_in_parallel` |
| `compute_language_confidence` | `compute_language_confidence_in_parallel` |
### 10.8 Methods to build the LanguageDetector
There might be classification tasks where you know beforehand that your language data is
definitely not written in Latin, for instance (what a surprise :-). The detection accuracy can
become better in such cases if you exclude certain languages from the decision process or just
explicitly include relevant languages:
```rust
use lingua::{LanguageDetectorBuilder, Language, IsoCode639_1, IsoCode639_3};
// Include all languages available in the library.
LanguageDetectorBuilder::from_all_languages();
// Include only languages that are not yet extinct (= currently excludes Latin).
LanguageDetectorBuilder::from_all_spoken_languages();
// Include only languages written with Cyrillic script.
LanguageDetectorBuilder::from_all_languages_with_cyrillic_script();
// Exclude only the Spanish language from the decision algorithm.
LanguageDetectorBuilder::from_all_languages_without(&[Language::Spanish]);
// Only decide between English and German.
LanguageDetectorBuilder::from_languages(&[Language::English, Language::German]);
// Select languages by ISO 639-1 code.
LanguageDetectorBuilder::from_iso_codes_639_1(&[IsoCode639_1::EN, IsoCode639_1::DE]);
// Select languages by ISO 639-3 code.
LanguageDetectorBuilder::from_iso_codes_639_3(&[IsoCode639_3::ENG, IsoCode639_3::DEU]);
```
## 11. WebAssembly support
This library can be compiled to [WebAssembly (WASM)](https://webassembly.org) which allows to use *Lingua*
in any JavaScript-based project, be it in the browser or in the back end running on [Node.js](https://nodejs.org).
The easiest way to compile is to use [`wasm-pack`](https://rustwasm.github.io/wasm-pack). After the installation,
you can, for instance, build the library with the web target so that it can be directly used in the browser:
wasm-pack build --target web
By default, all 75 supported languages are included in the compiled wasm file which has a size of 74 MB, approximately.
If you only need a subset of certain languages, you can tell `wasm-pack` which ones to include:
wasm-pack build --target web -- --no-default-features --features "french,italian,spanish"
This creates a directory named `pkg` on the top-level of this repository, containing the compiled wasm files
and JavaScript and TypeScript bindings. In an HTML file, you can then call *Lingua* like the following, for instance:
```html
import init, { LanguageDetectorBuilder } from './pkg/lingua.js';
init().then(_ => {
const detector = LanguageDetectorBuilder.fromAllLanguages().build();
console.log(detector.computeLanguageConfidenceValues("languages are awesome"));
});
```
There are also some integration tests available for Node.js and all major browsers.
To run them, simply say:
wasm-pack test --node --headless --chrome --firefox --safari
The output of `wasm-pack` will be hosted in a [separate repository](https://github.com/pemistahl/lingua-js) which
allows to add further JavaScript-related configuration, tests and documentation. *Lingua* will then be added to the
[npm registry](https://www.npmjs.com) as well, allowing for an easy download and installation within every JavaScript
or TypeScript project.
## 12. What's next for version 1.7.0?
Take a look at the [planned issues](https://github.com/pemistahl/lingua-rs/milestone/9).
## 13. Contributions
- [Josh Rotenberg](https://github.com/joshrotenberg) has written a [wrapper](https://github.com/joshrotenberg/lingua_ex)
for using *Lingua* with the [Elixir programming language](https://elixir-lang.org/).
- [Simon Liang](https://github.com/lhr0909) has written a [wrapper](https://github.com/xanthous-tech/lingua-node)
for using *Lingua* with [NodeJS](https://nodejs.org/).
Any contributions to *Lingua* are very much appreciated. Please read the instructions
in [`CONTRIBUTING.md`](https://github.com/pemistahl/lingua-rs/blob/main/CONTRIBUTING.md)
for how to add new languages to the library.