{"id":20815387,"url":"https://github.com/gluschenko/panlingo","last_synced_at":"2025-06-25T07:07:29.949Z","repository":{"id":248400060,"uuid":"817931130","full_name":"gluschenko/panlingo","owner":"gluschenko","description":"Collection of language detection libraries for .NET: FastText, CLD2, CLD3, MediaPipe, Lingua, Whatlang","archived":false,"fork":false,"pushed_at":"2025-06-18T21:30:32.000Z","size":1567,"stargazers_count":9,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-18T22:30:17.801Z","etag":null,"topics":["cld2","cld3","dotnet","dotnet-core","fasttext","interop","language-detection","language-identification","lingua","machine-learning","mediapipe","neural-networks","nlp","whatlang","wrapper"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gluschenko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-06-20T18:35:53.000Z","updated_at":"2025-06-18T21:30:41.000Z","dependencies_parsed_at":"2024-09-10T22:01:13.238Z","dependency_job_id":"acb9af54-c50b-4616-98fa-b6db86f2ed3b","html_url":"https://github.com/gluschenko/panlingo","commit_stats":null,"previous_names":["gluschenko/language-identification"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gluschenko/panlingo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluschenko%2Fpanlingo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluschenko%2Fpanlingo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluschenko%2Fpanlingo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluschenko%2Fpanlingo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gluschenko","download_url":"https://codeload.github.com/gluschenko/panlingo/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluschenko%2Fpanlingo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261823770,"owners_count":23215145,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cld2","cld3","dotnet","dotnet-core","fasttext","interop","language-detection","language-identification","lingua","machine-learning","mediapipe","neural-networks","nlp","whatlang","wrapper"],"created_at":"2024-11-17T21:22:35.037Z","updated_at":"2025-06-25T07:07:29.931Z","avatar_url":"https://github.com/gluschenko.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Panlingo\n\n[![GitHub CI](https://github.com/gluschenko/panlingo/actions/workflows/github-ci.yml/badge.svg)](https://github.com/gluschenko/panlingo/actions/workflows/github-ci.yml)\n\n## Overview\n\nWelcome to the **Panlingo** repository! 🚀\n\nThis project presents a comprehensive collection of language identification \nlibraries for .NET. Its primary purpose is to bring popular \nlanguage identification models to the .NET ecosystem, \nallowing developers to seamlessly integrate language detection \nfunctionality into their applications.\n\n## Libraries\n\n| Library | NuGet Package |\n| :------ | :------------ |\n| [Panlingo.LanguageIdentification.CLD2](./README_CLD2.md) |  [![NuGet Version](https://img.shields.io/nuget/vpre/Panlingo.LanguageIdentification.CLD2)](https://www.nuget.org/packages/Panlingo.LanguageIdentification.CLD2/) |\n| [Panlingo.LanguageIdentification.CLD3](./README_CLD3.md) |  [![NuGet Version](https://img.shields.io/nuget/vpre/Panlingo.LanguageIdentification.CLD3)](https://www.nuget.org/packages/Panlingo.LanguageIdentification.CLD3/) |\n| [Panlingo.LanguageIdentification.FastText](./README_FASTTEXT.md) |  [![NuGet Version](https://img.shields.io/nuget/vpre/Panlingo.LanguageIdentification.FastText)](https://www.nuget.org/packages/Panlingo.LanguageIdentification.FastText/) |\n| [Panlingo.LanguageIdentification.Whatlang](./README_WHATLANG.md) |  [![NuGet Version](https://img.shields.io/nuget/vpre/Panlingo.LanguageIdentification.Whatlang)](https://www.nuget.org/packages/Panlingo.LanguageIdentification.Whatlang/) |\n| [Panlingo.LanguageIdentification.MediaPipe](./README_MEDIAPIPE.md) |  [![NuGet Version](https://img.shields.io/nuget/vpre/Panlingo.LanguageIdentification.MediaPipe)](https://www.nuget.org/packages/Panlingo.LanguageIdentification.MediaPipe/) |\n| [Panlingo.LanguageIdentification.Lingua](./README_LINGUA.md) |  [![NuGet Version](https://img.shields.io/nuget/vpre/Panlingo.LanguageIdentification.Lingua)](https://www.nuget.org/packages/Panlingo.LanguageIdentification.Lingua/) |\n| [Panlingo.LanguageCode](./README_LANGUAGE_CODE.md) |  [![NuGet Version](https://img.shields.io/nuget/vpre/Panlingo.LanguageCode)](https://www.nuget.org/packages/Panlingo.LanguageCode/) |\n\n## Contents\n\n1. [Models](#models)\n2. [Features](#features)\n3. [Platform Support](#platform-support)\n4. [Key Concerns](#key-concerns)\n5. [Contributing](#contributing)\n6. [License](#license)\n\n## Models\n\n| Model         | Authors              | License    | Original source code | Wrapper docs         |\n| :------------ | :------------------- | :--------- | :------------------- | :------------------- |\n| **CLD2**      | Google, Inc.         | Apache-2.0 | [@CLD2Owners/cld2](https://github.com/CLD2Owners/cld2) | [link](./README_CLD2.md) |\n| **CLD3**      | Google, Inc.         | Apache-2.0 | [@google/cld3](https://github.com/google/cld3) | [link](./README_CLD3.md) |\n| **FastText**  | Meta Platforms, Inc. | MIT        | [@facebookresearch/fastText](https://github.com/facebookresearch/fastText) | [link](./README_FASTTEXT.md) |\n| **Whatlang**  | Serhii Potapov       | MIT        | [@greyblake/whatlang-rs](https://github.com/greyblake/whatlang-rs) | [link](./README_WHATLANG.md) |\n| **MediaPipe** | Google, Inc.         | Apache-2.0 | [@google-ai-edge/mediapipe](https://github.com/google-ai-edge/mediapipe) | [link](./README_MEDIAPIPE.md) |\n| **Lingua**    | Peter M. Stahl       | Apache-2.0 | [@pemistahl/lingua-rs](https://github.com/pemistahl/lingua-rs) | [link](./README_LINGUA.md) |\n\n## Key concerns\n\n- Zero-dependency development. \n- The original code of libraries (CLD2, CLD3, FastText, MediaPipe) is used as submodules without additional significant modifications or improvements (except for a small monkey-patching 😂). Third-party code is not included into this repository.\n- Preserve the original library behavior without breaking changes.\n\n## Features\n\n| Feature                        | CLD2      | CLD3           | FastText*          | Whatlang | MediaPipe**    | Lingua   |\n| :----------------------------- | :-------: | :------------: | :----------------: | :------: | :------------: | :------: |\n| **Single language prediction** | Yes       | Yes            | Yes                | Yes      | Yes            | Yes      |\n| **Multi language prediction**  | Yes       | Yes            | Yes                | No       | Yes            | Yes      |\n| **Supported languages**        | 83        | 107            | 176 or 217         | 69       | 110            | 75       |\n| **Unknown language detection** | Yes       | Yes            | No                 | No       | Yes            | No       |\n| **Algorithm**                  | quadgrams | neural network | neural network     | trigrams | neural network | trigrams |\n| **Script detection**           | No        | No             | Yes (only lid218e) | Yes      | No             | No       |\n| **Written in**                 | C++       | C++            | C++                | Rust     | C++            | Rust     |\n\n\\* When using these models: \n[lid176](https://fasttext.cc/docs/en/language-identification.html), \n[lid218e](https://huggingface.co/facebook/fasttext-language-identification)\n\n\\*\\* When using [MediaPipe Language Detector](https://storage.googleapis.com/mediapipe-assets/LanguageDetector%20Model%20Card.pdf)\n\n## Platform support\n\n| Model         |  Linux             | Windows            |  macOS             |\n| :------------ | :----------------: | :----------------: | :----------------: |\n| **CLD2**      | :white_check_mark: | :white_check_mark: | :white_check_mark: |\n| **CLD3**      | :white_check_mark: | :white_check_mark: | :white_check_mark: |\n| **FastText**  | :white_check_mark: | :white_check_mark: | :white_check_mark: |\n| **Whatlang**  | :white_check_mark: | :white_check_mark: | :white_check_mark: |\n| **MediaPipe** | :white_check_mark: | :white_check_mark: | :white_check_mark:\\* |\n| **Lingua**    | :white_check_mark: | :white_check_mark: | :white_check_mark:\\* |\n\n:white_check_mark: — Full support |\n:x: — No support |\n:construction: — Under research\n\n\\* **arm64** CPU only (Apple silicon M series)\n\n## Contributing\n\nWe welcome contributions from developers of all skill levels. Whether you're fixing a bug, adding a new feature, or improving documentation, we appreciate your help in making this project better.\n\n### Getting Started\n\nTo get started with contributing, follow these simple steps:\n\n1. **Clone the Repository**\n\n   First, clone the repository to your local machine with the following command:\n\n   ```bash\n   git clone --recurse-submodules --remote-submodules https://github.com/gluschenko/panlingo.git\n   ```\n\n2. **Create a Branch**\n\n   Before you start making changes, create a new branch to keep your work organized. Use a descriptive name for your branch to make it easy to understand its purpose:\n\n   ```bash\n   git checkout -b feature/your-feature-name\n   ```\n\n3. **Make Changes**\n\n   Now, you can make changes to the codebase. Please ensure your code follows our project's coding standards and includes relevant tests if applicable.\n\n4. **Commit Your Changes**\n\n   Once you've made your changes, commit them with a clear and informative commit message:\n\n   ```bash\n   git add .\n   git commit -m \"Add description of your changes\"\n   ```\n\n5. **Push Your Changes**\n\n   Push your branch to the remote repository:\n\n   ```bash\n   git push origin feature/your-feature-name\n   ```\n\n6. **Build**\n\n   Each library project in the solution has four configurations: `ReleaseLinuxOnly`, `DebugLinuxOnly`, `Release`, and `Debug`. \n   - The `ReleaseLinuxOnly` and `DebugLinuxOnly` configurations are for building on a local Linux or Windows machine (WSL is supported as well). It produces native binaries only for Linux.\n   - The `Release` and `Debug` configurations are intended for cross-platform builds, which are only supported in CI/CD environments like GitHub Actions.\n   \n   Here's how you can build the projects on a local Linux machine.\n   \n   **Requirements:**\n   - Windows 10 or higher.\n   - [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) set up for simulating a Linux environment.\n   - [Docker Desktop](https://www.docker.com/products/docker-desktop/) for container management.\n   - 45GB of free disk space for storing Docker images.\n   - Modern CPU with AVX support for optimal performance.\n   \n   **To build the entire solution:**\n   ```bash\n   cd src\n   dotnet build -c ReleaseLinuxOnly\n   ```\n   \n   **To build a specific library:**\n   ```bash\n   cd src/LanguageIdentification.FastText.Native\n   dotnet build -c ReleaseLinuxOnly\n   \n   cd src/LanguageIdentification.FastText\n   dotnet build -c ReleaseLinuxOnly\n   ```\n\n7. **Run**\n\n   Here's how you can run the test project on both Linux and Windows.\n\n   **Linux:**\n   To run the test project on a Linux machine, navigate to the test project's directory and use the following command:\n   ```bash\n   cd src/LanguageIdentification.FastText.ConsoleTest\n   dotnet run -c ReleaseLinuxOnly\n   ```\n   \n   **Windows:**\n   If you're on a Windows machine, you can still run the test project using WSL. Follow these steps:\n   ```bash\n   cd src/LanguageIdentification.FastText.ConsoleTest\n   wsl -d Ubuntu -e bash -c \"dotnet run -c ReleaseLinuxOnly\"\n   ```\n\n8. **Open a Pull Request**\n\n   Navigate to the repository on GitHub and open a pull request. Provide a detailed description of your changes and any additional information that might help reviewers understand your contribution.\n\n### Review Process\n\nAfter opening a pull request, it will be reviewed by one of the project maintainers. Feedback and suggestions might be provided to ensure the code meets our quality standards. Once approved, your changes will be merged into the main branch.\n\n### Code of Conduct\n\nPlease note that this project adheres to a [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code.\n\n## License\n\nThis project is licensed under the [MIT License](./LICENSE) © 2024–2025 Alexander Gluschenko.\n\nIncludes software from the following project(s):\n\n- [CLD2](https://github.com/CLD2Owners/cld2) — © 2013 Google Inc., licensed under Apache-2.0  \n- [CLD3](https://github.com/google/cld3) — © 2016 Google Inc., licensed under Apache-2.0  \n- [FastText](https://github.com/facebookresearch/fastText) — © 2016–present Meta Platforms, Inc., licensed under MIT  \n- [Lingua](https://github.com/pemistahl/lingua-rs) — © 2020–2023 Peter M. Stahl, licensed under Apache-2.0  \n- [MediaPipe Language Detector](https://github.com/google-ai-edge/mediapipe) — © 2023–2025 Google Inc., licensed under Apache-2.0  \n- [Whatlang](https://github.com/greyblake/whatlang-rs) — © 2017 Serhii Potapov and others, licensed under MIT\n\nSee the [LICENSE](./LICENSE) file for full details.\n\n---\n\nHappy hacking! 👩‍💻👨‍💻\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgluschenko%2Fpanlingo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgluschenko%2Fpanlingo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgluschenko%2Fpanlingo/lists"}