{"id":23734501,"url":"https://github.com/kgnlp/allophant","last_synced_at":"2026-03-04T23:31:15.238Z","repository":{"id":171932922,"uuid":"646903941","full_name":"kgnlp/allophant","owner":"kgnlp","description":"A multilingual phoneme recognizer capable of generalizing zero-shot to unseen phoneme inventories.","archived":false,"fork":false,"pushed_at":"2025-03-14T16:36:38.000Z","size":1359,"stargazers_count":25,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-09-27T04:07:09.133Z","etag":null,"topics":["cross-lingual","machine-learning","multilingual","neural-networks","phoneme-recognition","speech-recognition","zero-shot"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kgnlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-29T15:51:44.000Z","updated_at":"2025-06-28T14:20:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"5a3256b9-8ade-4c13-9659-547a8166eea5","html_url":"https://github.com/kgnlp/allophant","commit_stats":null,"previous_names":["kgnlp/allophant"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/kgnlp/allophant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgnlp%2Fallophant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgnlp%2Fallophant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgnlp%2Fallophant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgnlp%2Fallophant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kgnlp","download_url":"https://codeload.github.com/kgnlp/allophant/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgnlp%2Fallophant/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30099338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T22:49:54.894Z","status":"ssl_error","status_checked_at":"2026-03-04T22:49:48.883Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cross-lingual","machine-learning","multilingual","neural-networks","phoneme-recognition","speech-recognition","zero-shot"],"created_at":"2024-12-31T05:45:58.602Z","updated_at":"2026-03-04T23:31:15.198Z","avatar_url":"https://github.com/kgnlp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Allophant\n\nAllophant is a multilingual phoneme recognizer trained on spoken sentences in 34 languages, capable of generalizing zero-shot to unseen phoneme inventories.\n\nThis implementation was utilized in our INTERSPEECH 2023 paper [\"Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes\"](https://www.isca-archive.org/interspeech_2023/glocker23_interspeech.html) ([Citation](#citation))\n\n## Checkpoints\n\nPre-trained checkpoints for all evaluated models can be found on [Hugging Face](https://huggingface.co/collections/kgnlp/allophant-67141a8deba78564d9dcfdad):\n\n| Model Name       | UCLA Phonetic Corpus (PER) | UCLA Phonetic Corpus (AER) | Common Voice (PER) | Common Voice (AER) |\n| ---------------- | -------------------------: | -------------------------: | -----------------: | -----------------: |\n| [Multitask](https://huggingface.co/kgnlp/allophant)        | **45.62%** | 19.44% | **34.34%** | **8.36%** |\n| [Hierarchical](https://huggingface.co/kgnlp/allophant-hierarchical)     | 46.09% | **19.18%** | 34.35% | 8.56% |\n| [Multitask Shared](https://huggingface.co/kgnlp/allophant-shared) | 46.05% | 19.52% | 41.20% | 8.88% |\n| [Baseline Shared](https://huggingface.co/kgnlp/allophant-baseline-shared)  | 48.25% |   -    | 45.35% |  -    |\n| [Baseline](https://huggingface.co/kgnlp/allophant-baseline)         | 57.01% |   -    | 46.95% |  -    |\n\nNote that our baseline models were trained without phonetic feature classifiers and therefore only support phoneme recognition.\n\n## Result Files\n\nJSON files containing detailed error rates and statistics for all languages can be found in the [`interspeech_results`](/interspeech_results/) directory. Results on the [UCLA Phonetic Corpus](https://github.com/xinjli/ucla-phonetic-corpus) are stored in files ending in \"ucla\", while files containing results on the training subset of languages from [Mozilla Common Voice](https://commonvoice.mozilla.org) end in \"commonvoice\". See [Error Rates](#error-rates) for more information.\n\n## Installation\n\n### System Dependencies\n\nFor most Linux and macOS systems, pre-built binaries are available via pip. For installation on other platforms or when building from source, a Rust compiler is required for building the native `pyo3` extension. [Rustup](https://rustup.rs/) is recommended for managing Rust installations.\n\n#### Optional\n\nTorchaudio mp3 support requires ffmpeg to be installed on the system. E.g. for Debian-based Linux distributions:\n\n```bash\nsudo apt update \u0026\u0026 sudo apt install ffmpeg\n```\nFor transcribing training and evaluation data with eSpeak NG G2P, The [espeak-ng](https://github.com/espeak-ng/espeak-ng) package is required:\n\n```bash\nsudo apt install espeak-ng\n```\n\nWe transcribed Common Voice using version 1.51 for our paper.\n\n### Allophant Package\n\nAllophant can be installed via pip:\n\n```bash\npip install allophant\n```\n\nNote that the package currently requires Python \u003e= 3.10 and was tested on 3.12. For use on GPU, torch and torchaudio may need to be manually installed for your required CUDA or ROCm version. ([PyTorch installation](https://pytorch.org/get-started/locally/))\n\nFor development, an editable package can be installed as follows:\n\n```bash\ngit clone https://github.com/kgnlp/allophant\ncd allophant\npip install -e allophant\n```\n\n## Usage\n\n### Inference With Pre-trained Models\n\nA pre-trained model can be loaded with the `allophant` package from a huggingface checkpoint or local file:\n\n```python\nfrom allophant.estimator import Estimator\n\ndevice = \"cpu\"\nmodel, attribute_indexer = Estimator.restore(\"kgnlp/allophant\", device=device)\nsupported_features = attribute_indexer.feature_names\n# The phonetic feature categories supported by the model, including \"phonemes\"\nprint(supported_features)\n```\nAllophant supports decoding custom phoneme inventories, which can be constructed in multiple ways:\n\n```python\n# 1. For a single language:\ninventory = attribute_indexer.phoneme_inventory(\"es\")\n# 2. For multiple languages, e.g. in code-switching scenarios\ninventory = attribute_indexer.phoneme_inventory([\"es\", \"it\"])\n# 3. Any custom selection of phones for which features are available in the Allophoible database\ninventory = ['a', 'ai̯', 'au̯', 'b', 'e', 'eu̯', 'f', 'ɡ', 'l', 'ʎ', 'm', 'ɲ', 'o', 'p', 'ɾ', 's', 't̠ʃ']\n````\n\nAudio files can then be loaded, resampled and transcribed using the given\ninventory by first computing the log probabilities for each classifier:\n\n```python\nimport torch\nimport torchaudio\nfrom allophant.dataset_processing import Batch\n\n# Load an audio file and resample the first channel to the sample rate used by the model\naudio, sample_rate = torchaudio.load(\"utterance.wav\")\naudio = torchaudio.functional.resample(audio[:1], sample_rate, model.sample_rate)\n\n# Construct a batch of 0-padded single channel audio, lengths and language IDs\n# Language ID can be 0 for inference\nbatch = Batch(audio, torch.tensor([audio.shape[1]]), torch.zeros(1))\nmodel_outputs = model.predict(\n  batch.to(device),\n  attribute_indexer.composition_feature_matrix(inventory).to(device)\n)\n```\n\nFinally, the log probabilities can be decoded into the recognized phonemes or phonetic features:\n\n```python\nfrom allophant import predictions\n\n# Create a feature mapping for your inventory and CTC decoders for the desired feature set\ninventory_indexer = attribute_indexer.attributes.subset(inventory)\nctc_decoders = predictions.feature_decoders(inventory_indexer, feature_names=supported_features)\n\nfor feature_name, decoder in ctc_decoders.items():\n    decoded = decoder(model_outputs.outputs[feature_name].transpose(1, 0), model_outputs.lengths)\n    # Print the feature name and values for each utterance in the batch\n    for [hypothesis] in decoded:\n        # NOTE: token indices are offset by one due to the \u003cBLANK\u003e token used during decoding\n        recognized = inventory_indexer.feature_values(feature_name, hypothesis.tokens - 1)\n        print(feature_name, recognized)\n```\n\n### Configuration\n\nTo specify options for preprocessing, training, and the model architecture, a configuration file in [TOML format](https://toml.io) can be passed to most commands.\nFor automation purposes, JSON configuration files can be used instead with the `--config-json-data/-j` flag.\nTo start, a default configuration file with comments can be generated as follows:\n\n```bash\nallophant generate-config [path/to/config]\n```\n\n### Preprocessing\n\nThe `allophant-data` command contains all functionality for corpus processing and management available in `allophant`.\nFor training, corpora without phoneme-level transcriptions have to be transcribed beforehand with a grapheme-to-phoneme model.\n\n#### Transcription\n\nPhoneme transcriptions for a supported corpus format can be generated with `transcribe`.\nFor instance, for transcribing the German and English subsets of a corpus with eSpeak NG and [PHOIBLE features](https://github.com/phoible/dev/tree/master/raw-data/FEATURES) from [Allophoible](https://github.com/Aariciah/allophoible) using a batch size of 512 and at most 15,000 utterances per language:\n\n```bash\nallophant-data transcribe -p -e espeak-ng -b 512 -l de,en -t 15000 -f phoible path/to/corpus -o transcribed_data\n```\n\nNote that no audio data is moved or copied in this process.\nAll commands that load corpora also accept a path to the `*.bin` transcription file directly instead of a directory. This allows loading only specific splits, such as loading only the `test` split for [evaluation](#evaluation).\n\n#### Utterance Lengths\n\nAs an optional step, utterance lengths can be extracted from a transcribed corpus for more memory efficient batching. If a subset of the corpus was transcribed, lengths will only be stored for the transcribed utterances.\n\n```bash\nallophant-data save-lengths [-c /path/to/config.toml] path/to/transcribed_corpus path/to/output\n```\n\n### Training\n\nDuring training, the best checkpoint is saved after each evaluation step to the path provided via the `--save-path/-s` flag. To save every checkpoint instead, a directory needs to be passed to `--save-path/-s` and the `--save-all/-a` flag included. The number of worker threads is auto-detected from the number of available CPU threads but can be set manually with `-w number`. To train only on the CPU instead of using CUDA, the `--cpu` flag can be used. Finally, any progress logging to stderr can be disabled with `--no-progress`.\n\n```bash\nallophant train [-c /path/to/config.toml] [-w number] [--cpu] [--no-progress] [--save-all]\n  [-s /path/to/checkpoint.pt] [-l /path/to/lengths] path/to/transcribed_corpus\n```\n\nNote that at least the `--lengths/-l` flag with a path to previously computed utterance lengths has to be specified when the \"frames\" batching mode is enabled.\n\n### Evaluation\n\n#### Test Data Inference\n\nFor evaluation, test data can be transcribed with the `predict` sub-command. The resulting file contains metadata, transcriptions for phonemes and features, and gold standard labels from the test data.\n\n```bash\nallophant predict [--cpu] [-w number] [-t {ucla-phonetic,common-voice}] [-f phonemes,feature1,feature2]\n  [--fix-unicode] [--training-languages {include,exclude,only}] [-m {frames,utterances}] [-s number]\n  [--language-phonemes] [--no-progress] [-c] [-o /path/to/prediction_file.jsonl] /path/to/dataset huggingface/model_id or /path/to/checkpoint\n```\n\nUse `--dataset-type/-t` to select the data set type. Note that only Common Voice and the UCLA Phonetic Corpus are currently supported. Predictions will either be printed to stdout or saved to a file given by `--output/-o`. Gzip compression is either inferred from a \".jsonl.gz\" extension or can be forced with the `--compress/-c` flag. The `--training-languages` argument allows filtering utterances based on the languages that also occur in the training data, and should be set to \"exclude\" for zero-shot evaluation.\n\nUsing `--feature-subset/-f`, a comma separated list of features or \"phoneme\" such as `syllabic,round,phoneme` can be provided to predict only the given subset of classes. With the `--fix-unicode` option, `predict` attempts to resolve issues of phonemes from the test data missing from the database due to differences in their unicode binary representation.\n\nThe batch sizes defined in the model configuration for training can be overwritten with the `--batch-size/-s` and `--batch-mode/-m`. Note that if the batch mode is set to \"utterance\" either in the model configuration or by setting the `--batch-mode/-m` flag, [utterance lengths](#utterance-lengths) have to be provided via the `--lengths/-l` argument. A beam size can be specified for CTC decoding with beam search (`--ctc-beam/-b`). We used a beam size of 1 for greedy decoding in our paper.\n\n#### Error Rates\n\nThe `evaluate` sub-command computes edit statistics and phoneme and attribute error rates for each language of a given corpus or split.\n\n```bash\nallophant evaluate [--fix-unicode] [--no-remap] [--split-complex] [-j] [--no-progress]\n  [-o path/to/results.json] [-d] path/to/predictions.jsonl\n```\n\nWithout `--no-remap`, transcriptions are mapped to language inventories using the same mapping scheme used during training. In our paper, all results were computed without this mapping, meaning that the transcriptions were directly compared to labels without an additional mapping step. If `--fix-unicode` was used during prediction, it should also be used in `evaluate`. Evaluation supports splitting any complex phoneme segments before computing error statistics with the `--split-complex/-s` flag.\n\nFor further analysis of evaluation results, JSON output should be enabled via the `--json/-j` flag. The JSON file can then be read using `allophant.evaluation.EvaluationResults`. For quick inspection of human-readable (average) error rates from evaluation results saved in JSON format, use `allophant-error-rates`:\n\n```bash\nallophant-error-rates path/to/results_file\n```\n\n### Allophoible Inventories\n\nInventories and feature sets preprocessed from (a subset of) [Allophoible](https://github.com/Aariciah/allophoible) for training can be extracted with the `allophant-features` command.\n\n```bash\nallophant-features [-p /path/to/allophoible.csv] [--remove-zero] [--prefer-allophant-dialects]\n  [-o /path/to/output.csv] [en,fr,ko,ar,...]\n```\n\n# Citation\n\nWhen using our work, please cite our paper as follows:\n\n```bibtex\n@inproceedings{glocker2023allophant,\n    title={Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes},\n    author={Glocker, Kevin and Herygers, Aaricia and Georges, Munir},\n    year={2023},\n    booktitle={{Proc. Interspeech 2023}},\n    month={8}}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkgnlp%2Fallophant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkgnlp%2Fallophant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkgnlp%2Fallophant/lists"}