{"id":13482451,"url":"https://github.com/proycon/colibri-core","last_synced_at":"2025-04-12T16:37:54.694Z","repository":{"id":48325710,"uuid":"12996232","full_name":"proycon/colibri-core","owner":"proycon","description":"Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi      ch allows you to build, view, manipulate and query pattern models. ","archived":false,"fork":false,"pushed_at":"2024-12-17T12:52:34.000Z","size":10733,"stargazers_count":126,"open_issues_count":9,"forks_count":20,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-10T16:38:38.099Z","etag":null,"topics":["c-plus-plus","computational-linguistics","corpus","library","linguistics","ngram","ngrams","nlp","pattern-recognition","python","skipgram","text-processing"],"latest_commit_sha":null,"homepage":"https://proycon.github.io/colibri-core","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proycon.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":"codemeta.json","zenodo":null}},"created_at":"2013-09-21T14:12:05.000Z","updated_at":"2025-02-22T08:04:59.000Z","dependencies_parsed_at":"2022-08-26T02:41:27.636Z","dependency_job_id":"eb47c707-23bf-4728-a454-daa10e086d23","html_url":"https://github.com/proycon/colibri-core","commit_stats":{"total_commits":1254,"total_committers":6,"mean_commits":209.0,"dds":0.02472089314194581,"last_synced_commit":"54e235f76975e28a73a165551429f379dd6f4af2"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibri-core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibri-core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibri-core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibri-core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proycon","download_url":"https://codeload.github.com/proycon/colibri-core/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248597699,"owners_count":21130927,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c-plus-plus","computational-linguistics","corpus","library","linguistics","ngram","ngrams","nlp","pattern-recognition","python","skipgram","text-processing"],"created_at":"2024-07-31T17:01:02.124Z","updated_at":"2025-04-12T16:37:54.661Z","avatar_url":"https://github.com/proycon.png","language":"C++","funding_links":[],"categories":["C++","函式庫","Packages"],"sub_categories":["Tools","[Tools](#tools-1)","Speech Recognition","書籍","Libraries"],"readme":"# Colibri Core\n\n[![GitHub C++ build](https://github.com/proycon/colibri-core/actions/workflows/colibri-core.yml/badge.svg?branch=master)](https://github.com/proycon/colibri-core/actions/)\n[![GitHub Python build](https://github.com/proycon/colibri-core/actions/workflows/colibri-core-python.yml/badge.svg?branch=master)](https://github.com/proycon/colibri-core/actions/)\n[![DOI](https://zenodo.org/badge/12996232.svg)](https://zenodo.org/badge/latestdoi/12996232)\n[![GitHub release](https://img.shields.io/github/release/proycon/colibri-core.svg)](https://GitHub.com/proycon/colibri-core/releases/)\n[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)\n[![PyPi](https://badge.fury.io/py/colibricore.svg)](https://pypi.org/colibricore)\n\n*by Maarten van Gompel, proycon@anaproy.nl, Radboud University Nijmegen*\n\n*Licensed under GPLv3 (See http://www.gnu.org/licenses/gpl-3.0.html)*\n\nColibri Core is software to quickly and efficiently count and extract patterns\nfrom large corpus data, to extract various statistics on the extracted\npatterns, and to compute relations between the extracted patterns. The employed\nnotion of pattern or construction encompasses the following categories:\n\n* **n-gram** -- *n* consecutive words\n* **skipgram** -- An abstract pattern of predetermined length with one or multiple gaps (of specific size).\n* **flexgram** -- An abstract pattern with one or more gaps of variable-size.\n\nN-gram extraction may seem fairly trivial at first, with a few lines in your\nfavourite scripting language, you can move a simple sliding window of size **n**\nover your corpus and store the results in some kind of hashmap. This trivial\napproach however makes an unnecessarily high demand on memory resources, this\noften becomes prohibitive if unleashed on large corpora. Colibri Core tries to\nminimise these space requirements in several ways:\n\n* **Compressed binary representation** -- Each word type is assigned a numeric class, which is encoded in a compact binary format in which highly frequent classes take less space than less frequent classes. Colibri core always uses this representation rather than a full string representation, both on disk and in memory.\n* **Informed iterative counting** -- Counting is performed more intelligently by iteratively processing the corpus in several passes and quickly discarding patterns that won't reach the desired occurrence threshold.\n\nSkipgram and flexgram extraction are computationally more demanding but have\nbeen implemented with similar optimisations. Skipgrams are computed by\nabstracting over n-grams, and flexgrams in turn are computed either by\nabstracting over skipgrams, or directly from n-grams on the basis of\nco-occurrence information (mutual pointwise information).\n\nAt the heart of the sofware is the notion of pattern models. The core tool, to\nbe used from the command-line, is ``colibri-patternmodeller`` which enables you\nto build pattern models, generate statistical reports, query for specific\npatterns and relations, and manipulate models.\n\nA pattern model is simply a collection of extracted patterns (any of the three\ncategories) and their counts from a specific corpus. Pattern models come in two\nvarieties:\n\n* **Unindexed Pattern Model** -- The simplest form, which simply stores the patterns and their count.\n* **Indexed Pattern Model** -- The more informed form, which retains all indices to the original corpus, at the cost of more memory/diskspace.\n\nThe Indexed Pattern Model is much more powerful, and allows more statistics and\nrelations to be inferred.\n\nThe generation of pattern models is optionally parametrised by a minimum\noccurrence threshold, a maximum pattern length, and a lower-boundary on the\ndifferent types that may instantiate a skipgram (i.e. possible fillings of the\ngaps).\n\n## Technical Details\n\nColibri Core is available as a collection of **standalone command-line tools**,\nas a **C++ library**, and as a **Python library**.\n\nPlease consult the full documentation at \u003chttps://proycon.github.io/colibri-core\u003e\n\n## Installation\n\n### Python binding\n\nFor the Colibri Core Python library, just install using:\n\n```\npip install colibricore\n```\n\nWe strongly recommend you use a Virtual Environment for this. Do note that this\nis only available for unix-like systems, Windows is not supported.\n\n### Installation from packages\n\nFor the command-line tools, check if your distribution has a package available.\nThere are packages for Alpine Linux (`apk add colibri-core`) and for macOS with\nhomebrew (`brew tap fbkarsdorp/homebrew-lamachine \u0026\u0026 brew install\ncolibri-core`). Note that these do not contain the Python binding!\n\nOtherwise you will need to either use the container image or to build and install from source.\n\n### Installation from source\n\nIf no packages are available, you will need to compile from source or use the container build (e.g.\nDocker) as explained later on.\n\nIn order to do so, you need a sane build environment, install the necessary dependencies for your distribution:\n\nFor Debian/Ubuntu::\n\n```\n$ sudo apt-get install make gcc g++ pkg-config autoconf-archive libtool autotools-dev libbz2-dev zlib1g-dev libtar-dev python3 python3-dev cython3\n```\n\nFor RedHat-based systems (run as root)::\n\n```\n# yum install pkgconfig libtool autoconf automake autoconf-archive make gcc gcc-c++ libtar libtar-devel python3 python3-devel zlib zlib-devel python3-pip bzip2 bzip2-devel cython3\n```\n\nFor macOS with homebrew:\n\n```\n$ brew install autoconf automake libtool autoconf-archive python3 pkg-config\n```\n\nThen clone this repository and install as follows:\n\n```\n$ bash bootstrap\n$ ./configure\n$ make\n$ sudo make install\n```\n\n### Container usage\n\nThe Colibri Core command-line tools are also available as an OCI/Docker container.\n\nA pre-made container image can be obtained from Docker Hub as follows:\n\n``docker pull proycon/colibri-core``\n\nYou can also build a container image yourself as follows, make sure you are in the root of this repository:\n\n``docker build -t proycon/colibri-core .``\n\nThis builds the latest stable release, if you want to use the latest development version\nfrom the git repository instead, do:\n\n``docker build -t proycon/colibri-core --build-arg VERSION=development .``\n\nRun the frog container interactively as follows, it will dump you into a shell where the various command line tools are available:\n\n``docker run -t -i proycon/colibri-core``\n\nAdd the ``-v /path/to/your/data:/data`` parameter if you want to mount your data volume into the container at `/data`.\n\n## Demo\n\n![Colibri Core Demo](https://raw.githubusercontent.com/CLARIAH/wp3-demos/master/colibri-core.gif)\n\n\n## Publication\n\nThis software is extensively described in the following peer-reviewed publication:\n\n    van Gompel, M and van den Bosch, A (2016)\n    Efficient n-gram, Skipgram and Flexgram Modelling with Colibri Core.\n    *Journal of Open Research Software*\n    4: e30, DOI: http://dx.doi.org/10.5334/jors.105\n\nAccess the publication [here](http://dx.doi.org/10.5334/jors.105) and please cite it if you make use of\nColibri Core in your work.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fcolibri-core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproycon%2Fcolibri-core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fcolibri-core/lists"}