{"id":13490418,"url":"https://github.com/WojciechMula/pyahocorasick","last_synced_at":"2025-03-28T06:31:16.092Z","repository":{"id":37706093,"uuid":"10390556","full_name":"WojciechMula/pyahocorasick","owner":"WojciechMula","description":"Python module (C extension and plain python) implementing Aho-Corasick algorithm","archived":false,"fork":false,"pushed_at":"2024-03-21T10:54:32.000Z","size":745,"stargazers_count":981,"open_issues_count":32,"forks_count":127,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-23T15:16:02.118Z","etag":null,"topics":["aho-corasick","automaton","string-manipulation","trie"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WojciechMula.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-05-30T19:55:46.000Z","updated_at":"2025-03-23T08:11:16.000Z","dependencies_parsed_at":"2023-02-09T20:15:58.297Z","dependency_job_id":"0c282cc1-347f-42b0-a1ef-788c494993d0","html_url":"https://github.com/WojciechMula/pyahocorasick","commit_stats":{"total_commits":458,"total_committers":30,"mean_commits":"15.266666666666667","dds":0.4454148471615721,"last_synced_commit":"c98e999c7a5dcbcb8c16380a0ff96afb15065b17"},"previous_names":[],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WojciechMula%2Fpyahocorasick","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WojciechMula%2Fpyahocorasick/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WojciechMula%2Fpyahocorasick/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WojciechMula%2Fpyahocorasick/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WojciechMula","download_url":"https://codeload.github.com/WojciechMula/pyahocorasick/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245984423,"owners_count":20704791,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aho-corasick","automaton","string-manipulation","trie"],"created_at":"2024-07-31T19:00:46.402Z","updated_at":"2025-03-28T06:31:14.011Z","avatar_url":"https://github.com/WojciechMula.png","language":"C","funding_links":[],"categories":["C","文本数据和NLP","Feature Extraction","Text Data"],"sub_categories":["Text/NLP"],"readme":"========================================================================\n                          pyahocorasick\n========================================================================\n\n\n|build-ghactions| |docs|\n\n\n**pyahocorasick** is a fast and memory efficient library for exact or approximate\nmulti-pattern string search meaning that you can find multiple key strings\noccurrences at once in some input text.  The strings \"index\" can be built ahead\nof time and saved (as a pickle) to disk to reload and reuse later.  The library\nprovides an `ahocorasick` Python module that you can use as a plain dict-like\nTrie or convert a Trie to an automaton for efficient Aho-Corasick search.\n\n**pyahocorasick** is implemented in C and tested on Python 3.8 and up.\nIt works on 64 bits Linux, macOS and Windows.\n\nThe license_ is BSD-3-Clause. Some utilities, such as tests and the pure Python\nautomaton are dedicated to the Public Domain.\n\n\nTestimonials\n=============\n\n`Many thanks for this package. Wasn't sure where to leave a thank you note but\nthis package is absolutely fantastic in our application where we have a library\nof 100k+ CRISPR guides that we have to count in a stream of millions of DNA\nsequencing reads. This package does it faster than the previous C program we\nused for the purpose and helps us stick to just Python code in our pipeline.`\n\nMiika (AstraZeneca Functional Genomics Centre)\nhttps://github.com/WojciechMula/pyahocorasick/issues/145\n\n\nDownload and source code\n========================\n\nYou can fetch **pyahocorasick** from:\n\n- GitHub https://github.com/WojciechMula/pyahocorasick/\n- Pypi https://pypi.python.org/pypi/pyahocorasick/\n- Conda-Forge https://github.com/conda-forge/pyahocorasick-feedstock/\n\nThe **documentation** is published at https://pyahocorasick.readthedocs.io/\n\n\nQuick start\n===========\n\nThis module is written in C. You need a C compiler installed to compile native\nCPython extensions. To install::\n\n    pip install pyahocorasick\n\nThen create an Automaton::\n\n    \u003e\u003e\u003e import ahocorasick\n    \u003e\u003e\u003e automaton = ahocorasick.Automaton()\n\nYou can use the Automaton class as a trie. Add some string keys and their associated\nvalue to this trie. Here we associate a tuple of (insertion index, original string)\nas a value to each key string we add to the trie::\n\n    \u003e\u003e\u003e for idx, key in enumerate('he her hers she'.split()):\n    ...   automaton.add_word(key, (idx, key))\n\nThen check if some string exists in the trie::\n\n    \u003e\u003e\u003e 'he' in automaton\n    True\n    \u003e\u003e\u003e 'HER' in automaton\n    False\n\nAnd play with the ``get()`` dict-like method::\n\n    \u003e\u003e\u003e automaton.get('he')\n    (0, 'he')\n    \u003e\u003e\u003e automaton.get('she')\n    (3, 'she')\n    \u003e\u003e\u003e automaton.get('cat', 'not exists')\n    'not exists'\n    \u003e\u003e\u003e automaton.get('dog')\n    Traceback (most recent call last):\n      File \"\u003cstdin\u003e\", line 1, in \u003cmodule\u003e\n    KeyError\n\nNow convert the trie to an Aho-Corasick automaton to enable Aho-Corasick search::\n\n    \u003e\u003e\u003e automaton.make_automaton()\n\nThen search all occurrences of the keys (the needles) in an input string (our haystack).\n\nHere we print the results and just check that they are correct. The\n`Automaton.iter()` method return the results as two-tuples of the `end index` where a\ntrie key was found in the input string and the associated `value` for this key. Here\nwe had stored as values a tuple with the original string and its trie insertion\norder::\n\n    \u003e\u003e\u003e for end_index, (insert_order, original_value) in automaton.iter(haystack):\n    ...     start_index = end_index - len(original_value) + 1\n    ...     print((start_index, end_index, (insert_order, original_value)))\n    ...     assert haystack[start_index:start_index + len(original_value)] == original_value\n    ...\n    (1, 2, (0, 'he'))\n    (1, 3, (1, 'her'))\n    (1, 4, (2, 'hers'))\n    (4, 6, (3, 'she'))\n    (5, 6, (0, 'he'))\n\nYou can also create an eventually large automaton ahead of time and `pickle` it to\nre-load later. Here we just pickle to a string. You would typically pickle to a\nfile instead::\n\n    \u003e\u003e\u003e import pickle\n    \u003e\u003e\u003e pickled = pickle.dumps(automaton)\n    \u003e\u003e\u003e B = pickle.loads(pickled)\n    \u003e\u003e\u003e B.get('he')\n    (0, 'he')\n\n\nSee also:\n\n* FAQ and Who is using pyahocorasick? \n  https://github.com/WojciechMula/pyahocorasick/wiki/FAQ#who-is-using-pyahocorasick\n\n\nDocumentation\n=============\n\nThe full documentation including the API overview and reference is published on\n`readthedocs \u003chttp://pyahocorasick.readthedocs.io/\u003e`_.\n\n\nOverview\n\nWith an `Aho-Corasick automaton \u003chttp://en.wikipedia.org/wiki/Aho-Corasick%20algorithm\u003e`_\nyou can efficiently search all occurrences of multiple strings (the needles) in an\ninput string (the haystack) making a single pass over the input string. With\npyahocorasick you can eventually build large automatons and pickle them to reuse\nthem over and over as an indexed structure for fast multi pattern string matching.\n\nOne of the advantages of an Aho-Corasick automaton is that the typical worst-case\nand best-case **runtimes** are about the same and depends primarily on the size\nof the input string and secondarily on the number of matches returned.  While\nthis may not be the fastest string search algorithm in all cases, it can search\nfor multiple strings at once and its runtime guarantees make it rather unique.\nBecause pyahocorasick is based on a Trie, it stores redundant keys prefixes only\nonce using memory efficiently.\n\nA drawback is that it needs to be constructed and \"finalized\" ahead of time\nbefore you can search strings. In several applications where you search for\nseveral pre-defined \"needles\" in a variable \"haystacks\" this is actually an\nadvantage.\n\n**Aho-Corasick automatons** are commonly used for fast multi-pattern matching\nin intrusion detection systems (such as snort), anti-viruses and many other\napplications that need fast matching against a pre-defined set of string keys.\n\nInternally an Aho-Corasick automaton is typically based on a Trie with extra\ndata for failure links and an implementation of the Aho-Corasick search\nprocedure.\n\nBehind the scenes the **pyahocorasick** Python library implements these two data\nstructures:  a `Trie \u003chttp://en.wikipedia.org/wiki/trie\u003e`_ and an Aho-Corasick\nstring matching automaton. Both are exposed through the `Automaton` class.\n\nIn addition to Trie-like and Aho-Corasick methods and data structures,\n**pyahocorasick** also implements dict-like methods: The pyahocorasick\n**Automaton** is a **Trie** a dict-like structure indexed by string keys each\nassociated with a value object. You can use this to retrieve an associated value\nin a time proportional to a string key length.\n\npyahocorasick is available in two flavors:\n\n* a CPython **C-based extension**, compatible with Python 3 only. Use older\n  version 1.4.x for Python 2.7.x and 32 bits support.\n\n* a simpler pure Python module, compatible with Python 2 and 3. This is only\n  available in the source repository (not on Pypi) under the etc/py/ directory\n  and has a slightly different API.\n\n\nUnicode and bytes\n-----------------\n\nThe type of strings accepted and returned by ``Automaton`` methods are either\n**unicode** or **bytes**, depending on a compile time settings (preprocessor\ndefinition of ``AHOCORASICK_UNICODE`` as set in `setup.py`).\n\nThe ``Automaton.unicode`` attributes can tell you how the library was built.\nOn Python 3, unicode is the default.\n\n\n.. warning::\n\n    When the library is built with unicode support, an Automaton will store 2 or\n    4 bytes per letter, depending on your Python installation. When built\n    for bytes, only one byte per letter is needed.\n\n\nBuild and install from PyPi\n===========================\n\nTo install for common operating systems, use pip. Pre-built wheels should be\navailable on Pypi at some point in the future::\n\n    pip install pyahocorasick\n\nTo build from sources you need to have a C compiler installed and configured which\nshould be standard on Linux and easy to get on MacOSX.\n\nTo build from sources, clone the git repository or download and extract the source\narchive.\n\nInstall `pip` (and its `setuptools` companion) and then run (in a `virtualenv` of\ncourse!)::\n\n    pip install .\n\nIf compilation succeeds, the module is ready to use.\n\n\nSupport\n=======\n\nSupport is available through the `GitHub issue tracker\n\u003chttps://github.com/WojciechMula/pyahocorasick/issues\u003e`_ to report bugs or ask\nquestions.\n\n\nContributing\n============\n\nYou can submit contributions through `GitHub pull requests\n\u003chttps://github.com/WojciechMula/pyahocorasick/pull\u003e`_.\n\n- There is a Makefile with a default target that builds and runs tests.\n- The tests can run with a `pip installe -e .[testing] \u0026\u0026 pytest -vvs`\n- See also the .github directory for CI tests and workflow\n\n\nAuthors\n=======\n\nThe initial author and maintainer is Wojciech Muła. `Philippe Ombredanne\n\u003chttps://github.com/pombredanne\u003e`_ is Wojciech's sidekick and helps maintaining,\nand rewrote documentation, setup CI servers and did a some work to make this\nmodule more accessible to end users.\n\nAlphabetic list of authors and contributors:\n\n* **Andrew Grigorev**\n* **Ayan Mahapatra**\n* **Bogdan**\n* **David Woakes**\n* **Edward Betts**\n* **Frankie Robertson**\n* **Frederik Petersen**\n* **gladtosee**\n* **INADA Naoki**\n* **Jan Fan**\n* **Pastafarianist**\n* **Philippe Ombredanne**\n* **Renat Nasyrov**\n* **Sylvain Zimmer**\n* **Xiaopeng Xu**\n\nand many others!\n\nThis library would not be possible without help of many people, who contributed in\nvarious ways.\nThey created `pull requests \u003chttps://github.com/WojciechMula/pyahocorasick/pull\u003e`_,\nreported bugs as `GitHub issues \u003chttps://github.com/WojciechMula/pyahocorasick/issues\u003e`_\nor via direct messages, proposed fixes, or spent their valuable time on testing.\n\nThank you.\n\n\nLicense\n=======\n\nThis library is licensed under very liberal\n`BSD-3-Clause \u003chttp://spdx.org/licenses/BSD-3-Clause.html\u003e`_ license. Some\nportions of the code are dedicated to the public domain such as the pure Python\nautomaton and test code.\n\nFull text of license is available in LICENSE file.\n\n\nOther Aho-Corasick implementations for Python you can consider\n==============================================================\n\nWhile **pyahocorasick** tries to be the finest and fastest Aho Corasick library\nfor Python you may consider these other libraries:\n\n\n* `py_aho_corasick \u003chttps://github.com/JanFan/py-aho-corasick\u003e`_ by Jan\n\n * Written in pure Python.\n * Poor performance.\n\n* `ahocorapy \u003chttps://github.com/abusix/ahocorapy\u003e`_ by abusix\n\n * Written in pure Python.\n * Better performance than py-aho-corasick.\n * Using pypy, ahocorapy's search performance is only slightly worse than pyahocorasick's.\n * Performs additional suffix shortcutting (more setup overhead, less search overhead for suffix lookups).\n * Includes visualization tool for resulting automaton (using pygraphviz).\n * MIT-licensed, 100% test coverage, tested on all major python versions (+ pypy)\n\n* `noaho \u003chttps://github.com/JDonner/NoAho\u003e`_ by Jeff Donner\n\n * Written in C. Does not return overlapping matches.\n * Does not compile on Windows (July 2016).\n * No support for the pickle protocol.\n\n* `acora \u003chttps://github.com/scoder/acora\u003e`_ by Stefan Behnel\n\n * Written in Cython.\n * Large automaton may take a long time to build (July 2016)\n * No support for a dict-like protocol to associate a value to a string key.\n\n* `ahocorasick \u003chttps://hkn.eecs.berkeley.edu/~dyoo/python/ahocorasick/\u003e`_ by Danny Yoo\n\n * Written in C.\n * seems unmaintained (last update in 2005).\n * GPL-licensed.\n\n\n.. |build-ghactions| image:: https://github.com/WojciechMula/pyahocorasick/actions/workflows/test-and-build.yml/badge.svg\n   :target: https://github.com/WojciechMula/pyahocorasick/actions/workflows/test-and-build.yml\n   :alt: GitHub Action build on test -  Master branch status\n\n.. |docs| image:: https://readthedocs.org/projects/pyahocorasick/badge/?version=latest\n   :alt: Documentation Status\n   :target: https://pyahocorasick.readthedocs.io/en/latest/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWojciechMula%2Fpyahocorasick","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FWojciechMula%2Fpyahocorasick","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWojciechMula%2Fpyahocorasick/lists"}