https://github.com/paebbels/pytokenizer

A streaming tokenizer.
https://github.com/paebbels/pytokenizer

python3 streaming-parser streaming-tokenizer tokenizer

Last synced: 2 months ago
JSON representation

A streaming tokenizer.

• Host: GitHub
• URL: https://github.com/paebbels/pytokenizer
• Owner: Paebbels
• License: other
• Created: 2019-10-13T04:00:15.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2021-04-28T22:09:04.000Z (about 4 years ago)
• Last Synced: 2025-02-02T02:28:03.133Z (4 months ago)
• Topics: python3, streaming-parser, streaming-tokenizer, tokenizer
• Language: Python
• Homepage:
• Size: 40 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
[![Sourcecode on GitHub](https://img.shields.io/badge/Paebbels-pyTokenizer-323131.svg?logo=github&longCache=true)](https://github.com/Paebbels/pyTokenizer)

[![License](https://img.shields.io/badge/code%20license-Apache%20License%2C%202.0-lightgrey?logo=GitHub)](LICENSE.md)

[![GitHub tag (latest SemVer incl. pre-release)](https://img.shields.io/github/v/tag/Paebbels/pyTokenizer?logo=GitHub&include_prereleases)](https://github.com/Paebbels/pyTokenizer/tags)

[![GitHub release (latest SemVer incl. including pre-releases)](https://img.shields.io/github/v/release/Paebbels/pyTokenizer?logo=GitHub&include_prereleases)](https://github.com/Paebbels/pyTokenizer/releases/latest)

[![GitHub release date](https://img.shields.io/github/release-date/Paebbels/pyTokenizer?logo=GitHub&)](https://github.com/Paebbels/pyTokenizer/releases)  

[![GitHub Workflow Status](https://img.shields.io/github/workflow/status/Paebbels/pyTokenizer/Test,%20Coverage%20and%20Release?label=Workflow&logo=GitHub)](https://github.com/Paebbels/pyTokenizer/actions?query=workflow%3A%22Test%2C+Coverage+and+Release%22)

[![PyPI](https://img.shields.io/pypi/v/pyTokenizer?logo=PyPI)](https://pypi.org/project/pyTokenizer/)

![PyPI - Status](https://img.shields.io/pypi/status/pyTokenizer?logo=PyPI)

![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pyTokenizer?logo=PyPI)

[![Dependent repos (via libraries.io)](https://img.shields.io/librariesio/dependent-repos/pypi/pyTokenizer)](https://github.com/Paebbels/pyTokenizer/network/dependents)  

[![Libraries.io status for latest release](https://img.shields.io/librariesio/release/pypi/pyTokenizer)](https://libraries.io/github/Paebbels/pyTokenizer)

[![Requires.io](https://img.shields.io/requires/github/Paebbels/pyTokenizer)](https://requires.io/github/Paebbels/pyTokenizer/requirements/?branch=master)  

[![Codacy - Quality](https://img.shields.io/codacy/grade/7ea6c9eea5b94493837ea55c07de3562?logo=Codacy)](https://www.codacy.com/manual/Paebbels/pyTokenizer)

[![Codacy - Coverage](https://img.shields.io/codacy/coverage/7ea6c9eea5b94493837ea55c07de3562?logo=Codacy)](https://www.codacy.com/manual/Paebbels/pyTokenizer)

[![Codecov - Branch Coverage](https://img.shields.io/codecov/c/github/Paebbels/pyTokenizer?logo=Codecov)](https://codecov.io/gh/Paebbels/pyTokenizer)

[![Libraries.io SourceRank](https://img.shields.io/librariesio/sourcerank/pypi/pyTokenizer)](https://libraries.io/github/Paebbels/pyTokenizer/sourcerank)  

[![Read the Docs](https://img.shields.io/readthedocs/pytokenizer)](https://pyTokenizer.readthedocs.io/en/latest/)

# pyTokenizer

A streaming tokenizer.

## Contributors

* [Patrick Lehmann](https://github.com/Paebbels) (Maintainer)

## License

This Python package (source code) is licensed under [Apache License 2.0](LICENSE.md).

-------------------------

SPDX-License-Identifier: Apache-2.0