https://github.com/tomdionysus/stringtree-python

Fast forward-only tokenizer in python
https://github.com/tomdionysus/stringtree-python

Last synced: about 2 months ago
JSON representation

Fast forward-only tokenizer in python

• Host: GitHub
• URL: https://github.com/tomdionysus/stringtree-python
• Owner: tomdionysus
• Created: 2015-08-28T03:35:04.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2015-09-05T07:28:40.000Z (over 9 years ago)
• Last Synced: 2024-04-12T16:08:55.559Z (9 months ago)
• Language: Python
• Size: 1.57 MB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# stringtree-python

[![Build Status](https://travis-ci.org/tomdionysus/stringtree-python.svg?branch=master)](https://travis-ci.org/tomdionysus/stringtree-python)

[![Coverage Status](https://coveralls.io/repos/tomdionysus/stringtree-python/badge.svg?branch=master&service=github)](https://coveralls.io/github/tomdionysus/stringtree-python?branch=master)

StringTree is a fast forward-only tokeniser and partial string matcher, that is, it can:

* Load a dictionary of arbitarty size and record count - e.g. an actual dictionary, an english word list - where each record is associated with a key - e.g. a numeric identifier for the word.

* Parse an arbitary data string in a single pass, finding and storing instances of each item in the dictionary and storing their offsets and associated keys.

* Host a set of strings in such a way as to efficiently match partial input strings against the dictionary

This has become my 'hello world' over the years with any new language. I use it to get to know a language, as implementing it correctly involves many of the usual concepts needed get started coding from the hip (syntax, grammar, classes, public/private instance vars, statics, pass-by value/pass-by-reference etc.) not to mention usual code support skills like how to set up unit tests for this language and environment, etc.

## Testing

    PYTHONPATH=./ py.test

## Demo

    PYTHONPATH=./ python ./examples/demo.py

## Code of Conduct

The StringTree project is committed to the [Contributor Covenant](http://contributor-covenant.org). Please read [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) before making any contributions or comments.

## References

* http://docs.python.org

* http://en.wikipedia.org/wiki/Trie