https://github.com/aymanizz/python-tokenizer
A tokenizer for the python programming language in C
https://github.com/aymanizz/python-tokenizer
c python tokenizer
Last synced: about 2 months ago
JSON representation
A tokenizer for the python programming language in C
- Host: GitHub
- URL: https://github.com/aymanizz/python-tokenizer
- Owner: aymanizz
- Created: 2018-10-31T21:55:15.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-01-09T21:59:15.000Z (over 7 years ago)
- Last Synced: 2025-07-12T00:33:27.887Z (12 months ago)
- Topics: c, python, tokenizer
- Language: C
- Size: 36.1 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# A Python Tokenizer
A tokenizer for the python programming language in C.
## Differences
There are a couple of differences between this tokenizer and what cpython
tokenizer does.
This tokenizer:
- **does not** emit NL.
- **does not** recognize encoding comment.
- **does not** support unicode.
- **does** emit tokens for each reserved keyword.
I plan on adding support for some of these but not all. The implementation will
differ from the cpython's in many ways.
## Building And Testing
1. Clone the repository and initialize the submodules.
```
git clone --recurse-submodules https://github.com/aymanizz/python-tokenizer.git
```
2. Run make. The following make commands (targets) are available:
- build the tokenizer: `make`.
- build the tests runner and run it: `make test`.
- remove the binaries directory: `make clean`.
Ouput executable files can be found in `bin` directory after building.