https://github.com/dsal3389/tokenizer
generic code tokenizer
https://github.com/dsal3389/tokenizer
c token tokenizer
Last synced: 18 days ago
JSON representation
generic code tokenizer
- Host: GitHub
- URL: https://github.com/dsal3389/tokenizer
- Owner: dsal3389
- Created: 2023-08-27T13:53:52.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2023-08-27T14:08:58.000Z (almost 3 years ago)
- Last Synced: 2025-03-03T12:44:36.462Z (over 1 year ago)
- Topics: c, token, tokenizer
- Language: C
- Homepage:
- Size: 2.93 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# tokenizer
a simple generic code tokenizer written in C, the tokenizer doesn't support specific language,
it will just group words and strings
## build
```sh
make build
```
## example
```py
# test.py
print("hello world")
def foo() -> int:
return 0
```
output for `tokenizer test.py -gs`
```sh
(NEWLINE)
print
(
"hello world"
)
(NEWLINE)
(NEWLINE)
(NEWLINE)
#
(SPACE)
comment
(NEWLINE)
def
(SPACE)
foo
(
)
(SPACE)
-
>
(SPACE)
int
:
(NEWLINE)
(SPACEx4)
return
(SPACE)
0
```