https://github.com/antaripchatterjee/gentok

A simple tokenizer library created with c11
https://github.com/antaripchatterjee/gentok

Last synced: about 2 months ago
JSON representation

A simple tokenizer library created with c11

• Host: GitHub
• URL: https://github.com/antaripchatterjee/gentok
• Owner: antaripchatterjee
• License: mit
• Created: 2022-04-26T15:24:07.000Z (over 2 years ago)
• Default Branch: master
• Last Pushed: 2024-06-04T18:37:23.000Z (7 months ago)
• Last Synced: 2024-06-04T20:41:32.362Z (7 months ago)
• Language: C
• Size: 48.8 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# GenTok

A simple tokenizer library created with c11.

## Installation

If you are on windows platform, use a tool like [MSYS2](https://www.msys2.org/).

Use the below commands to build and release the gentok library.

```bash

# To build the debug version of the test program

make

# To build the release version of the test program

make release

# To build the static library

make install

# To clean up the working directory

make clean

```

## License

This library is released with the [MIT License](https://github.com/antaripchatterjee/gentok/blob/master/LICENSE).

## Note

This library is in its intial stage. It has a long way to go before I use it as a real tokenizer.