https://github.com/dropdevrahul/golexer
A simple tokenizer library for parsing text or any text files
https://github.com/dropdevrahul/golexer
golang golang-library lexer tokenizer
Last synced: 7 months ago
JSON representation
A simple tokenizer library for parsing text or any text files
- Host: GitHub
- URL: https://github.com/dropdevrahul/golexer
- Owner: dropdevrahul
- Created: 2023-02-23T21:44:57.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2023-03-30T16:15:00.000Z (almost 3 years ago)
- Last Synced: 2025-05-17T17:39:49.703Z (9 months ago)
- Topics: golang, golang-library, lexer, tokenizer
- Language: Go
- Homepage:
- Size: 7.81 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
### Golexer
A simple golnag library/utility to tokenize a file into individual tokens usually for parsing scripts/programming languages.
Please note that this utility for now cannot be used to parse simple text files containing only plain text and is more appriopriate
for lexing syntax based languages/scripts.
### Usage
```
make build
./target/golexer path/to/file
```
You can also pipe stuff into the program
```
echo "some text" | ./target/golexer
```
### Example
A file like :
```
package main
import (
"fmt"
"log"
"os"
"github.com/dropdevrahul/golexer"
)
func main() {
usage := `Usage: golexer FILE`
//save := flag.Bool("c", false, "Save results to a file instead of printing")
if len(os.Args) < 2 {
fmt.Println(usage)
return
}
tz := golexer.NewTokenizer()
tokens, err := tz.LexFile(os.Args[1])
if err != nil {
log.Panic(err)
}
for _, t := range tokens {
fmt.Printf("%s\n", t.Value)
}
fmt.Println()
}
```
will be parse into
```
package
main
import
(
"fmt"
"log"
"os"
"github.com/dropdevrahul/golexer/golexer"
)
func
main
(
)
{
usage
:=
`Usage:
golexer
FILE`
//save
:=
flag.Bool
(
"c"
,
false
,
"Save results to a file instead of printing"
)
if
len
(
os.Args
)
<
2
{
fmt.Println
(
usage
)
return
}
tz
:=
golexer.NewTokenizer
(
)
tokens
,
err
:=
tz.LexFile
(
os.Args
[
1
]
)
if
err
!=
nil
{
log.Panic
(
err
)
}
for
_
,
t
:=
range
tokens
{
fmt.Printf
(
"%s\n"
,
t.Value
)
}
fmt.Println
(
)
}
```