https://github.com/bbuck/go-lexer
Lexer based on Rob Pike's talk on YouTube (view README)
https://github.com/bbuck/go-lexer
golang lexer
Last synced: 10 months ago
JSON representation
Lexer based on Rob Pike's talk on YouTube (view README)
- Host: GitHub
- URL: https://github.com/bbuck/go-lexer
- Owner: bbuck
- Created: 2015-05-30T08:06:52.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2022-05-12T14:29:49.000Z (almost 4 years ago)
- Last Synced: 2025-04-15T17:11:46.394Z (10 months ago)
- Topics: golang, lexer
- Language: Go
- Size: 7.81 KB
- Stars: 85
- Watchers: 7
- Forks: 22
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
This package provides a Lexer that functions similarly to Rob Pike's discussion
about lexer design in this [talk](https://www.youtube.com/watch?v=HxaD_trXwRE).
You can define your token types by using the `lexer.TokenType` type (`int`) via
```go
const (
StringToken lexer.TokenType = iota
IntegerToken
etc...
)
```
And then you define your own state functions (`lexer.StateFunc`) to handle
analyzing the string.
```go
func StringState(l *lexer.L) lexer.StateFunc {
l.Next() // eat starting "
l.Ignore() // drop current value
for l.Peek() != '"' {
l.Next()
}
l.Emit(StringToken)
return SomeStateFunction
}
```
It should be easy to make this Lexer consumable by a parser generated by go yacc doing something alone the lines of the following:
```go
type MyLexer struct {
lexer.L
}
func (m *MyLexer) Lex(lval *yySymType) int {
tok, done := m.NextToken()
if done {
return EOFToken
} else {
lval.val = tok.Value
return tok.Type
}
}
```
# License
MIT