https://github.com/mdm-code/scanner
Custom Go text token scanner implementation.
https://github.com/mdm-code/scanner
go golang scan scanner token tokenizer
Last synced: 5 months ago
JSON representation
Custom Go text token scanner implementation.
- Host: GitHub
- URL: https://github.com/mdm-code/scanner
- Owner: mdm-code
- License: mit
- Created: 2023-09-24T20:48:00.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-01-25T20:32:52.000Z (over 2 years ago)
- Last Synced: 2024-01-26T21:42:29.018Z (over 2 years ago)
- Topics: go, golang, scan, scanner, token, tokenizer
- Language: Go
- Homepage: https://pkg.go.dev/github.com/mdm-code/scanner
- Size: 32.2 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Custom Go text token scanner implementation
Package `scanner` is a custom text scanner implementation. It has the same
idiomatic Go scanner programming interface, and it lets the client to freely
navigate the buffer. The scanner is also capable of peeking ahead of the
cursor. Read runes are rendered as tokens with additional information on their
position in the buffer. Consult the [package documentation](https://pkg.go.dev/github.com/mdm-code/scanner) or see
[Usage](#usage) to see how to use it.
## Installation
Use the following command to add the package to an existing project.
```sh
go get github.com/mdm-code/scanner
```
## Usage
Here is a snippet showing the basic usage of the scanner to read text as a stream
of tokens using the public API of the `scanner` package.
```go
package main
import (
"bufio"
"fmt"
"log"
"os"
"github.com/mdm-code/scanner"
)
func main() {
r := bufio.NewReader(os.Stdin)
s, err := scanner.New(r)
if err != nil {
log.Fatalln(err)
}
var ts []scanner.Token
for s.Scan() {
t := s.Token()
ts = append(ts, t)
}
fmt.Println(ts)
}
```
## Development
Consult [Makefile](Makefile) to see how to format, examine code with `go vet`,
run unit test, run code linter with `golint` in order to get test coverage and
check if the package builds all right.
Remember to install `golint` before you try to run tests and test the build:
```sh
go install golang.org/x/lint/golint@latest
```
## License
Copyright (c) 2023 Michał Adamczyk.
This project is licensed under the [MIT license](https://opensource.org/licenses/MIT).
See [LICENSE](LICENSE) for more details.