https://github.com/jackmott/rust-lexer

A port of Rob Pike's Go lexer to Rust
https://github.com/jackmott/rust-lexer

Last synced: 3 months ago
JSON representation

A port of Rob Pike's Go lexer to Rust

• Host: GitHub
• URL: https://github.com/jackmott/rust-lexer
• Owner: jackmott
• License: mit
• Created: 2019-11-25T16:02:02.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2019-11-26T17:07:42.000Z (over 6 years ago)
• Last Synced: 2025-04-08T17:50:14.987Z (about 1 year ago)
• Language: Rust
• Size: 5.86 KB
• Stars: 48
• Watchers: 3
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# A port of Rob Pike's Go lexer to Rust

Rob Pike has an excellent video showing how to write a nice lexical scanner in golang: https://www.youtube.com/watch?v=HxaD_trXwRE

This is a port of that approach to Rust. This is just a sample binary you can play with, not meant as a general purpose library. 

The lexer sends tokens on a channel so that a parser could consume them from a separate thread.  No string allocations are made, 

tokens refer to slices of the original input string.