{"id":46095958,"url":"https://github.com/zalgonoise/lex","last_synced_at":"2026-03-01T18:36:53.862Z","repository":{"id":65145919,"uuid":"582171306","full_name":"zalgonoise/lex","owner":"zalgonoise","description":"a generic lexer library written in Go","archived":false,"fork":false,"pushed_at":"2023-02-10T13:37:48.000Z","size":100,"stargazers_count":18,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-06-21T18:06:52.795Z","etag":null,"topics":["generic","go","golang","lex","lexemes","lexer"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zalgonoise.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-26T00:51:50.000Z","updated_at":"2024-04-04T16:26:02.000Z","dependencies_parsed_at":"2023-02-19T02:45:24.375Z","dependency_job_id":null,"html_url":"https://github.com/zalgonoise/lex","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zalgonoise/lex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalgonoise%2Flex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalgonoise%2Flex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalgonoise%2Flex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalgonoise%2Flex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zalgonoise","download_url":"https://codeload.github.com/zalgonoise/lex/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalgonoise%2Flex/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29979120,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T16:35:47.903Z","status":"ssl_error","status_checked_at":"2026-03-01T16:35:44.899Z","response_time":124,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generic","go","golang","lex","lexemes","lexer"],"created_at":"2026-03-01T18:36:53.754Z","updated_at":"2026-03-01T18:36:53.846Z","avatar_url":"https://github.com/zalgonoise.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lex\n\n*a generic lexer library written in Go*\n\n_______________\n\n## Concept\n\n`lex` is a lexer library for Go, based on the concept of the [`text/template`](https://pkg.go.dev/text/template) lexer, as a generic implementation. The logic behind this lexer is mostly based off of [Rob Pike](https://github.com/robpike)'s talk about [Lexical Scanning in Go](https://www.youtube.com/watch?v=HxaD_trXwRE), which is also seen in the standard library (in [`text/template/parse/lex.go`](https://cs.opensource.google/go/go/+/refs/tags/go1.19.4:src/text/template/parse/lex.go)).\n\nThe lexer is a state-machine that analyzes input data (split into single units) by traversing the slice and classifying *blobs* of the slice (as lexemes) with a certain token. The lexer emits items, which are composed of three main elements:\n- the starting position in the slice where the lexeme starts\n- the (comparable type) token classifying this piece of data\n- a slice containing the actual data \n\nThe lexer is usually in-sync with a parser (also a state-machine, running in tandem with a lexer), that will consume the items emited by the lexer to build a parse tree. The parse tree, as the name implies, is a tree graph data structure that will layout the received tokens with a certain path / structure, with some logic in mind. It is finally able to output the processed tree as an output type, configurable by the developer, too.\n\n## Why generics?\n\nGenerics are great for when there is a solid algorithm that serves for many types, and can be abstracted enough to work without major workarounds; and this approach to a lexer / parser is very straight-forward and yet so simple (the Go way). Of course when I refer [Rob Pike](https://github.com/robpike)'s talk about lexers I am aware that the context is parsing text (for templating). The *approach with generics* will limit the potential that shines in the original implementation, one way or the other (simply with EOF being a zero value, for example -- zero types should not be used for this).\n\nBut all in all, it was a great exercise to practice using generics. Maybe I will just use this library once or twice, maybe it will be actually useful for some. I am just in it for the ride. :)\n\n## Overview\n\nThe idea behind implementing a generic algorithm for a lexer came from trying to build a graph (data structure) representing the logic blocks in a Go file. Watching the talk above was a breath of fresh air when it came to the design of the lexer and its simple approach. So, it would be nice to leverage this algorithm for the Go code graph idea from before. By making the logic generic, one could implement an `Item` type to hold a defined token type, and a set of (any type of) values and `StateFn` state-functions to tokenize input data. In concept this works for any given type, as the point is to label elements of a slice with identifying tokens, that will be processed into a parse tree (with a specific parser implementation).\n\nCaveats are precisely using very *open* types for this implementation. The `text/template` lexer will, for example, define its EOF token as `-1` -- a constant found in the [`lex.go` file](https://cs.opensource.google/go/go/+/refs/tags/go1.19.4:src/text/template/parse/lex.go;l=93). For this implementation, the lexer will return a zero-value token, so the caller should prepare their token types considering that the zero value will be reserved for EOF. Scrolling through the input will use the `pos int` position, and will not have a width -- because the lexer will consume the input as a list of the defined data type. Concerns about the width of a unit need to be handled in the types or `StateFn` implementation, not the Lexer.\n\nAdditionally, as it is exposed as an interface (which is not set in stone, though), it introduces a few helper methods to either validate input data, navigate through the index, and controlling the cursor of the slice. It is implementing the [`cur.Cursor[T any] interface`](https://github.com/zalgonoise/cur).\n\n## Installation \n\n\u003e Note: this library is not ready out-of-the box! You will need to implement your own `StateFn` state-functions with defined types. This repo will expose simple examples to understand the flow of the lexer, below.\n\nYou can add this library to your Go project with a `go get` command:\n\n```\ngo get github.com/zalgonoise/lex\n```\n\n## Features\n\n### Entities\n\n#### Lexer\n\nThe Lexer is a state-machine that keeps track of the generated Items as the input data is consumed and labled with tokens. It's part cursor, part controller/verifier (within the `StateFn`s), but its main job is keep the state-functions running as the items are consumed, returning lexical items to the caller as they are generated.\n\nThe Lexer should be accompanied with a Parser, that consumes the tokenized Items to build a parse tree.\n\nThis Lexer exposes methods that should be perceived as utilities for the caller when building `StateFn`s. In reality, when actually *running* the Lexer, the caller will loop through its `NextItem()` method until it hits an EOF token.\n\nThe methods in the Lexer will be covered individually below, as well as the design decisions when writing it this way.\n\n```go\n// Lexer describes the behavior of a lexer / parser with cursor capabilities\n//\n// Once spawned, it will consume all tokens as its `NextItem()` method is called,\n// returning processed `Item[C, T]` as it goes\n//\n// Its `Emit()` method pushes items into the stack to be returned, and its `Accept()`,\n// `Check()` and `AcceptRun()` methods act as verifiers for a (set of) token(s)\ntype Lexer[C comparable, T any] interface {\n\n\t// Cursor navigates through a slice in a controlled manner, allowing the\n\t// caller to move forward, backwards, and jump around the slice as they need\n\tcur.Cursor[T]\n\n\t// Emitter describes the behavior of an object that can emit lex.Items\n\t//\n\t// It contains a single method, NextItem, that processes the tokens sequentially\n\t// through the corresponding StateFn\n\t//\n\t// As each item is processed, it is returned to the Lexer by `Emit()`, and\n\t// finally returned to the caller.\n\t//\n\t// Note that multiple calls to `NextItem()` should be made when tokenizing input data;\n\t// usually in a for-loop while the output item is not EOF.\n\tEmitter[C, T]\n\n\t// Emit pushes the set of units identified by token `itemType` to the items channel,\n\t// that returns it in the NextItem() method.\n\t//\n\t// The emitted item will be a subsection of the input data slice, from the lexer's\n\t// starting index to the current position index.\n\t//\n\t// It also sets the lexer's starting index to the current position index.\n\tEmit(itemType C)\n\n\t// Ignore will set the starting point as the current position, ignoring any preceeding units\n\tIgnore()\n\n\t// Backup will rewind the index for the width of the current item\n\tBackup()\n\n\t// Width returns the size of the set of units ready to be emitted with a token\n\tWidth() int\n\n\t// Start returns the current starting-point index for when an item is emitted\n\tStart() int\n\n\t// Check passes the current token through the input `verifFn` function as a validator, returning\n\t// its result\n\tCheck(verifFn func(item T) bool) bool\n\n\t// Accept passes the current token through the input `verifFn` function as a validator, returning\n\t// its result\n\t//\n\t// If the validation passes, the cursor has moved one step forward (the unit was consumed)\n\t//\n\t// If the validation fails, the cursor rolls back one step\n\tAccept(verifFn func(item T) bool) bool\n\n\t// AcceptRun iterates through all following tokens, passing them through the input `verifFn`\n\t// function as a validator\n\t//\n\t// Once it fails the verification, the cursor is rolledback once, leaving the caller at the unit\n\t// that failed the verifFn\n\tAcceptRun(verifFn func(item T) bool)\n}\n\n// Emitter describes the behavior of an object that can emit lex.Items\ntype Emitter[C comparable, T any] interface {\n\t// NextItem processes the tokens sequentially, through the corresponding StateFn\n\t//\n\t// As each item is processed, it is returned to the Lexer by `Emit()`, and\n\t// finally returned to the caller.\n\t//\n\t// Note that multiple calls to `NextItem()` should be made when tokenizing input data;\n\t// usually in a for-loop while the output item is not EOF.\n\tNextItem() Item[C, T]\n}\n```\n\n##### Lex\n\nThis is a simple lexer that will consume a slice of T (any type). It's the simplest implementation that goes in-line with the standard library implementations of `text/template` and Go tokens and is most efficient on bounded / buffered data:\n\n```go\n// Lex implements the Lexer interface, by accepting a slice of a type\ntype Lex[C comparable, T any] struct {\n\tinput []T\n\tstart int\n\tpos   int\n\tstate StateFn[C, T]\n\titems chan Item[C, T]\n}\n```\n\n##### LexBuffer\n\nThis is a lexer implementation prepared for data streams, to generate tokens as available from the reader. It uses a generic reader (`gio.Reader[T]`) to continuously consume tokens.\n\nIt also allows defining a custom look-back size for its inner buffer:\n\n\n```go\n// LexBuffer implements the Lexer interface, by accepting a gio.Reader of any type\ntype LexBuffer[C comparable, T any] struct {\n\tinput              gio.Reader[T]\n\tbuf                []T\n\tstart              int\n\tpos                int\n\tstate              StateFn[C, T]\n\titems              chan Item[C, T]\n\tbufferLookbackSize int\n}\n\n// Size sets a custom buffer look-back size whenever an item is emited\nfunc (l *LexBuffer[C, T]) Size(maxSize int) {\n\tif maxSize \u003c 0 {\n\t\tl.bufferLookbackSize = 0\n\t\treturn\n\t}\n\tl.bufferLookbackSize = maxSize\n}\n```\n\n#### Item\n\nAn Item is an object holding a token and a set of values (lexemes) corresponding to that token. It is a key-value data structure, where the value-half is a slice of any type -- which could be populated with any number of items.\n\nItems are created as the lexer tokenizes symbols / units from the input data, and returned to the caller on each `Lexer.NextItem()` call. \n\n```go\n// Item represents a set of any type of tokens identified by a comparable type\ntype Item[T comparable, V any] struct {\n\tType  T\n\tValue []V\n}\n```\n\n\n\n#### StateFn\n\nA StateFn (state-function) describes a recursive function that will consume each unit of the input data, and either take action, emit tokens, or to finally pass along the data analysis action to a different StateFn. These functions will be implemented by the consumer of the library, describing the steps (their) lexer will take when consuming data. Examples further below.\n\nThe point to the StateFn is to keep the Lexer in control of the state, and the StateFn controlling the cursor and actions the Lexer needs to take. This will allow the biggest flexibility possible, as different Lexer StateFn will derive different flavors for the lexer. Thinking of it as a Markdown lexer, the StateFn would define which and how Markdown tokens are classified, emitted, or even ignored.\n\n\n```go\n// StateFn is a recursive function that updates the Lexer state according to a specific\n// input token along the lexing / parsing action.\n//\n// It returns another StateFn or nil, as it consumes each token with a certain logic applied,\n// passing along the lexing / parsing to the next StateFn\ntype StateFn[C comparable, T any] func(l Lexer[C, T]) StateFn[C, T]\n```\n\n## Implementing\n\n**Note**: *Examples and tests can be found in the [`example`](./example/) directory; from the lexer to the parser*\n\n________\n\n### Token type\n\nImplementing a Lexer requires considering the format of the input data and how it can be tokenized. For this example, the input data is a string, where the lexeme units will be runes.\n\n\u003e The `TemplateItem` will be a comparable (unique) TextToken type, where the lexemes will be runes\n\n```go\n// TemplateItem represents the lex.Item for a runes lexer based on TextToken identifiers\ntype TemplateItem[C TextToken, I rune] lex.Item[C, I]\n```\n\nFor this, the developer needs to define a token type (with an enumeration of expected tokens, where the zero-value for the type is EOF).\n\n\u003e A set of expected tokens are enumerated. In this case the text template will take text\n\u003e between single braces (like `{this}`), and ...replace the braces with double angle-brackets\n\u003e (like `\u003e\u003ethis\u003c\u003c`). Not very fancy but serves as an example.\n\n```go\n// TextToken is a unique identifier for this text template implementation\ntype TextToken int\n\nconst (\n\tTokenEOF TextToken = iota\n\tTokenError\n\tTokenIDENT\n\tTokenTEMPL\n\tTokenLBRACE\n\tTokenRBRACE\n)\n```\n\n\nAfter defining the type, a (set of) `StateFn`(s) need to be created, in context of the input data and how it should be tokenized. Each `StateFn` will hold the responsibility of tokenizing a certain lexeme, and each `StateFn` will have a different flow and responsibility.\n\n### Lexer and state functions\n\n\u003e `initState` switches on the next lexable unit's value, to either emit an item or simply return a new state. This state should be able to listen to all types of (supported) symbols since this example supports so (a user could start a template in the very first char, and end it on the last one)\n\u003e\n\u003e The checks for `l.Width() \u003e 0` ensures that an existing *stack* is being pushed before \n\u003e advancing to the next token in a different procedure (e.g., consider all identifier tokens\n\u003e before going into the `stateBRACE` routine)\n\n\n```go\n// initState describes the StateFn to kick off the lexer. It is also the default fallback StateFn\n// for any other StateFn\nfunc initState[C TextToken, T rune](l lex.Lexer[C, T]) lex.StateFn[C, T] {\n\tswitch l.Next() {\n\tcase '}':\n\t\tif l.Width() \u003e 0 {\n\t\t\tl.Prev()\n\t\t\tl.Emit((C)(TokenIDENT))\n\t\t}\n\t\tl.Ignore()\n\t\treturn stateRBRACE[C, T]\n\tcase '{':\n\t\tif l.Width() \u003e 0 {\n\t\t\tl.Prev()\n\t\t\tl.Emit((C)(TokenIDENT))\n\t\t}\n\t\tl.Ignore()\n\t\treturn stateLBRACE[C, T]\n\tcase 0:\n\t\treturn nil\n\tdefault:\n\t\treturn stateIDENT[C, T]\n\t}\n}\n```\n\n\u003e `stateIDENT` absorbs all text characters until it hits a `{`, `}` or EOF. Then, if the following \n\u003e character is a `{`, or a `}` it returns the `stateLBRACE` or `stateRBRACE` routine, respectively. \n\u003e If it hits EOF, it will return a EOF token and a nil `StateFn`.\n\n\n```go\n// stateIDENT describes the StateFn to parse text tokens.\nfunc stateIDENT[C TextToken, T rune](l lex.Lexer[C, T]) lex.StateFn[C, T] {\n\tl.AcceptRun(func(item T) bool {\n\t\treturn item != '}' \u0026\u0026 item != '{' \u0026\u0026 item != 0\n\t})\n\tswitch l.Next() {\n\tcase '}':\n\t\tif l.Width() \u003e 0 {\n\t\t\tl.Prev()\n\t\t\tl.Emit((C)(TokenIDENT))\n\t\t}\n\t\treturn stateRBRACE[C, T]\n\tcase '{':\n\t\tif l.Width() \u003e 0 {\n\t\t\tl.Prev()\n\t\t\tl.Emit((C)(TokenIDENT))\n\t\t}\n\t\treturn stateLBRACE[C, T]\n\tdefault:\n\t\tif l.Width() \u003e 0 {\n\t\t\tl.Emit((C)(TokenIDENT))\n\t\t}\n\t\tl.Emit((C)(TokenEOF))\n\t\treturn nil\n\t}\n}\n```\n\n\u003e `stateLBRACE` tokenizes the `{` character, returning the initial state after skipping this character\n\n```go\n// stateLBRACE describes the StateFn to check for and emit an LBRACE token\nfunc stateLBRACE[C TextToken, T rune](l lex.Lexer[C, T]) lex.StateFn[C, T] {\n\tl.Next() // skip this symbol\n\tl.Emit((C)(TokenLBRACE))\n\treturn initState[C, T]\n}\n```\n\n\u003e Similarly, `stateRBRACE` tokenizes the `}` character:\n\n```go\n// stateRBRACE describes the StateFn to check for and emit an RBRACE token\nfunc stateRBRACE[C TextToken, T rune](l lex.Lexer[C, T]) lex.StateFn[C, T] {\n\tl.Next() // skip this symbol\n\tl.Emit((C)(TokenRBRACE))\n\treturn initState[C, T]\n}\n```\n\n\u003e Finally `stateError` tokenizes an error if found (none in this lexer's example)\n\n```go\n// stateError describes an errored state in the lexer / parser, ignoring this set of tokens and emitting an\n// error item\nfunc stateError[C TextToken, T rune](l lex.Lexer[C, T]) lex.StateFn[C, T] {\n\tl.Backup()\n\tl.Prev() // mark the previous char as erroring token\n\tl.Emit((C)(TokenError))\n\treturn initState[C, T]\n}\n```\n\n### Parser\n\n#### Parse functions\n\n\u003e Just like the lexer, start by defining a top-level ParseFn that will scan for all expected tokens\n\u003e\n\u003e This function will peek into the next item from the lexer and return the appropriate ParseFn before actually consuming the token\n\n```go\n// initParse describes the ParseFn to kick off the parser. It is also the default fallback \n// for any other ParseFn\nfunc initParse[C TextToken, T rune](t *parse.Tree[C, T]) parse.ParseFn[C, T] {\n\tfor t.Peek().Type != C(TokenEOF) {\n\t\tswitch t.Peek().Type {\n\t\tcase (C)(TokenIDENT):\n\t\t\treturn parseText[C, T]\n\t\tcase (C)(TokenLBRACE), (C)(TokenRBRACE):\n\t\t\treturn parseTemplate[C, T]\n\t\t}\n\t}\n\treturn nil\n}\n```\n\n\u003e `parseText` simply consumes the item as a new node under the current.\n\n```go\n// parseText consumes the next item as a text token, creating a node for it under the\n// current one in the tree. \nfunc parseText[C TextToken, T rune](t *parse.Tree[C, T]) parse.ParseFn[C, T] {\n\tt.Node(t.Next())\n\treturn initParse[C, T]\n}\n```\n\n\u003e `parseTemplate` is a state where we're about to consume either a `{` or a `}`. \n\u003e\n\u003e For `{` tokens, a template Node is created, as it will be a wrapper for one or more text or template items. Returns the initial state.\n\u003e\n\u003e For `}` tokens, the node that is parent to the `{` is set as the current position (closing the template)\n\n```go\n// parseTemplate creates a node for a template item, for which it expects both a text item edge\n// that which also needs to contain an end-template edge.\n//\n// If it encounters a `}` token to close the template, it sets the position up three levels\n// (back to the template's parent)\nfunc parseTemplate[C TextToken, T rune](t *parse.Tree[C, T]) parse.ParseFn[C, T] {\n\tswitch t.Peek().Type {\n\tcase (C)(TokenLBRACE):\n\t\tt.Set(t.Parent())\n\t\tt.Node(t.Next())\n\tcase (C)(TokenRBRACE):\n\t\tt.Node(t.Next())\n\t\tt.Set(t.Parent().Parent.Parent)\n\t}\n\treturn initParse[C, T]\n}\n```\n\n#### Process functions\n\n\u003e `processFn` is the *top-level* processor function, that will consume the nodes in the Tree.\n\u003e\n\u003e It will use a strings.Builder to create the returned string, and iterate through the Tree's\n\u003e root Node's edges and switching on its Type.\n\u003e\n\u003e The content written to the strings.Builder comes from the appropriate `NodeFn` for the Node type.\n\n```go\n// processFn is the ProcessFn that will process the Tree's Nodes, returning a string and an error\nfunc processFn[C TextToken, T rune, R string](t *parse.Tree[C, T]) (R, error) {\n\tvar sb = new(strings.Builder)\n\tfor _, n := range t.List() {\n\t\tswitch n.Type {\n\t\tcase (C)(TokenIDENT):\n\t\t\tproc, err := processText[C, T, R](n)\n\t\t\tif err != nil {\n\t\t\t\treturn (R)(sb.String()), err\n\t\t\t}\n\t\t\tsb.WriteString((string)(proc))\n\t\tcase (C)(TokenLBRACE):\n\t\t\tproc, err := processTemplate[C, T, R](n)\n\t\t\tif err != nil {\n\t\t\t\treturn (R)(sb.String()), err\n\t\t\t}\n\t\t\tsb.WriteString((string)(proc))\n\t\t}\n\t}\n\n\treturn (R)(sb.String()), nil\n}\n```\n\n\u003e for text it's straight-forward, it just casts the T-type values as rune, and returns a string value of it\n\n```go\n// processText converts the T-type items into runes, and returns a string value of it\nfunc processText[C TextToken, T rune, R string](n *parse.Node[C, T]) (R, error) {\n\tvar val = make([]rune, len(n.Value), len(n.Value))\n\tfor idx, r := range n.Value {\n\t\tval[idx] = (rune)(r)\n\t}\n\treturn (R)(val), nil\n}\n```\n\n\u003e for templates, a few checks need to be made -- in this particular example it is to ensure that templates are terminated.\n\u003e \n\u003e the `processTemplate ProcessFn` does that exactly -- it replaces the wrapper text with the appropriate content, adds in the text in the next node, and looks into that text node's edges for a `}` item (to mark the template as closed). Otherwise returns an error:\n\n```go\n// processTemplate prcesses the text within two template nodes\n//\n// Returns an error if a template is not terminated appropriately\nfunc processTemplate[C TextToken, T rune, R string](n *parse.Node[C, T]) (R, error) {\n\tvar sb = new(strings.Builder)\n\tvar ended bool\n\n\tsb.WriteString(\"\u003e\u003e\")\n\tfor _, node := range n.Edges {\n\t\tswitch node.Type {\n\t\tcase (C)(TokenIDENT):\n\t\t\tproc, err := processText[C, T, R](node)\n\t\t\tif err != nil {\n\t\t\t\treturn (R)(sb.String()), err\n\t\t\t}\n\t\t\tfor _, e := range node.Edges {\n\t\t\t\tif e.Type == (C)(TokenRBRACE) {\n\t\t\t\t\tended = true\n\t\t\t\t}\n\t\t\t}\n\t\t\tsb.WriteString((string)(proc))\n\t\tcase (C)(TokenLBRACE):\n\t\t\tproc, err := processTemplate[C, T, R](node)\n\t\t\tif err != nil {\n\t\t\t\treturn (R)(sb.String()), err\n\t\t\t}\n\t\t\tsb.WriteString((string)(proc))\n\t\t}\n\t}\n\tif !ended {\n\t\treturn (R)(sb.String()), fmt.Errorf(\"parse error on line: %d\", n.Pos)\n\t}\n\n\tsb.WriteString(\"\u003c\u003c\")\n\treturn (R)(sb.String()), nil\n}\n```\n\n#### Wrapper\n\n\u003e Perfect! Now all components are wired-up among themselves, and it just needs a simple entrypoint function\n\u003e\n\u003e For this, we can use the template `Parse` function to run it all at once:\n\n```go\n// Run parses the input templated data (a string as []rune), returning\n// a processed string and an error\nfunc Run[C TextToken, T rune, R string](s []T) (R, error) {\n\treturn parse.Run(\n\t\ts,\n\t\tinitState[C, T],\n\t\tinitParse[C, T],\n\t\tprocessFn[C, T, R],\n\t)\n}\n```\n\n## \n\n## Benchmarks\n\n### Lex benchmark\n\n```\ngoos: linux\ngoarch: amd64\npkg: github.com/zalgonoise/lex/example/simple-template\ncpu: AMD Ryzen 3 PRO 3300U w/ Radeon Vega Mobile Gfx\nPASS\nbenchmark                    iter       time/iter   bytes alloc         allocs\n---------                    ----       ---------   -----------         ------\nBenchmarkLexer/Simple-4    817377   2342.00 ns/op      400 B/op   13 allocs/op\nBenchmarkLexer/Complex-4   116460   9643.00 ns/op     1040 B/op   53 allocs/op\nok      github.com/zalgonoise/lex/example/simple-template       3.166s\n```\n\n### LexBuffer benchmark\n\n\u003e `SqueezeTheBuffer` test will take a repetition of the sample template that is 3615 characters in size, to test the efficacy of the buffer approach when handling larger sets, namely for capacity / growth control.\n\u003e\n\u003e This buffer will use the default values of 1025 for the initial buffer capacity, and creating a new buffer whenever the capacity is below 96. \n\n```\ngoos: linux\ngoarch: amd64\npkg: github.com/zalgonoise/lex/example/buffered-template\ncpu: AMD Ryzen 3 PRO 3300U w/ Radeon Vega Mobile Gfx\nPASS\nbenchmark                             iter         time/iter   bytes alloc           allocs\n---------                             ----         ---------   -----------           ------\nBenchmarkLexer/Simple-4             274857     5977.00 ns/op     4580 B/op     26 allocs/op\nBenchmarkLexer/Complex-4             61220    20332.00 ns/op     5592 B/op    159 allocs/op\nBenchmarkLexer/SqueezeTheBuffer-4     2644   509741.00 ns/op    53460 B/op   5019 allocs/op\nok      github.com/zalgonoise/lex/example/buffered-template     4.554s\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzalgonoise%2Flex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzalgonoise%2Flex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzalgonoise%2Flex/lists"}