{"id":17968185,"url":"https://github.com/a-h/lexical","last_synced_at":"2025-07-29T05:06:05.610Z","repository":{"id":57509973,"uuid":"105472358","full_name":"a-h/lexical","owner":"a-h","description":"A set of tools for building parsers using the Go programming language.","archived":false,"fork":false,"pushed_at":"2023-04-03T09:02:57.000Z","size":104,"stargazers_count":43,"open_issues_count":0,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-20T00:11:44.842Z","etag":null,"topics":["go","golang","parse","parser-combinators","parsing","strea"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/a-h.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-01T20:32:10.000Z","updated_at":"2025-02-17T16:00:41.000Z","dependencies_parsed_at":"2024-06-18T22:52:30.929Z","dependency_job_id":"00bd703e-5b75-4172-9718-a3163448c85f","html_url":"https://github.com/a-h/lexical","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/a-h%2Flexical","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/a-h%2Flexical/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/a-h%2Flexical/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/a-h%2Flexical/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/a-h","download_url":"https://codeload.github.com/a-h/lexical/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245426486,"owners_count":20613374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","parse","parser-combinators","parsing","strea"],"created_at":"2024-10-29T14:20:28.977Z","updated_at":"2025-03-25T08:31:59.386Z","avatar_url":"https://github.com/a-h.png","language":"Go","readme":"\u003e **Warning**\n\u003e This library is now unmaintained in favour of a Go generics rewrite at https://github.com/a-h/parse\n\n# Lexical\n\nA set of parsing tools for Go inspired by [Sprache](https://github.com/sprache/Sprache/).\n\n## Input\n\nParsers first need to read data to see if the data matches a pattern. If the data doesn't match, then the parser will need to move back to the end position of the last successful parse to try a different pattern.\n\nJust storing everything in RAM works if your file sizes are small, but your process uses a lot of RAM.\n\nWriting to a file to do this would mean seeking on disk, potentially making the performance suffer.\n\nInstead, the `Stream` type provides a way of reading runes (characters) from an input `bufio.Reader` into a cache in RAM. Once a token has been consumed by the parser, the consumed bytes are discarded. The amount of RAM consumed will depend on the parser that uses it.\n\nThe `Stream` type implements the `parse.Input` interface:\n\n```go\n// Input represents the input to a parser.\ntype Input interface {\n\t// Collect collects all of the string data parsed so far and returns it, then starts a new collection\n\t// from the current position in the input.\n\tCollect() string\n\t// Advance advances the input by a single rune and consumes it.\n\tAdvance() (rune, error)\n\t// Retreat retreats the input position by a single rune and unconsumes it.\n\tRetreat() (rune, error)\n\t// Peek returns the next rune from the input without consuming it.\n\tPeek() (rune, error)\n\t// Position returns the line and column number of the current position within the stream.\n\tPosition() (line, column int)\n\t// Index returns the current index of the parser input.\n\tIndex() int64\n}\n```\n\n## Parser Functions\n\nParser functions provide a way of matching patterns in a given input. They are designed to be able to be composed together to make more complex operations.\n\nThe [examples](./examples) directory contains several examples of composing the primitive functions.\n\n### Functions\n\n* `Any`\n    * Parse any of the provided parse functions, or roll back.\n* `AnyRune`\n    * Parse any rune.\n* `AtLeast`\n    * Parse the provided function at least the number of times specified, or roll back.\n* `AtMost`\n    * Parse the provided function at least once, and at most the number of times specified, or roll back.\n* `Letter`\n    * Parse any letter in the Unicode Letter range or roll back.\n* `Many`\n    * Parse the provided parse function a number of times or roll back.\n* `Optional`\n    * Attempt to parse, but don't roll back if a match isn't found.\n* `Or`\n    * Return the first successful result of the provided parse functions, or roll back.\n* `Rune`\n    * Parse the specified rune (character) or fallback.\n* `RuneIn`\n    * Parse a rune from the input stream if it's in the specified string, or roll back.\n* `RuneInRanges`\n    * Parse a rune from the input stream if it's in the specified Unicode ranges, or roll back.\n* `RuneNotIn`\n    * Parse a rune from the input stream if it's not in the specified string, or roll back.\n* `RuneWhere`\n    * Parse a rune from the input stream if the predicate function passed in succeeds, or roll back.\n* `String`\n    * Parse a string from the input stream if it exactly matches the provided string, or roll back.\n* `StringUntil`\n    * Parse a string from the input stream until the specified _until_ parser is matched.\n* `Then`\n    * Return the results of the first and second parser passed through the combiner function which converts the two results into a single output (a map / reduce operation), or roll back if either doesn't match.\n* `Times`\n    * Parse using the specified function a set number of times or roll back.\n* `ZeroToNine`\n    * Parse a rune from the input stream if it's within the set of 1234567890.\n\n### Examples\n\nUsing the `Or` function to parse either 'A' or 'B':\n\n```go\nparser := parse.Or(parse.Rune('A'), parse.Rune('B'))\n\nmatchesA := parser(input.NewFromString(\"A\")).Success // true\nmatchesB := parser(input.NewFromString(\"B\")).Success // true\nmatchesC := parser(input.NewFromString(\"C\")).Success // false\n\nfmt.Println(matchesA) // true\nfmt.Println(matchesB) // true\nfmt.Println(matchesC) // false\n\n```\n\nThe `Or` function only returns a single result but the `Many` function is more complex, because you generally want to do something with the results, such as convert the runes or strings captured by the parser into another value. The `parse.WithIntegerCombiner` and `parse.WithStringConcatCombiner` functions provide some default implementations.\n\nThe [examples](./examples) directory contains several examples of taking the primitive parse results and returning other types such as dates and URLs.\n\n\n```go\n// parse.WithIntegerCombiner concatentates the captured runes into a string,\n// and parses the result to an integer.\noneToThreeNumbers := parse.Many(parse.WithIntegerCombiner,\n    1, // minimum match count\n    3, // maximum match count\n    parse.ZeroToNine)\n\nresultA := oneToThreeNumbers(input.NewFromString(\"123\"))\nfmt.Println(resultA.Success) // true\nfmt.Println(resultA.Item)    // integer value of 123\n\nresultB := oneToThreeNumbers(input.NewFromString(\"1234\"))\nfmt.Println(resultB.Success) // true\nfmt.Println(resultB.Item)    // integer value of 123\n\n// This Many function will stop reading at the 'a'.\nresultC := oneToThreeNumbers(input.NewFromString(\"1a234\"))\nfmt.Println(resultC.Success) // true\nfmt.Println(resultC.Item)    // integer value of 1\n\n// Parse letters into a string\nupToThreeLetters := parse.AtMost(parse.WithStringConcatCombiner, 3, parse.Letter)\nletters := upToThreeLetters(input.NewFromString(\"ABC1\"))\n// Check that we got a string back from the parser with a 'type assertion'.\nresultItem, ok := letters.Item.(string)\nif !ok || resultItem != \"ABC\" {\n    t.Errorf(\"for 'ABC1', expected to extract 'ABC', but extracted '%v'\", letters.Item)\n}\n```\n\n## Scanner\n\nThe `Scanner` type combines the parser functions and `Stream` type to allow parsing of input files. See `scanner_test.go` for a working example.\n\n```go\nstream := input.NewFromString(`\u003ca\u003eExample\u003c/a\u003e`)\n\nscanner := New(stream, xmlTokens)\nvar err error\nfor {\n    item, err := scanner.Next()\n    // Do something with the results based on the \n    // token's type.\n    switch v := item.(type) {\n        case string:\n            fmt.Println(v)\n        case int:\n            fmt.Println(v)\n    }\n    if err != nil {\n        break\n    }\n}\nif err != nil \u0026\u0026 err != io.EOF {\n    panic(\"error\")\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fa-h%2Flexical","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fa-h%2Flexical","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fa-h%2Flexical/lists"}