{"id":26769905,"url":"https://github.com/bbuck/go-lexer","last_synced_at":"2025-04-15T17:11:47.442Z","repository":{"id":992844,"uuid":"36549390","full_name":"bbuck/go-lexer","owner":"bbuck","description":"Lexer based on Rob Pike's talk on YouTube (view README)","archived":false,"fork":false,"pushed_at":"2022-05-12T14:29:49.000Z","size":8,"stargazers_count":85,"open_issues_count":1,"forks_count":22,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-15T17:11:46.394Z","etag":null,"topics":["golang","lexer"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bbuck.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-30T08:06:52.000Z","updated_at":"2025-04-03T14:55:42.000Z","dependencies_parsed_at":"2022-07-21T11:04:59.506Z","dependency_job_id":null,"html_url":"https://github.com/bbuck/go-lexer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuck%2Fgo-lexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuck%2Fgo-lexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuck%2Fgo-lexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbuck%2Fgo-lexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bbuck","download_url":"https://codeload.github.com/bbuck/go-lexer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249116245,"owners_count":21215143,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","lexer"],"created_at":"2025-03-28T22:46:22.216Z","updated_at":"2025-04-15T17:11:47.421Z","avatar_url":"https://github.com/bbuck.png","language":"Go","readme":"This package provides a Lexer that functions similarly to Rob Pike's discussion\nabout lexer design in this [talk](https://www.youtube.com/watch?v=HxaD_trXwRE).\n\nYou can define your token types by using the `lexer.TokenType` type (`int`) via\n\n```go\nconst (\n        StringToken lexer.TokenType = iota\n        IntegerToken\n        etc...\n)\n```\n\nAnd then you define your own state functions (`lexer.StateFunc`) to handle\nanalyzing the string.\n\n```go\nfunc StringState(l *lexer.L) lexer.StateFunc {\n        l.Next() // eat starting \"\n        l.Ignore() // drop current value\n        for l.Peek() != '\"' {\n                l.Next()\n        }\n        l.Emit(StringToken)\n\n        return SomeStateFunction\n}\n```\n\nIt should be easy to make this Lexer consumable by a parser generated by go yacc doing something alone the lines of the following:\n\n```go\ntype MyLexer struct {\n        lexer.L\n}\n\nfunc (m *MyLexer) Lex(lval *yySymType) int {\n        tok, done := m.NextToken()\n        if done {\n                return EOFToken\n        } else {\n                lval.val = tok.Value\n                return tok.Type\n        }\n}\n```\n\n# License\n\nMIT","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbuck%2Fgo-lexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbbuck%2Fgo-lexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbuck%2Fgo-lexer/lists"}