{"id":15443192,"url":"https://github.com/manishmeganathan/symbolizer","last_synced_at":"2025-10-07T21:03:48.117Z","repository":{"id":64299579,"uuid":"568812585","full_name":"manishmeganathan/symbolizer","owner":"manishmeganathan","description":"Go Package for Parsing Simple Symbols","archived":false,"fork":false,"pushed_at":"2023-04-18T07:54:25.000Z","size":45,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-06T03:02:20.827Z","etag":null,"topics":["golang","lexer-parser","lexical-analysis","parser","symbol-parser"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manishmeganathan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-21T13:14:01.000Z","updated_at":"2022-11-23T02:29:24.000Z","dependencies_parsed_at":"2024-06-20T11:27:36.634Z","dependency_job_id":null,"html_url":"https://github.com/manishmeganathan/symbolizer","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/manishmeganathan/symbolizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishmeganathan%2Fsymbolizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishmeganathan%2Fsymbolizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishmeganathan%2Fsymbolizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishmeganathan%2Fsymbolizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manishmeganathan","download_url":"https://codeload.github.com/manishmeganathan/symbolizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manishmeganathan%2Fsymbolizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278846393,"owners_count":26056099,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","lexer-parser","lexical-analysis","parser","symbol-parser"],"created_at":"2024-10-01T19:33:41.472Z","updated_at":"2025-10-07T21:03:48.079Z","avatar_url":"https://github.com/manishmeganathan.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Symbolizer 🔣\n\n[godoclink]: https://godoc.org/github.com/manishmeganathan/symbolizer\n[![go docs](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)][godoclink]\n![go version](https://img.shields.io/github/go-mod/go-version/manishmeganathan/symbolizer?style=flat-square)\n![latest tag](https://img.shields.io/github/v/tag/manishmeganathan/symbolizer?color=brightgreen\u0026label=latest%20tag\u0026sort=semver\u0026style=flat-square)\n![license](https://img.shields.io/github/license/manishmeganathan/symbolizer?color=g\u0026style=flat-square)\n![test status](https://img.shields.io/github/workflow/status/manishmeganathan/symbolizer/Go%20Tests?label=tests\u0026style=flat-square)\n![issue count](https://img.shields.io/github/issues/manishmeganathan/symbolizer?style=flat-square\u0026color=yellow)\n\nA Go Package for Parsing Simple Symbols\n\n### Overview\nThis package is designed for parsing very simple symbols and **not** large files or multi-file directories. It exposes a type ``Parser`` constructable\nwith ``NewParser`` using the string input that needs to be parsed and optional ``ParserOption`` functions to modify its behaviour.\n\n### Installation\n```\ngo get github.com/manishmeganathan/symbolizer\n```\n\n### Token Model\nThe ``Token`` type in this package contains the ``TokenKind``, the literal value as string as well the start position of the token.\n``TokenKind`` are pseudo-runes that represents unicode code points for values above 0. Special tokens for literals and control are\nrepresented as negative variants with values extending below 0. It can extended with custom variants but be mindful of collsions.\n\n```go\n// TokenKind is an enum for representing token grouping/values.\n// For unicode tokens, the TokenKind is equal to its code point value.\n// For literal such identifiers and numerics, the TokenKind values descend from 0.\n// \n// Note: Custom TokenKind values can be used by external packages for keyword detection\n// for special literals, but these values should be below -10 to prevent collisions\ntype TokenKind int32\n\nconst (\n\tTokenEoF TokenKind = -(iota + 1)\n\tTokenIdentifier\n\tTokenNumber\n\tTokenString\n\tTokenHexNumber\n)\n\n// Token represents a lexical Token.\n// It may be either a lone unicode character or some literal value\ntype Token struct {\n\tKind     TokenKind\n\tLiteral  string\n\tPosition int\n}\n```\n\n### Usage Examples\n\n```go\n// TokenIterator describes a routine that leverages the Token inspection methods of Parser\n// to view the current Token and check if the current Token is of a specific TokenKind.\nfunc TokenIterator() {\n    symbol := \"map[string]string\"\n    parser := symbolizer.NewParser(symbol)\n  \n    // Check if the cursor has reach the end of the symbol\n    for !parser.IsCursor(symbolizer.TokenEoF) { \n        // Print the current token\n        fmt.Println(parser.Cursor()) \n        // Advance the cursor and ingest the next token\n        parser.Advance()\n    }\n\n    // Output:\n    // {\u003cident\u003e map 0}\n    // {\u003cunicode:'['\u003e [ 3}\n    // {\u003cident\u003e string 4}\n    // {\u003cunicode:']'\u003e ] 10}\n    // {\u003cident\u003e string 11}\n}\n```\n\n```go\n// TokenLookAhead describes a routine that leverages the Token look ahead methods of Parser\n// to view the next Token without ingesting it. ExpectPeek can be used to move the parser\n// if the next token is of a specific kind.\nfunc TokenLookAhead() {\n    symbol := \"[32]string\"\n    parser := symbolizer.NewParser(symbol)\n\n    // Print the current and next token\n    fmt.Println(parser.Cursor())\n    fmt.Println(parser.Peek())\n    \n    // Check if the next token is a numeric\n    // If it is, then the parse cursor is moved forward\n    if parser.ExpectPeek(symbolizer.TokenNumber) {\n        fmt.Println(\"numeric encountered\")\n    }\n    \n    // Print the current and next token after the peek expectation\n    fmt.Println(parser.Cursor())\n    fmt.Println(parser.Peek())\n\n    // Output:\n    // {\u003cunicode:'['\u003e [ 0}\n    // {\u003cnum\u003e 32 1}\n    // numeric encountered\n    // {\u003cnum\u003e 32 1}\n    // {\u003cunicode:']'\u003e ] 3}\n}\n```\n\n```go\n// CustomSymbols describes a routine that injects some custom keywords and token kinds\n// into the parser, which can then be used to inspect tokens just as would regular TokenKind variants.\nfunc CustomSymbols() {\n    symbol := \"map[string]string\"\n\t\n    // Define a custom token kind enum\n    type MyTokenKind int32\n    const Datatype = -10\n    \n    // Defines a mapping of identifier to custom token kinds\n    keywords := map[string]symbolizer.TokenKind{\"map\": Datatype, \"string\": Datatype}\n    // Create a Parser with a ParserOption that injects the custom keywords\n    parser := symbolizer.NewParser(symbol, symbolizer.Keywords(keywords))\n    \n    // Check if the cursor has reach the end of the symbol\n    for !parser.IsCursor(symbolizer.TokenEoF) {\n        // Print the current token\n        fmt.Println(parser.Cursor())\n        // Advance the cursor and ingest the next token\n        parser.Advance()\n    }\n\n    // Output:\n    // \t{\u003ccustom:-10\u003e map 0}\n    // \t{\u003cunicode:'['\u003e [ 3}\n    // \t{\u003ccustom:-10\u003e string 4}\n    // \t{\u003cunicode:']'\u003e ] 10}\n    //  {\u003ccustom:-10\u003e string 11}\n}\n```\n\n```go\n// SymbolSplit describes a routine that splits a Symbols into sub-strings (sub symbols) based on \n// some delimiter rune. Use a Whitespace Ignorant Parser if they should be ignored while splitting.\nfunc SymbolSplit() {\n    symbol := \"23, 56, 8902342\"\n\t\n\t// Create a Parser with a ParseOption to ignore whitespaces\n    parser := NewParser(symbol, IgnoreWhitespaces())\n    // Split the parser contents with the comma delimiter\n    components := parser.Split(',')\n\t\n\t// Print all the components\n    for _, component := range components {\n        fmt.Println(component)\n    }\n\t\n    // Output: \n    // 23\n    // 56\n    // 8902342\n}\n```\n\n```go\n// SymbolUnwrap describes a routine that unwraps an inner sub-string (inner symbol) from within some \n// Enclosure, which is defined as a pair of unicode characters (cannot be the same). Unwrap can also\n// handle nested contents of the same enclosure and works to resolve each opening with a closing.\nfunc SymbolUnwrap() {\n    symbol := \"(outer[inner])\"\n    parser := NewParser(symbol)\n\n\t// Unwrap the symbol from within a set of parenthesis\n    unwrapped, err := parser.Unwrap(EnclosureParens())\n    if err != nil {\n        panic(err)\n    }\n\n\t// Print the unwrapped symbol\n    fmt.Println(unwrapped)\n\n    // Output: \n    // outer[inner]\n}\n```\n\n### Notes:\nThis package is still a work in progress and can be heavily extended for a lot of different use cases.\nIf you are using this package and need some new functionality, please open an issue or a pull request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanishmeganathan%2Fsymbolizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanishmeganathan%2Fsymbolizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanishmeganathan%2Fsymbolizer/lists"}