{"id":25471259,"url":"https://github.com/dropdevrahul/golexer","last_synced_at":"2025-07-04T08:06:23.194Z","repository":{"id":149533291,"uuid":"605773025","full_name":"dropdevrahul/golexer","owner":"dropdevrahul","description":"A simple tokenizer library for parsing text or any text files","archived":false,"fork":false,"pushed_at":"2023-03-30T16:15:00.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-17T17:39:49.703Z","etag":null,"topics":["golang","golang-library","lexer","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dropdevrahul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-23T21:44:57.000Z","updated_at":"2023-02-23T21:48:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"d52f568c-6e7b-4144-9506-f12066c47bc4","html_url":"https://github.com/dropdevrahul/golexer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/dropdevrahul/golexer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropdevrahul%2Fgolexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropdevrahul%2Fgolexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropdevrahul%2Fgolexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropdevrahul%2Fgolexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dropdevrahul","download_url":"https://codeload.github.com/dropdevrahul/golexer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropdevrahul%2Fgolexer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263472277,"owners_count":23471812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","golang-library","lexer","tokenizer"],"created_at":"2025-02-18T09:37:41.510Z","updated_at":"2025-07-04T08:06:23.167Z","avatar_url":"https://github.com/dropdevrahul.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Golexer\n\nA simple golnag library/utility to tokenize a file into individual tokens usually for parsing scripts/programming languages.\n\nPlease note that this utility for now cannot be used to parse simple text files containing only plain text and is more appriopriate \nfor lexing syntax based languages/scripts.\n\n\n### Usage\n\n```\nmake build\n\n./target/golexer path/to/file\n\n```\n\nYou can also pipe stuff into the program\n\n```\necho \"some text\" | ./target/golexer \n```\n\n\n### Example\n\nA file like :\n\n```\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\n\t\"github.com/dropdevrahul/golexer\"\n)\n\nfunc main() {\n\tusage := `Usage: golexer FILE`\n\t//save := flag.Bool(\"c\", false, \"Save results to a file instead of printing\")\n\n\tif len(os.Args) \u003c 2 {\n\t\tfmt.Println(usage)\n\t\treturn\n\t}\n\n\ttz := golexer.NewTokenizer()\n\n\ttokens, err := tz.LexFile(os.Args[1])\n\tif err != nil {\n\t\tlog.Panic(err)\n\t}\n\tfor _, t := range tokens {\n\t\tfmt.Printf(\"%s\\n\", t.Value)\n\t}\n\tfmt.Println()\n}\n```\n\nwill be parse into\n\n```\npackage\nmain\nimport\n(\n\"fmt\"\n\"log\"\n\"os\"\n\"github.com/dropdevrahul/golexer/golexer\"\n)\nfunc\nmain\n(\n)\n{\nusage\n:=\n`Usage:\ngolexer\nFILE`\n//save\n:=\nflag.Bool\n(\n\"c\"\n,\nfalse\n,\n\"Save results to a file instead of printing\"\n)\nif\nlen\n(\nos.Args\n)\n\u003c\n2\n{\nfmt.Println\n(\nusage\n)\nreturn\n}\ntz\n:=\ngolexer.NewTokenizer\n(\n)\ntokens\n,\nerr\n:=\ntz.LexFile\n(\nos.Args\n[\n1\n]\n)\nif\nerr\n!=\nnil\n{\nlog.Panic\n(\nerr\n)\n}\nfor\n_\n,\nt\n:=\nrange\ntokens\n{\nfmt.Printf\n(\n\"%s\\n\"\n,\nt.Value\n)\n}\nfmt.Println\n(\n)\n}\n\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdropdevrahul%2Fgolexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdropdevrahul%2Fgolexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdropdevrahul%2Fgolexer/lists"}