{"id":20506319,"url":"https://github.com/felipensp/liblex","last_synced_at":"2025-04-13T21:21:17.396Z","repository":{"id":146714682,"uuid":"45855278","full_name":"felipensp/liblex","owner":"felipensp","description":"C library for Lexical Analysis","archived":false,"fork":false,"pushed_at":"2015-12-13T12:54:13.000Z","size":50,"stargazers_count":30,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-27T11:43:23.849Z","etag":null,"topics":["c","lex","lexical","lib","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/felipensp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-09T17:41:33.000Z","updated_at":"2025-02-01T05:35:29.000Z","dependencies_parsed_at":"2023-04-15T19:27:48.542Z","dependency_job_id":null,"html_url":"https://github.com/felipensp/liblex","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipensp%2Fliblex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipensp%2Fliblex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipensp%2Fliblex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipensp%2Fliblex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/felipensp","download_url":"https://codeload.github.com/felipensp/liblex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248783271,"owners_count":21160899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","lex","lexical","lib","tokenizer"],"created_at":"2024-11-15T19:56:00.274Z","updated_at":"2025-04-13T21:21:17.367Z","avatar_url":"https://github.com/felipensp.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# liblex\nC library for Lexical Analysis\n\n[![Build Status](https://travis-ci.org/felipensp/liblex.svg?branch=master)](https://travis-ci.org/felipensp/liblex)\n[![Join the chat at https://gitter.im/felipensp/liblex](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/felipensp/liblex?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n\n#### Usage:\n\n```c\n/*\n * Lexer example\n * \n * Author: Felipe Pena \u003cfelipensp at gmail.com\u003e\n */\n \n#include \u003cstdio.h\u003e\n#include \u003cliblex.h\u003e\n\nenum { /* Tokens */\n\tMYLEXER_PLUS = 1,\n\tMYLEXER_MINUS,\n\tMYLEXER_DIV,\n\tMYLEXER_MULT,\n\tMYLEXER_MOD,\n\tMYLEXER_START_COMMENT,\n\tMYLEXER_END_COMMENT,\n\tMYLEXER_WHITESPACE,\n\tMYLEXER_NUMBER,\n\tMYLEXER_IGNORE\n};\n\nenum { /* States */\n\tINITIAL = 0,\n\tCOMMENT\n};\n\nint start_comment_callback(llex *lex, char *str, size_t len) \n{\n\tllex_set_state(lex, COMMENT);\n\treturn MYLEXER_START_COMMENT;\n}\n\nint end_comment_callback(llex *lex, char *str, size_t len) \n{\n\tllex_set_state(lex, INITIAL);\n\treturn MYLEXER_END_COMMENT;\n}\n\nint number_callback(llex *lex, char *str, size_t len) \n{\n\treturn MYLEXER_NUMBER;\n}\n\nint main(int argc, char **argv)\n{\n\tllex lex;\n\tllex_token_id token_id;\n\t\t\n\tllex_init(\u0026lex);\n\tllex_set_buffer(\u0026lex, \"1 - 2 + 3 / 4 \\n\"\n\t\t\t\t\t\t  \"/* ignored str */\");\n\t\n\tllex_set_state(\u0026lex, INITIAL);\n\tllex_add_token_callback(\u0026lex, \"/*\", start_comment_callback);\n\t\n\tllex_set_state(\u0026lex, COMMENT);\n\tllex_add_token_callback(\u0026lex, \"*/\", end_comment_callback);\n\tllex_add_token_regex(\u0026lex, \"(?:(?!\\\\*/).)+\", MYLEXER_IGNORE);\n\t\n\tllex_set_state(\u0026lex, INITIAL);\n\tllex_add_token(\u0026lex, \"+\", MYLEXER_PLUS);\n\tllex_add_token(\u0026lex, \"-\", MYLEXER_MINUS);\n\tllex_add_token(\u0026lex, \"/\", MYLEXER_DIV);\n\tllex_add_token(\u0026lex, \"*\", MYLEXER_MULT);\n\tllex_add_token_regex(\u0026lex, \"\\\\s+\", MYLEXER_WHITESPACE);\n\tllex_add_token_regex_callback(\u0026lex, \"\\\\d+\", number_callback);\n\t\n\twhile ((token_id = llex_tokenizer(\u0026lex)) \u003e 0) {\n\t\tprintf(\"Token id: %d - State: %d - '%.*s' - Start: %d:%d / End: %d:%d\\n\",\n\t\t\ttoken_id, \n\t\t\tlex.current_state,\n\t\t\tlex.current_len,\n\t\t\tlex.current_token,\n\t\t\tlex.buffer_col_start,\n\t\t\tlex.buffer_line_start,\n\t\t\tlex.buffer_col_end,\t\t\t\n\t\t\tlex.buffer_line_end);\n\t}\n\tif (token_id == -1) {\n\t\tprintf(\"Unknown string `%s'\\n\", lex.current_token);\n\t}\n\t\n\tllex_cleanup(\u0026lex);\n\t\n\treturn 0;\n}\n```\n\nOutputs:\n\n```\nToken id: 9 - State: 0 - '1' - Start: 0:1 / End: 1:1\nToken id: 8 - State: 0 - ' ' - Start: 1:1 / End: 2:1\nToken id: 2 - State: 0 - '-' - Start: 2:1 / End: 3:1\nToken id: 8 - State: 0 - ' ' - Start: 3:1 / End: 4:1\nToken id: 9 - State: 0 - '2' - Start: 4:1 / End: 5:1\nToken id: 8 - State: 0 - ' ' - Start: 5:1 / End: 6:1\nToken id: 1 - State: 0 - '+' - Start: 6:1 / End: 7:1\nToken id: 8 - State: 0 - ' ' - Start: 7:1 / End: 8:1\nToken id: 9 - State: 0 - '3' - Start: 8:1 / End: 9:1\nToken id: 8 - State: 0 - ' ' - Start: 9:1 / End: 10:1\nToken id: 3 - State: 0 - '/' - Start: 10:1 / End: 11:1\nToken id: 8 - State: 0 - ' ' - Start: 11:1 / End: 12:1\nToken id: 9 - State: 0 - '4' - Start: 12:1 / End: 13:1\nToken id: 8 - State: 0 - ' \n' - Start: 13:1 / End: 1:2\nToken id: 6 - State: 1 - '/*' - Start: 1:2 / End: 3:2\nToken id: 10 - State: 1 - ' ignored str ' - Start: 3:2 / End: 16:2\nToken id: 7 - State: 0 - '*/' - Start: 16:2 / End: 18:2\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelipensp%2Fliblex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffelipensp%2Fliblex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelipensp%2Fliblex/lists"}