{"id":18183681,"url":"https://github.com/jaysmito101/lexpp","last_synced_at":"2025-04-01T21:31:05.687Z","repository":{"id":134769325,"uuid":"425020494","full_name":"Jaysmito101/lexpp","owner":"Jaysmito101","description":"Small Extremely Powerful Header Only C++ Lexical Analyzer/String Parser Library","archived":false,"fork":false,"pushed_at":"2024-01-01T07:38:40.000Z","size":87,"stargazers_count":66,"open_issues_count":1,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-15T21:04:56.295Z","etag":null,"topics":["cpp11","header-only","header-only-library","lexer","lexical-analyzer","string-parser"],"latest_commit_sha":null,"homepage":"https://github.com/Jaysmito101/lexpp/wiki","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jaysmito101.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-11-05T16:48:31.000Z","updated_at":"2025-01-29T20:05:42.000Z","dependencies_parsed_at":"2024-01-01T08:55:56.144Z","dependency_job_id":"33db84cd-a6ed-4702-b0c6-0d6f139304b0","html_url":"https://github.com/Jaysmito101/lexpp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jaysmito101%2Flexpp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jaysmito101%2Flexpp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jaysmito101%2Flexpp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jaysmito101%2Flexpp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jaysmito101","download_url":"https://codeload.github.com/Jaysmito101/lexpp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246665478,"owners_count":20814320,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp11","header-only","header-only-library","lexer","lexical-analyzer","string-parser"],"created_at":"2024-11-02T20:03:53.744Z","updated_at":"2025-04-01T21:31:05.302Z","avatar_url":"https://github.com/Jaysmito101.png","language":"C++","funding_links":["https://patreon.com/jaysmito101"],"categories":[],"sub_categories":[],"readme":"\u003cbr/\u003e\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/Jaysmito101/lexpp/master/images/ReadmeHeader.png\" border=\"0\"\u003e\u003c/\n\u003c/p\u003e\n\n\u003cbr/\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"GitHub repo size\" src=\"https://img.shields.io/github/repo-size/Jaysmito101/lexpp?style=for-the-badge\"\u003e\n  \u003cimg alt=\"Lines of code\" src=\"https://img.shields.io/tokei/lines/github/Jaysmito101/lexpp?style=for-the-badge\"\u003e\n  \u003cimg alt=\"GitHub commit activity\" src=\"https://img.shields.io/github/commit-activity/w/Jaysmito101/lexpp?style=for-the-badge\"\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://patreon.com/jaysmito101\"\u003e\u003cimg src=\"https://img.shields.io/endpoint.svg?url=https%3A%2F%2Fshieldsio-patreon.vercel.app%2Fapi%3Fusername%3Djaysmito101%26type%3Dpledges\u0026style=for-the-badge\" alt=\"Support me on Patreon\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003cbr/\u003e\n\n\n\n# lexpp\nSmall Extremely Powerful Header Only C++ Lexical Analyzer/String Parser Library\n\n\nLexpp is made with simplicity and size in mind. The entire library is about 500 lines!\n\nLexpp is very powerful and can be used for almost all parsing needs!\n\nYou can check the \u003ca  href=\"https://github.com/Jaysmito101/lexpp/tree/master/examples\"\u003eexamples/\u003c/a\u003e for more elaborate usage.\n\n# Docs : \n### https://github.com/Jaysmito101/lexpp/wiki\n### For any help join our discord server https://discord.gg/muq5nDxF9t\n\n# How to Use\n\nJust place the `lexpp.h` file in your project include directory.\n\nIn one cpp file define `LEXPP_IMPLEMENTATION` before importing lexpp like  this:\n\n\n    #define LEXPP_IMPLEMENTATION\n    #include \"lexpp.h\"\n    \nYou are all done to use lexpp!\n\n\n# Basic Examples\n\n## String parsing\n\u003cdetails\u003e\n    \u003csummary\u003e Click To See Code \u003c/summary\u003e\n    \n    std::string data = \"some text to parse! \";\n    std::vector\u003cstd::string\u003e tokens = lexpp::lex(data, {\"\u003c=\", \"\u003c\u003c\", \"\\n\", \"::\", \",\", \"}\", \"{\", \";\", \" \"}, false);\n\n    for(std::string\u0026 token : tokens){\n        std::cout \u003c\u003c token \u003c\u003c std::endl;\n    }\n        \n \u003c/details\u003e\n\n\n## Using Custom Token Classifier\n        \nSome Structs we will need\n\u003cdetails\u003e\n        \u003csummary\u003e Click To See Code \u003c/summary\u003e\n        \n    enum MyTokens{\n        Keyword = 0,\n        Number,\n        String,\n        Other\n    };\n    \n    static std::string TokenToString(int tok){\n    switch(tok){\n        case Keyword: return \"Keyword\";\n        case Number:  return \"Number\";\n        case String:  return \"String\";\n        case Other:   return \"Other\";\n    }\n    }\n        \n\u003c/details\u003e\n\nNow the Lexing\n        \n\u003cdetails\u003e\n        \u003csummary\u003e Click To See Code \u003c/summary\u003e\n        \n    std::vector\u003cstd::string\u003e keywords = {\"for\", \"void\", \"return\", \"if\", \"int\"};\n    std::vector\u003clexpp::Token\u003e tokens = lexpp::lex(data, {\"\u003c=\", \"\u003c\u003c\", \"\\n\", \"::\", \",\", \"}\", \"{\", \"(\", \")\" \";\", \" \"}, [keywords](std::string\u0026 token, bool* discard, bool is_separator) -\u003e int {\n        if(std::find(keywords.begin(), keywords.end(), token) != keywords.end()){\n            return MyTokens::Keyword;\n        }\n        if(is_number(token))\n            return MyTokens::Number;\n        else\n            return MyTokens::String;\n    }, false);\n\n    for(lexpp::Token\u0026 token : tokens){\n        std::cout \u003c\u003c TokenToString(token.type) \u003c\u003c \" -\u003e \" \u003c\u003c token.value \u003c\u003c std::endl;\n    }\n    \n\u003c/details\u003e\n\n## Using the `TokenParser` class\n\nWe need to extend the `TokenParser` class to have our cuastom token parser\n\n\u003cdetails\u003e\n        \u003csummary\u003e Click To See Code \u003c/summary\u003e\n        \n    class MyTokenParser : public lexpp::TokenParser\n    {\n    public:\n    MyTokenParser(std::string data, std::string separators)\n    :TokenParser(data, separators, false){}\n\n    virtual int process_token(std::string\u0026 token, bool* discard, bool isSeparator) override\n    {\n        if(std::find(keywords.begin(), keywords.end(), token) != keywords.end())\n            return MyTokens::Keyword;\n        else if(is_number(token))\n            return MyTokens::Number;\n        else if(isSeparator)\n            return MyTokens::Other;\n        else\n            return MyTokens::String;\n    }    \n\n    std::vector\u003cstd::string\u003e keywords = {\"for\", \"void\", \"return\", \"if\", \"int\"};\n    };\n        \n\u003c/details\u003e\n    \nNow using the class with the lexer\n\n\u003cdetails\u003e\n        \u003csummary\u003e Click To See Code \u003c/summary\u003e\n            \n    std::vector\u003clexpp::Token\u003e tokens =     lexpp::lex(std::make_shared\u003cMyTokenParser\u003e(data, \"\\n :,[]{}().\\t\"));\n    for(lexpp::Token\u0026 token : tokens){\n        std::cout \u003c\u003c TokenToString(token.type) \u003c\u003c \" -\u003e \" \u003c\u003c token.value \u003c\u003c std::endl;\n    }\n            \n\u003c/details\u003e\n            \n## Making an email parser with lexpp\n\nFirst a struct to store out data\n            \n\u003cdetails\u003e\n        \u003csummary\u003e Click To See Code \u003c/summary\u003e     \n    \n    struct Email{\n        std::string name;\n        std::string domainFront;\n        std::string domainEnd;\n        std::string domain;\n    };\n    \n\u003c/details\u003e\n    \nNow we need to make our custom token parser for email parsing\n\n\u003cdetails\u003e\n        \u003csummary\u003e Click To See Code \u003c/summary\u003e\n      \n    class EmailTokenParser : public lexpp::TokenParser\n    {\n    public:\n    EmailTokenParser(std::string data, std::string separators = \"\\n@.\")\n    :TokenParser(data, separators, true){}\n\n    virtual int process_token(std::string\u0026 token, bool* discard, bool isSeparator) override\n    {\n        if(isSeparator){\n            if(ci == 2){\n                currMail.domain = currMail.domainFront + \".\" + currMail.domainEnd;\n                emailIds.push_back(currMail);\n                ci = 0;\n                *discard = true;\n                return 0;  \n            }\n            if(token.size() \u003c= 0){\n                *discard = true;\n                return 0;  \n            }\n            if(token == \"\\n\"){\n                ci = 0;\n                *discard = true;\n                return 0;  \n            }\n            else if(token == \"@\"){\n                ci = 1;\n                *discard = true;\n                return 0;                \n            }\n            else if(token == \".\"){\n                ci = 2;\n                *discard = true;\n                return 0;                \n            }\n        }\n\n        if(ci == 0)\n            currMail.name = token;\n        else if(ci == 1)\n            currMail.domainFront = token;\n        else if(ci == 2)\n            currMail.domainEnd = token;\n    }    \n\n    int ci = 0;\n    Email currMail;\n    std::vector\u003cEmail\u003e emailIds;\n    };\n      \n\u003c/details\u003e\n    \nNow finally calling lex\n\u003cdetails\u003e\n        \u003csummary\u003e Click To See Code \u003c/summary\u003e\n    \n    std::shared_ptr\u003cEmailTokenParser\u003e tok_parser = std::make_shared\u003cEmailTokenParser\u003e(data+\"\\n\", \"\\n@.\");\n    lexpp::lex(tok_parser);\n    for(Email\u0026 email : tok_parser-\u003eemailIds){\n        std::cout \u003c\u003c \"Email : \\nNAME: \" \u003c\u003c email.name \u003c\u003c \"\\nDOMAIN : \" \u003c\u003c email.domain \u003c\u003c std::endl;\n    }\n    \n\u003c/details\u003e\n\n# Support\n\nI am just a Highschool student so I may not have the best quality of code but still i am trying my best to write good code!\n\nAny support would be highly appretiated!\n\nFor example you could add a feature and contribute via pull requests or you could even report any issues with the program!\n\nAnd the best thing you could do to support this project is spread word about this so that more people who might be interested in this may use this!\n\nPlease considering tweeting about this! \n\n\n\u003cimg alt=\"Visitors\" src=\"https://visitor-badge.glitch.me/badge?page_id=Jaysmito101.lexpp\u0026left_color=gray\u0026right_color=green\u0026style=for-the-badge\"\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaysmito101%2Flexpp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaysmito101%2Flexpp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaysmito101%2Flexpp/lists"}