{"id":29894156,"url":"https://github.com/juxt/reap","last_synced_at":"2025-08-01T04:45:27.257Z","repository":{"id":56068595,"uuid":"247404978","full_name":"juxt/reap","owner":"juxt","description":"A Clojure library for decoding and encoding strings used by web protocols.","archived":false,"fork":false,"pushed_at":"2025-05-14T15:42:22.000Z","size":417,"stargazers_count":77,"open_issues_count":4,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-29T14:45:00.998Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/juxt.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-03-15T05:05:14.000Z","updated_at":"2025-05-25T13:02:58.000Z","dependencies_parsed_at":"2024-01-29T13:30:54.186Z","dependency_job_id":"bb762e93-d952-4e26-8122-a243bbb802a2","html_url":"https://github.com/juxt/reap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/juxt/reap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juxt%2Freap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juxt%2Freap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juxt%2Freap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juxt%2Freap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/juxt","download_url":"https://codeload.github.com/juxt/reap/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juxt%2Freap/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268171716,"owners_count":24207417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-01T04:45:23.770Z","updated_at":"2025-08-01T04:45:27.226Z","avatar_url":"https://github.com/juxt.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Reap\n\nRegular Expressions for Accurate Parsing\n\nA Clojure library for decoding and encoding strings used by web protocols.\n\n[WARNING]\n--\nSTATUS: Ready to use, the API is stable.\n--\n\n== Quick Start\n\nSuppose you want to decode an Accept header from an HTTP request. For example, Firefox sends one like this:\n\n`Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8`\n\nwhereas a Chrome browser on Windows 7 might send:\n\n`Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9`\n\nYou can use *reap* to parse that header's value into data you can more easily work with.\n\nHere's how:\n\n[source,clojure]\n----\n(require\n  '[juxt.reap.decoders.rfc7231 :refer [accept]]\n  '[juxt.reap.regex :as re])\n\n(let [decoder (accept {})]\n  (decoder\n    (re/input\n      \"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\")))\n----\n\nThis will return the following sequence of items:\n\n[source,clojure]\n----\n#:juxt.reap.rfc7231\n{:media-range \"text/html\",\n :type \"text\",\n :subtype \"html\",\n :parameters {}}\n\n#:juxt.reap.rfc7231\n{:media-range \"application/xhtml+xml\",\n :type \"application\",\n :subtype \"xhtml+xml\",\n :parameters {}}\n\n#:juxt.reap.rfc7231\n{:media-range \"application/xml\",\n :type \"application\",\n :subtype \"xml\",\n :parameters {},\n :qvalue 0.9}\n\n#:juxt.reap.rfc7231\n{:media-range \"image/webp\",\n :type \"image\",\n :subtype \"webp\",\n :parameters {}}\n\n#:juxt.reap.rfc7231\n{:media-range \"image/apng\",\n :type \"image\",\n :subtype \"apng\",\n :parameters {}}\n\n#:juxt.reap.rfc7231\n{:media-range \"*/*\",\n :type \"*\",\n :subtype \"*\",\n :parameters {},\n :qvalue 0.8}\n\n#:juxt.reap.rfc7231\n{:media-range \"application/signed-exchange\",\n :type \"application\",\n :subtype \"signed-exchange\",\n :parameters {\"v\" \"b3\"},\n :qvalue 0.9}\n----\n\n*reap* contains parsers for most things you'd want to parse when writing web\napplications, so you can focus on writing your app without worrying about\nwriting parsers. It's fast too, so you don't have to worry about a performance\nimpact.\n\n== Introduction\n\nThe Internet is a system of interoperable computer software written to\na set of exacting specifications\n(https://tools.ietf.org/rfc/index[RFCs]) published by the\nhttps://www.ietf.org/[Internet Engineering Task Force].\n\nMany Internet protocols, notably HTTP, are textual in nature.\n\nSoftware components of the Internet must be able to efficiently encode and\ndecode strings of text accurately in order to process correctly.\n\n=== Problem Statement\n\nThere are not many tools of sufficient quality which can help with the decoding\nand encoding of text strings, especially those defined in RFCs.\n\nTherefore, programmers are often left to write their own 'quick and dirty'\ncode. This leads to software that does not properly implement (and is not fully\nconformant with) the rules defined in the RFCs.\n\nProgrammers often have to strike a balance between conforming to the\nrules layed down by the RFCs and competing priorities such as meeting\nperformance requirements and project deadlines.\n\nUnfortunately, code that violates any aspect of a specification can\nlead to an unhealthy Internet. Time is wasted debugging\ninteroperability problems, buggy implementations cause problems for\nusers and lead to, in some cases, security vulnerabilities.\n\n=== Example: the HTTP Accept header\n\nIn RFC 7231 (which defines part of HTTP), the `Accept`\nheader is specified by the following rule:\n\n[source]\n----\nAccept = [ ( \",\" / ( media-range [ accept-params ] ) ) *( OWS \",\" [\n    OWS ( media-range [ accept-params ] ) ] ) ]\n----\n\nAs well as indicating the ways that various punctuation and other characters can\nbe combined, the rule makes reference to other rules, such as `media-range`:\n\n[source]\n----\nmedia-range = ( \"*/*\" / ( type \"/*\" ) / ( type \"/\" subtype ) ) *( OWS\n    \";\" OWS parameter )\n----\n\nA `type` here is a `token`, defined in another RFC (RFC 7230), which\nstates a `token` is a sequence of at least one `tchar`:\n\n[source]\n----\ntoken = 1*tchar\ntchar = \"!\" / \"#\" / \"$\" / \"%\" / \"\u0026\" / \"'\" / \"*\" / \"+\" / \"-\" / \".\" /\n    \"^\" / \"_\" / \"`\" / \"|\" / \"~\" / DIGIT / ALPHA\n----\n\nLet's leave aside DIGIT and ALPHA and return to the `parameter` rule,\nwhich itself is non-trivial:\n\n[source]\n----\nparameter = token \"=\" ( token / quoted-string )\n----\n\nThe rule tells us that values can be tokens, but can _alternatively_\nbe separated by quotation marks:\n\n[source]\n----\nquoted-string = DQUOTE *( qdtext / quoted-pair ) DQUOTE\n----\n\nWhat is contained within these quotation marks is subject to further\nexacting rules about which characters and character ranges are valid\nand how characters can be escaped by using ``quoted-pair``s:\n\n[source]\n----\nqdtext = HTAB / SP / \"!\" / %x23-5B ; '#'-'['\n    / %x5D-7E ; ']'-'~'\n    / obs-text\nobs-text = %x80-FF\nquoted-pair = \"\\\" ( HTAB / SP / VCHAR / obs-text )\n----\n\nA `media-range`, itself containing parameters (where values are required) can be\noptionally followed by a special parameter indicating the term's `weight`,\noptionally followed by further parameters (where values are optional), called\naccept extensions.\n\nThese are the rules for just one HTTP request header, and it's by far\nfrom the most complex!\n\nSo it's no surprise that programmers who resort to writing custom\nparsing code might skip a few details.\n\n\n=== Ingredients\n\n*reap* is built from some old ideas.\n\n==== Lisp (1958)\n\nClojure is used as the implementation language to facilitate faster research and prototyping.\nIf this project proves useful/stable it might be a good idea to port to Java and provide a Clojure wrapper.\n\n==== Regular Expressions (1950s)\n\n*reap* uses https://en.wikipedia.org/wiki/Regular_expressions[regular expressions] to parse terminals.\n\n==== Allen's Interval Algebra (1983)\n\nhttps://en.wikipedia.org/wiki/Allen's_interval_algebra[Allen's interval algebra] allows character intervals to be manipulated and combined, to form optimal ranges which optimise the performance of the regular expression.\n\n==== Parser Combinators (1989)\n\nhttps://en.wikipedia.org/wiki/Parser_combinator[Parser combinators] are used to combine parsers built from regular expressions.\n\nSee\n\n==== Parsing Expression Grammars (2004)\n\n*reap* uses a technique known as https://en.wikipedia.org/wiki/Parsing_expression_grammar[Parsing expression grammar (PEG)].\nThere are other dedicated PEG parsing libraries, including https://github.com/ericnormand/squarepeg[squarepeg] and https://github.com/aroemers/crustimoney[crustimoney].\nReap is focussed on the practical problem of parsing real-world strings found in web protocols, rather than providing a general PEG parsing library.\nWe don't currently support packrat caching (memoization), although that may be added in the future.\n\nThe alternative to PEG is a Context Free Grammar. There are a number of excellent tools for generating CFG parsers, from venerable ones such as flex/bison to more modern ones including https://www.antlr.org/[Antlr].\n\nIn the Clojure eco-system, we have https://github.com/aphyr/clj-antlr[clj-antlr] and https://github.com/Engelberg/instaparse[Instaparse].\n\nNote, however, this useful comparison between PEG and CFG parsers.\n\n[quote, Bryan Ford, https://bford.info/pub/lang/peg.pdf]\n____\nChomsky’s generative system of grammars, from which the ubiqui-\ntous context-free grammars (CFGs) and regular expressions (REs)\narise, was originally designed as a formal tool for modelling and\nanalyzing natural (human) languages. Due to their elegance and\nexpressive power, computer scientists adopted generative grammars\nfor describing machine-oriented languages as well. The ability of\na CFG to express ambiguous syntax is an important and powerful\ntool for natural languages. Unfortunately, this power gets in the\nway when we use CFGs for machine-oriented languages that are\nintended to be precise and unambiguous. Ambiguity in CFGs is\ndifficult to avoid even when we want to, and it makes general CFG\nparsing an inherently super-linear-time problem.\n____\n\n== User Guide\n\nFunctions marked with the metadata tag `:juxt.reap/codec` take an 'options' argument and return a map of entries.\n\n`:juxt.reap/decode`:: A single-arity parser function, taking a\n`java.util.regex.Matcher` as the only argument and returning a Clojure map or\nsequence.\n\n`:juxt.reap/encode`:: A single-arity function, taking a Clojure map or sequence\nand returning a string.\n\n=== Options\n\nThe 'options' argument is a map containing the following optional entries:\n\n`:juxt.reap/decode-preserve-case`:: Set to true to prevent the parser from transforming tokens that are treated as case-insensitive to lower-case. This lossy transformation simplifies case-insensitive comparisons. Defaults to nil (false).\n\n`:juxt.reap/encode-case-transform`:: Set to `:lower` to transform generated tokens to lower-case, where applicable (where the token is semantically case-insensitive). Set to `:canonical` to transform tokens and header values to their canonical case. Defaults to nil.\n\n== References\n\nhttps://tools.ietf.org/html/rfc7230[Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing]\n\nhttps://tools.ietf.org/html/rfc7231[Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content]\n\nhttps://tools.ietf.org/html/rfc7232[Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests]\n\nhttps://tools.ietf.org/html/rfc7233[Hypertext Transfer Protocol (HTTP/1.1): Range Requests]\n\nhttps://tools.ietf.org/html/rfc7234[Hypertext Transfer Protocol (HTTP/1.1): Caching]\n\nhttps://tools.ietf.org/html/rfc7235[Hypertext Transfer Protocol (HTTP/1.1): Authentication]\n\nhttps://github.com/Engelberg/instaparse[Instaparse]\n\nhttps://github.com/Engelberg/instaparse/blob/master/docs/ABNF.md[Instaparse: ABNF Input Format]\n\nhttps://cse.unl.edu/~choueiry/Documents/Allen-CACM1983.pdf[Maintaining Knowledge about Temporal Intervals, James F. Allen]\n\nhttps://bford.info/pub/lang/peg.pdf[Parsing Expression Grammars: A Recognition-Based Syntactic Foundation]\n\nhttps://github.com/ericnormand/squarepeg[squarepeg]: Eric Normand's PEG parsing library (Clojure)\n\nhttps://github.com/aroemers/crustimoney[crustimoney]: Arnout Roemers' Clojure library for \"PEG parsing, supporting various grammars, packrat caching and cuts.\"\n\nhttps://www.infoq.com/presentations/Parser-Combinators/[Parser Combinators: How to Parse (nearly) Anything]: Nate Young's Strange Look talk.\n\n== License\n\nThe MIT License (MIT)\n\nCopyright © 2020-2024 JUXT LTD.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of\nthis software and associated documentation files (the \"Software\"), to deal in\nthe Software without restriction, including without limitation the rights to\nuse, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of\nthe Software, and to permit persons to whom the Software is furnished to do so,\nsubject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS\nFOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR\nCOPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER\nIN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN\nCONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuxt%2Freap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuxt%2Freap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuxt%2Freap/lists"}