{"id":13520393,"url":"https://github.com/AuracleTech/jayce","last_synced_at":"2025-03-31T16:31:25.716Z","repository":{"id":62441275,"uuid":"474376928","full_name":"AuracleTech/jayce","owner":"AuracleTech","description":"jayce is a tokenizer 🌌","archived":false,"fork":false,"pushed_at":"2024-03-16T16:22:43.000Z","size":257,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-09-18T01:05:22.076Z","etag":null,"topics":["tokenizer"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AuracleTech.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-03-26T14:48:04.000Z","updated_at":"2023-09-17T22:32:29.000Z","dependencies_parsed_at":"2023-01-25T09:50:12.684Z","dependency_job_id":"7150663c-c9f0-4127-ad5b-ac132b03768f","html_url":"https://github.com/AuracleTech/jayce","commit_stats":{"total_commits":27,"total_committers":2,"mean_commits":13.5,"dds":0.07407407407407407,"last_synced_commit":"c24863e5262e5e785239e9b0cff978f32c6ba75a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AuracleTech%2Fjayce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AuracleTech%2Fjayce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AuracleTech%2Fjayce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AuracleTech%2Fjayce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AuracleTech","download_url":"https://codeload.github.com/AuracleTech/jayce/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222404818,"owners_count":16978815,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["tokenizer"],"created_at":"2024-08-01T05:02:19.191Z","updated_at":"2024-11-02T03:31:31.186Z","avatar_url":"https://github.com/AuracleTech.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# jayce\n\njayce is a tokenizer 🌌\n\n##### Example\n\n```rust\nuse jayce::{Duo, Tokenizer};\nuse std::sync::OnceLock;\n\nconst SOURCE: \u0026str = \"Excalibur = 5000$; // Your own language!\";\n\nfn duos() -\u003e \u0026'static Vec\u003cDuo\u003c\u0026'static str\u003e\u003e {\n    static DUOS: OnceLock\u003cVec\u003cDuo\u003c\u0026'static str\u003e\u003e\u003e = OnceLock::new();\n    DUOS.get_or_init(|| {\n        vec![\n            Duo::new(\"whitespace\", r\"^[^\\S\\n]+\", false),\n            Duo::new(\"commentLine\", r\"^//(.*)\", false),\n            Duo::new(\"commentBlock\", r\"^/\\*(.|\\n)*?\\*/\", false),\n            Duo::new(\"newline\", r\"^\\n\", false),\n            Duo::new(\"price\", r\"^[0-9]+\\$\", true),\n            Duo::new(\"semicolon\", r\"^;\", true),\n            Duo::new(\"operator\", r\"^=\", true),\n            Duo::new(\"name\", r\"^[a-zA-Z_]+\", true),\n        ]\n    })\n}\n\nfn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    let mut tokenizer = Tokenizer::new(SOURCE, duos());\n\n    while let Some(token) = tokenizer.consume()? {\n        println!(\"{:?}\", token);\n    }\n\n    Ok(())\n}\n```\n\n##### Result\n\n```rust,ignore\nToken { kind: \"name\",      value: \"Excalibur\", pos: (1, 1)  }\nToken { kind: \"operator\",  value: \"=\",         pos: (1, 11) }\nToken { kind: \"price\",     value: \"5000$\",     pos: (1, 13) }\nToken { kind: \"semicolon\", value: \";\",         pos: (1, 18) }\n```\n\n##### Info\n\n`Tokenizer::consume` returns `Result Option Token`\n\n1. `Ok Some` match found\n2. `Ok None` end of source\n3. `Err` an error occurs\n\n`Tokenizer::consume_all` returns `Result Vec Token`\n\n1. `Ok Vec Token` tokens matched\n2. `Err` an error occurs\n\n##### Performances\n\ninitialization in ~`3 nanoseconds`\ntokenization of [Yuumi](https://github.com/AuracleTech/yuumi) in ~`4 milliseconds`\n\n##### Features\n\n- `generic-simd`\n- `runtime-dispatch-simd` default enabled, to disable modify `Cargo.toml` as follows\n\n```toml\njayce = { version = \"X.X.X\", default-features = false }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAuracleTech%2Fjayce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAuracleTech%2Fjayce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAuracleTech%2Fjayce/lists"}