{"id":18616709,"url":"https://github.com/sile/erl_tokenize","last_synced_at":"2025-10-30T11:59:57.972Z","repository":{"id":21274432,"uuid":"92060689","full_name":"sile/erl_tokenize","owner":"sile","description":"An Erlang source code tokenizer written in Rust.","archived":false,"fork":false,"pushed_at":"2025-02-16T11:58:58.000Z","size":160,"stargazers_count":12,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-07T17:17:26.469Z","etag":null,"topics":["erlang","lexer","rust","tokenizer"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sile.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-22T14:06:46.000Z","updated_at":"2025-03-30T22:23:46.000Z","dependencies_parsed_at":"2024-03-10T03:21:16.965Z","dependency_job_id":"8724ac99-7904-4f56-a2dd-435f4e2f6fed","html_url":"https://github.com/sile/erl_tokenize","commit_stats":{"total_commits":143,"total_committers":3,"mean_commits":"47.666666666666664","dds":"0.027972027972028024","last_synced_commit":"bb625ec274e807a3eb9f97a04f8b0ac42ccb4a73"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sile%2Ferl_tokenize","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sile%2Ferl_tokenize/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sile%2Ferl_tokenize/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sile%2Ferl_tokenize/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sile","download_url":"https://codeload.github.com/sile/erl_tokenize/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248325274,"owners_count":21084900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erlang","lexer","rust","tokenizer"],"created_at":"2024-11-07T03:37:40.139Z","updated_at":"2025-10-30T11:59:52.937Z","avatar_url":"https://github.com/sile.png","language":"Rust","readme":"erl_tokenize\n============\n\n[![erl_tokenize](https://img.shields.io/crates/v/erl_tokenize.svg)](https://crates.io/crates/erl_tokenize)\n[![Documentation](https://docs.rs/erl_tokenize/badge.svg)](https://docs.rs/erl_tokenize)\n[![Actions Status](https://github.com/sile/erl_tokenize/workflows/CI/badge.svg)](https://github.com/sile/erl_tokenize/actions)\n![License](https://img.shields.io/crates/l/erl_tokenize)\n\nErlang source code tokenizer written in Rust.\n\n[Documentation](https://docs.rs/erl_tokenize)\n\nExamples\n--------\n\nTokenizes the Erlang code `io:format(\"Hello\").`:\n\n```rust\nuse erl_tokenize::Tokenizer;\n\nlet src = r#\"io:format(\"Hello\").\"#;\nlet tokenizer = Tokenizer::new(src);\nlet tokens = tokenizer.collect::\u003cResult\u003cVec\u003c_\u003e, _\u003e\u003e().unwrap();\n\nassert_eq!(tokens.iter().map(|t| t.text()).collect::\u003cVec\u003c_\u003e\u003e(),\n           [\"io\", \":\", \"format\", \"(\", r#\"\"Hello\"\"#, \")\", \".\"]);\n```\n\nExecutes the example `tokenize` command:\n\n```bash\n$ cargo run --example tokenize -- /dev/stdin \u003c\u003cEOS\n-module(foo).\n\n-export([bar/0]).\n\nbar() -\u003e qux.\nEOS\n\n[Position { filepath: None, offset: 0, line: 1, column: 1 }] Symbol(Hyphen)\n[Position { filepath: None, offset: 1, line: 1, column: 2 }] Atom(\"module\")\n[Position { filepath: None, offset: 7, line: 1, column: 8 }] Symbol(OpenParen)\n[Position { filepath: None, offset: 8, line: 1, column: 9 }] Atom(\"foo\")\n[Position { filepath: None, offset: 11, line: 1, column: 12 }] Symbol(CloseParen)\n[Position { filepath: None, offset: 12, line: 1, column: 13 }] Symbol(Dot)\n[Position { filepath: None, offset: 13, line: 1, column: 14 }] Whitespace(Newline)\n[Position { filepath: None, offset: 14, line: 2, column: 1 }] Whitespace(Newline)\n[Position { filepath: None, offset: 15, line: 3, column: 1 }] Symbol(Hyphen)\n[Position { filepath: None, offset: 16, line: 3, column: 2 }] Atom(\"export\")\n[Position { filepath: None, offset: 22, line: 3, column: 8 }] Symbol(OpenParen)\n[Position { filepath: None, offset: 23, line: 3, column: 9 }] Symbol(OpenSquare)\n[Position { filepath: None, offset: 24, line: 3, column: 10 }] Atom(\"bar\")\n[Position { filepath: None, offset: 27, line: 3, column: 13 }] Symbol(Slash)\n[Position { filepath: None, offset: 28, line: 3, column: 14 }] Integer(BigUint { data: [] })\n[Position { filepath: None, offset: 29, line: 3, column: 15 }] Symbol(CloseSquare)\n[Position { filepath: None, offset: 30, line: 3, column: 16 }] Symbol(CloseParen)\n[Position { filepath: None, offset: 31, line: 3, column: 17 }] Symbol(Dot)\n[Position { filepath: None, offset: 32, line: 3, column: 18 }] Whitespace(Newline)\n[Position { filepath: None, offset: 33, line: 4, column: 1 }] Whitespace(Newline)\n[Position { filepath: None, offset: 34, line: 5, column: 1 }] Atom(\"bar\")\n[Position { filepath: None, offset: 37, line: 5, column: 4 }] Symbol(OpenParen)\n[Position { filepath: None, offset: 38, line: 5, column: 5 }] Symbol(CloseParen)\n[Position { filepath: None, offset: 39, line: 5, column: 6 }] Whitespace(Space)\n[Position { filepath: None, offset: 40, line: 5, column: 7 }] Symbol(RightArrow)\n[Position { filepath: None, offset: 42, line: 5, column: 9 }] Whitespace(Space)\n[Position { filepath: None, offset: 43, line: 5, column: 10 }] Atom(\"qux\")\n[Position { filepath: None, offset: 46, line: 5, column: 13 }] Symbol(Dot)\n[Position { filepath: None, offset: 47, line: 5, column: 14 }] Whitespace(Newline)\nTOKEN COUNT: 29\nELAPSED: 0.007222 seconds\n```\n\nReferences\n----------\n\n- [erl_scan](http://erlang.org/doc/man/erl_scan.html) module\n- [Erlang Data Types](http://erlang.org/doc/reference_manual/data_types.html)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsile%2Ferl_tokenize","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsile%2Ferl_tokenize","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsile%2Ferl_tokenize/lists"}