{"id":46140322,"url":"https://github.com/jassielof/toonz","last_synced_at":"2026-03-02T06:00:52.705Z","repository":{"id":323537282,"uuid":"1090754852","full_name":"jassielof/toonz","owner":"jassielof","description":"A Zig implementation of the TOON (Token-Oriented Object Notation) format.","archived":false,"fork":false,"pushed_at":"2026-02-17T04:07:55.000Z","size":227,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-17T10:33:48.578Z","etag":null,"topics":["ai","llm","parser","toon","toon-format","zig"],"latest_commit_sha":null,"homepage":"https://jassielof.github.io/toonz/","language":"Zig","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jassielof.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-06T05:05:06.000Z","updated_at":"2026-02-17T04:07:59.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jassielof/toonz","commit_stats":null,"previous_names":["jassielof/ztoon","jassielof/toonz"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jassielof/toonz","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jassielof%2Ftoonz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jassielof%2Ftoonz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jassielof%2Ftoonz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jassielof%2Ftoonz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jassielof","download_url":"https://codeload.github.com/jassielof/toonz/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jassielof%2Ftoonz/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29993532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-02T01:47:34.672Z","status":"online","status_checked_at":"2026-03-02T02:00:07.342Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","llm","parser","toon","toon-format","zig"],"created_at":"2026-03-02T06:00:51.442Z","updated_at":"2026-03-02T06:00:52.696Z","avatar_url":"https://github.com/jassielof.png","language":"Zig","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TOONZ\n\nA Zig implementation of the TOON (Token-Oriented Object Notation) format - a compact, human-readable encoding of the JSON data model designed for LLM input efficiency.\n\n**[Official Specification (v3.0)](https://github.com/toon-format/spec/blob/main/SPEC.md)** | **[Format Overview](https://toonformat.dev/guide/format-overview)**\n\n## About TOON\n\nTOON combines YAML's indentation-based structure with CSV-style tabular arrays to achieve significant token reduction while maintaining lossless JSON compatibility. It's particularly efficient for uniform arrays of objects, achieving ~40% fewer tokens than JSON in mixed-structure benchmarks while improving LLM accuracy.\n\nSee the [official TypeScript implementation](js/) and the [full specification](spec/) for complete format details.\n\n## Features\n\n### Core Functionality ✅\n\n- **Encoding/Decoding**: Full JSON ↔ TOON ↔ ZON conversion\n  - `std.json.Value` ↔ TOON string serialization\n  - Parse TOON to `std.json.Value` or custom Zig types\n  - ZON (Zig Object Notation) support for native Zig integration\n\n- **Data Types**: Complete JSON data model support\n  - Primitives: strings, numbers, booleans, null\n  - Smart string quoting (only when necessary per spec)\n  - Objects: Indentation-based structure (no braces)\n  - Arrays: Inline format for primitives, multi-line for objects\n\n- **Tabular Arrays**: Optimized encoding for uniform object arrays\n  - Format: `[N]{field1,field2,...}:` header with row data\n  - Automatic tabular eligibility detection\n  - Configurable delimiters: comma `,` (default), tab `\\t`, pipe `|`\n  - Delimiter auto-detection from array headers `[N,]` / `[N\\t]` / `[N|]`\n\n### Advanced Features ✅\n\n- **Key Folding** (`key_folding = .safe`):\n  - Collapse single-key wrapper chains into dotted notation\n  - Example: `data.metadata.items` instead of nested indentation\n  - Configurable depth limit via `flatten_depth` option\n\n- **Path Expansion** (`expand_paths = .safe`):\n  - Reconstruct dotted keys into nested objects on decode\n  - Pairs with key folding for lossless round-trips\n  - Example: `data.items` → `{ \"data\": { \"items\": ... } }`\n\n- **Strict Mode Validation** (`strict = true`):\n  - Array length validation against declared `[N]` counts\n  - Tabular row count verification\n  - Field count consistency checking\n  - Malformed header detection\n\n- **Error Handling**:\n  - Comprehensive error messages with line/column information\n  - Stack overflow protection via `max_depth` limits (default: 256)\n  - Detailed diagnostics for debugging\n\n### CLI Tool ✅\n\nSmart command-line interface with automatic format detection:\n\n```bash\n# Auto-detect based on file extension\ntoonz input.json              # → TOON output\ntoonz data.toon               # → JSON output\ntoonz config.zon              # → TOON output\n\n# Explicit commands\ntoonz serialize input.json    # JSON/ZON → TOON\ntoonz deserialize data.toon   # TOON → JSON\n\n# Output format control\ntoonz data.toon --json        # → JSON\ntoonz data.toon --zon         # → ZON (Zig Object Notation)\ntoonz input.json --toon       # → TOON\n\n# File I/O\ntoonz input.json -o output.toon\necho '{\"key\":\"value\"}' | toonz serialize\n\n# Format command (coming soon)\ntoonz format data.toon        # Reformat TOON file\n```\n\n**Supported modes:**\n- File extension detection (`.json`, `.toon`, `.zon`)\n- Stdin/stdout streaming with pipes\n- Manual command specification (`serialize`, `deserialize`, `format`)\n- Output format override flags (`--json`, `--zon`, `--toon`)\n\n## Installation \u0026 Usage\n\n### Building from Source\n\n```bash\n# Clone the repository\ngit clone --recursive https://github.com/jassielof/toonz\ncd toonz\n\n# Build\nzig build\n\n# Run tests\nzig build test\n\n# Install\nzig build install\n```\n\n### Library Usage\n\n```zig\nconst std = @import(\"std\");\nconst toonz = @import(\"toonz\");\n\npub fn main() !void {\n    var gpa = std.heap.GeneralPurposeAllocator(.{}){};\n    defer _ = gpa.deinit();\n    const allocator = gpa.allocator();\n\n    // Encode: JSON → TOON\n    const json_string =\n        \\\\{\n        \\\\  \"users\": [\n        \\\\    {\"id\": 1, \"name\": \"Alice\", \"active\": true},\n        \\\\    {\"id\": 2, \"name\": \"Bob\", \"active\": false}\n        \\\\  ]\n        \\\\}\n    ;\n\n    const json_value = try std.json.parseFromSlice(\n        std.json.Value,\n        allocator,\n        json_string,\n        .{}\n    );\n    defer json_value.deinit();\n\n    const toon_output = try toonz.serialize.stringify(\n        json_value.value,\n        .{ .delimiter = ',' },  // Options: delimiter, key_folding, etc.\n        allocator\n    );\n    defer allocator.free(toon_output);\n\n    std.debug.print(\"{s}\\n\", .{toon_output});\n    // Output:\n    // users[2]{id,name,active}:\n    //   1,Alice,true\n    //   2,Bob,false\n\n    // Decode: TOON → JSON\n    const toon_input =\n        \\\\users[2]{id,name,active}:\n        \\\\  1,Alice,true\n        \\\\  2,Bob,false\n    ;\n\n    const parsed = try toonz.Parse(std.json.Value).parse(\n        allocator,\n        toon_input,\n        .{ .strict = true, .expand_paths = .off }\n    );\n    defer parsed.deinit();\n\n    // Use parsed.value as std.json.Value\n}\n```\n\n## Status \u0026 Roadmap\n\n### Completed Features ✅\n\n**Core Implementation:**\n- ✅ Full JSON ↔ TOON ↔ ZON encoding/decoding\n- ✅ All primitive types (strings, numbers, booleans, null)\n- ✅ Smart string quoting (spec-compliant minimal quoting)\n- ✅ Nested objects with indentation-based structure\n- ✅ Inline arrays for primitives\n- ✅ Multi-line arrays for complex values\n- ✅ Tabular arrays with `[N]{fields}:` format\n- ✅ Automatic tabular eligibility detection\n- ✅ Field order preservation (deterministic output)\n\n**Advanced Features:**\n- ✅ Alternative delimiters: comma `,`, tab `\\t`, pipe `|`\n- ✅ Automatic delimiter detection from headers `[N,]` / `[N\\t]` / `[N|]`\n- ✅ Key folding (`key_folding = .safe`) with configurable depth\n- ✅ Path expansion (`expand_paths = .safe`) for dotted keys\n- ✅ Strict mode validation (array lengths, field counts, headers)\n- ✅ Comprehensive error messages with line/column info\n- ✅ Stack overflow protection (`max_depth` limit)\n\n**CLI \u0026 Tooling:**\n- ✅ CLI with auto-detection (file extension, flags)\n- ✅ Stdin/stdout streaming support\n- ✅ Explicit commands (`serialize`, `deserialize`)\n- ✅ Output format selection (`--json`, `--zon`, `--toon`)\n- ✅ Help and usage information\n\n**Testing:**\n- ✅ Basic unit tests\n- ✅ JSON round-trip tests\n- ✅ Spec fixture integration (parse \u0026 stringify tests)\n- ✅ Reference implementation compatibility tests\n\n### In Progress 🚧\n\n- 🚧 **Format Command**: TOON file reformatting/prettification\n  - Parser infrastructure complete\n  - Encoder infrastructure complete\n  - CLI integration pending\n\n### Planned Enhancements 📋\n\n**Spec Compliance:**\n- 📋 Complete conformance test suite coverage\n  - All fixtures from `spec/tests/fixtures/encode/`\n  - All fixtures from `spec/tests/fixtures/decode/`\n  - Edge cases and validation scenarios\n\n**Performance:**\n- 📋 Benchmark suite (compare with JSON, official TS implementation)\n- 📋 Profile hot paths (tabular arrays, string escaping)\n- 📋 Streaming API optimization for large datasets\n- 📋 Memory allocation profiling and reduction\n\n**Delimiter Intelligence:**\n- 📋 Auto-select optimal delimiter based on content analysis\n  - Detect delimiter conflicts in data\n  - Suggest best delimiter for token efficiency\n\n**Developer Experience:**\n- 📋 API documentation generation (`zig build docs`)\n- 📋 More examples and usage patterns\n- 📋 Integration guide for Zig projects\n\n### Comparison with Official TypeScript Implementation\n\n**Feature Parity:**\n| Feature | toonz (Zig) | @toon-format/toon (TS) | Notes |\n|---------|------------|------------------------|-------|\n| Core encode/decode | ✅ | ✅ | Full compatibility |\n| Tabular arrays | ✅ | ✅ | Same format |\n| Delimiters (`,` `\\t` `\\|`) | ✅ | ✅ | All supported |\n| Key folding | ✅ | ✅ | Safe mode |\n| Path expansion | ✅ | ✅ | Safe mode |\n| Strict validation | ✅ | ✅ | Array lengths, fields |\n| Streaming API | ⚠️ | ✅ | Basic support, needs enhancement |\n| Format command | 🚧 | ✅ | In progress |\n| CLI stats/benchmarks | ❌ | ✅ | Planned |\n\n**Zig-Specific Features:**\n- ✅ ZON (Zig Object Notation) support\n- ✅ Native Zig type integration\n- ✅ Comptime validation\n- ✅ Stack overflow protection (not in spec, safety feature)\n\n### Known Limitations\n\n**Current:**\n- Format command not yet implemented (CLI stub exists)\n- Streaming API less mature than TypeScript version\n- No built-in benchmarking/stats in CLI (use external tools)\n\n**Design Choices:**\n- `max_depth` default is 256 (prevents stack overflow on malicious input)\n  - TypeScript relies on JS engine stack limits\n  - This is a safety feature, not a limitation\n\n## Testing\n\nTests use the official spec fixtures from the `spec/` submodule:\n\n```bash\n# Run all tests\nzig build test\n\n# Tests include:\n# - Basic encode/decode round-trips\n# - JSON compatibility\n# - Spec fixture conformance (spec/tests/fixtures/)\n# - Reference implementation comparison\n```\n\nTest fixtures are organized by:\n- `spec/tests/fixtures/encode/` - Encoding (JSON → TOON) tests\n- `spec/tests/fixtures/decode/` - Decoding (TOON → JSON) tests\n\nEach fixture tests specific features: tabular arrays, delimiters, key folding, edge cases, etc.\n\n## API Reference\n\n### Serialize (JSON/ZON → TOON)\n\n```zig\nconst toonz = @import(\"toonz\");\n\n// Encode std.json.Value to TOON\nconst toon_string = try toonz.serialize.stringify(\n    json_value,        // std.json.Value\n    .{                 // Options\n        .indent = 2,\n        .delimiter = ',',\n        .key_folding = .safe,\n        .flatten_depth = null,  // No limit\n    },\n    allocator\n);\n\n// Encode to writer (streaming)\ntry toonz.serialize.stringifyToWriter(\n    json_value,\n    options,\n    writer,\n    allocator\n);\n\n// Main API struct\nconst Stringify = toonz.Stringify;\nconst result = try Stringify.value(json_value, options, allocator);\n```\n\n**Options:**\n- `indent: u64` - Number of spaces for indentation (default: `2`)\n- `delimiter: ?u8` - Delimiter for arrays: `,`, `\\t`, `|` (default: `,`)\n- `key_folding: enum { off, safe }` - Collapse single-key chains (default: `.off`)\n- `flatten_depth: ?u64` - Max folding depth, `null` = unlimited (default: `null`)\n\n### Deserialize (TOON → JSON/Value)\n\n```zig\nconst toonz = @import(\"toonz\");\n\n// Parse TOON to std.json.Value\nconst parsed = try toonz.Parse(std.json.Value).parse(\n    allocator,\n    toon_input,        // []const u8\n    .{                 // Options\n        .indent = null,       // Auto-detect\n        .strict = true,\n        .expand_paths = .safe,\n        .max_depth = 256,\n    }\n);\ndefer parsed.deinit();\n// Use: parsed.value\n\n// Parse to custom Zig type\nconst MyStruct = struct {\n    name: []const u8,\n    age: u32,\n};\nconst parsed_struct = try toonz.Parse(MyStruct).parse(\n    allocator,\n    toon_input,\n    .{}\n);\ndefer parsed_struct.deinit();\n```\n\n**Options:**\n- `indent: ?usize` - Expected indentation, `null` = auto-detect (default: `null`)\n- `strict: ?bool` - Enforce strict validation (default: `true`)\n- `expand_paths: enum { off, safe }` - Expand dotted keys (default: `.off`)\n- `max_depth: usize` - Max nesting depth for safety (default: `256`)\n\n### Value Type\n\nThe internal `Value` type represents TOON/JSON values:\n\n```zig\nconst toonz = @import(\"toonz\");\nconst Value = toonz.Value;\n\nconst value = Value{\n    .object = std.StringHashMap(Value).init(allocator),\n};\n\n// Types: .null, .bool, .number, .string, .array, .object\nswitch (value) {\n    .object =\u003e |obj| // std.StringHashMap(Value)\n    .array =\u003e |arr| // std.ArrayList(Value)\n    .string =\u003e |s| // []const u8\n    .number =\u003e |n| // f64\n    .bool =\u003e |b| // bool\n    .null =\u003e // void\n}\n\n// Cleanup\nvalue.deinit(allocator);\n```\n\n## Project Structure\n\n```\ntoonz/\n├── src/\n│   ├── lib/               # Core library\n│   │   ├── root.zig       # Public API exports\n│   │   ├── Value.zig      # Value type definition\n│   │   ├── serialize/     # Encoding (JSON → TOON)\n│   │   │   ├── root.zig\n│   │   │   ├── Options.zig\n│   │   │   ├── encoders.zig\n│   │   │   ├── folding.zig\n│   │   │   └── ...\n│   │   ├── deserialize/   # Decoding (TOON → JSON)\n│   │   │   ├── root.zig\n│   │   │   ├── Parse.zig\n│   │   │   ├── Scanner.zig\n│   │   │   ├── expand.zig\n│   │   │   └── types/\n│   │   └── format/        # Formatting/prettification\n│   ├── cli/               # Command-line tool\n│   │   ├── main.zig\n│   │   └── commands/\n│   │       ├── serialize.zig\n│   │       ├── deserialize.zig\n│   │       └── format.zig\n│   └── tests/             # Test suite\n│       ├── suite.zig\n│       ├── basic.zig\n│       ├── json.zig\n│       └── spec/          # Spec fixture tests\n├── spec/                  # Official spec submodule\n│   ├── SPEC.md\n│   └── tests/fixtures/\n├── js/                    # Official TS implementation\n│   └── packages/\n├── build.zig\n└── README.md\n```\n\n## Specification Compliance\n\nThis implementation follows the **[TOON Specification v3.0](https://github.com/toon-format/spec/blob/main/SPEC.md)**.\n\nKey spec sections implemented:\n- **§3**: Encoding Normalization (Reference Encoder)\n- **§4**: Decoding Interpretation (Reference Decoder)\n- **§6**: Header Syntax (`[N]{fields}:` format)\n- **§7**: Strings and Keys (smart quoting rules)\n- **§8**: Objects (indentation-based structure)\n- **§9**: Arrays (inline and tabular formats)\n- **§11**: Delimiters (comma, tab, pipe with detection)\n- **§13**: Conformance and Options (strict mode, folding, expansion)\n\n**Differences from spec:**\n- Added `max_depth` safety limit (not in spec, prevents stack overflow)\n- ZON support is a Zig-specific extension\n\n## Contributing\n\nContributions are welcome! This implementation aims for full spec compliance.\n\n**Priority areas:**\n1. Complete conformance test coverage\n2. Format command implementation\n3. Performance optimizations\n4. Documentation improvements\n\n**Development:**\n```bash\n# Clone with submodules\ngit clone --recursive https://github.com/jassielof/toonz\ncd toonz\n\n# Build and test\nzig build\nzig build test\n\n# Generate docs\nzig build docs\n```\n\nPlease ensure tests pass before submitting PRs.\n\n## License\n\nMIT License - See [LICENSE](LICENSE) file for details.\n\n## Related Projects\n\n- **Official Spec**: [toon-format/spec](https://github.com/toon-format/spec)\n- **TypeScript (reference)**: [toon-format/toon](https://github.com/toon-format/toon) (this repo's `js/` submodule)\n- **Other implementations**: See the [official README](js/README.md#other-implementations) for Python, Rust, Go, .NET, and more\n\n## Credits\n\n- **TOON Format**: Created by [Johann Schopplich](https://github.com/johannschopplich)\n- **Zig Implementation**: [Jassiel Ovando](https://github.com/jassielof)\n- **Specification**: [toon-format/spec](https://github.com/toon-format/spec)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjassielof%2Ftoonz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjassielof%2Ftoonz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjassielof%2Ftoonz/lists"}