{"id":34717937,"url":"https://github.com/argahsuknesib/toon-ld","last_synced_at":"2026-01-13T20:48:55.302Z","repository":{"id":329837441,"uuid":"1119541767","full_name":"argahsuknesib/toon-ld","owner":"argahsuknesib","description":"Token Oriented Object Notation (TOON) for Linked Data","archived":false,"fork":false,"pushed_at":"2025-12-21T23:31:26.000Z","size":1050,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-23T10:39:55.823Z","etag":null,"topics":["context-window","json-ld","knowledge-graph","linked-data","llm","rag","rdf","rust","semantic-web","serialization","token-optimization","token-oriented-object-notation","wasm"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/argahsuknesib.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-19T12:43:52.000Z","updated_at":"2025-12-22T09:54:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/argahsuknesib/toon-ld","commit_stats":null,"previous_names":["argahsuknesib/toon-ld"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/argahsuknesib/toon-ld","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argahsuknesib%2Ftoon-ld","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argahsuknesib%2Ftoon-ld/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argahsuknesib%2Ftoon-ld/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argahsuknesib%2Ftoon-ld/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/argahsuknesib","download_url":"https://codeload.github.com/argahsuknesib/toon-ld/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argahsuknesib%2Ftoon-ld/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28400040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T14:36:09.778Z","status":"ssl_error","status_checked_at":"2026-01-13T14:35:19.697Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["context-window","json-ld","knowledge-graph","linked-data","llm","rag","rdf","rust","semantic-web","serialization","token-optimization","token-oriented-object-notation","wasm"],"created_at":"2025-12-25T01:16:57.552Z","updated_at":"2026-01-13T20:48:55.286Z","avatar_url":"https://github.com/argahsuknesib.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TOON-LD\n\n[![npm](https://img.shields.io/npm/v/toon-ld)](https://www.npmjs.com/package/toon-ld)\n[![PyPI](https://img.shields.io/pypi/v/toon-ld)](https://pypi.org/project/toon-ld/)\n[![Crates.io](https://img.shields.io/crates/v/toon-ld)](https://crates.io/crates/toon-ld)\n\n**Token-Oriented Object Notation for Linked Data** — A lossless **Knowledge Graph Compression** format for LLM Context Windows.\n\nTOON-LD reduces token usage by **40-60%** compared to JSON-LD, allowing you to fit twice as much structured data into your prompts for RAG (Retrieval-Augmented Generation) applications.\n\nIt works by extending standard TOON syntax with Linked Data semantics, meaning **every valid TOON-LD document is also a valid TOON document**. Base TOON parsers can process it natively, while TOON-LD processors unlock the full semantic graph.\n\n## Why TOON-LD?\n\n**The Problem:** Knowledge Graphs (JSON-LD) are incredibly verbose. Using them in RAG pipelines burns through token budgets and hits context limits fast.\n\n**The Solution:** TOON-LD acts as a compression layer. It combines the semantic expressiveness of RDF with radical token efficiency through tabular arrays. By eliminating repetitive keys and using CSV-like rows for uniform data, TOON-LD fits significantly more information into LLM context windows without losing structure.\n\n## Features\n\n- **Pure TOON Extension**: Every TOON-LD document is valid TOON (like JSON-LD extends JSON)\n- **Tabular Arrays**: Serialize arrays of objects as CSV-like rows with shared headers\n- **40-60% Token Reduction**: Fewer tokens means lower costs and more data in context\n- **Full JSON-LD Compatibility**: Round-trip conversion without data loss\n- **All JSON-LD 1.1 Keywords**: Complete support for `@context`, `@graph`, `@id`, `@type`, value nodes, etc.\n- **Cross-Platform**: Rust, WebAssembly (npm), and Python (PyPI) implementations\n- **High Performance**: Optimized serialization with automatic tabular array detection\n\n## Benchmarks\n\nReal-world token savings across different dataset sizes:\n\n| Records | JSON-LD Size | TOON-LD Size | Size Saved | Tokens Saved |\n|---------|--------------|--------------|------------|--------------|\n| 10      | 862 B        | 518 B        | **39.9%**  | **54.2%**    |\n| 100     | 8,782 B      | 5,109 B      | **41.8%**  | **56.3%**    |\n| 1,000   | 90,682 B     | 53,710 B     | **40.8%**  | **56.5%**    |\n| 10,000  | 936,682 B    | 566,711 B    | **39.5%**  | **53.4%**    |\n\n**Key takeaway**: Token savings scale well and are especially valuable for LLM context windows.\n\n### Sparsity Analysis\n\nTOON-LD's efficiency depends on data sparsity. Shape-based partitioning (enabled by default) ensures TOON-LD remains efficient even for highly heterogeneous data.\n\n![Token Efficiency Graph](docs/images/benchmark_sparsity.png)\n\n![Savings Percentage Graph](docs/images/benchmark_savings.png)\n\n- **Low Sparsity (0-30%)**: Both Union and Partition approaches save ~40-50% tokens.\n- **High Sparsity (60%+)**: Partitioning significantly outperforms the Union schema, maintaining efficiency where standard tabular formats fail.\n\n#### Token Cost Analysis\n\n**Union Schema**: High cost when `null_count` is large (sparse data).\n**Partitioned Schema**: Low cost when partitions have dense, non-overlapping fields.\n\n**Break-even point**: ~30% sparsity threshold balances both approaches.\n\n**Partitioning excels when:**\n- High field diversity (heterogeneous graphs)\n- Large datasets\n- Mixed entity types\n\n## Quick Example\n\n**JSON-LD:**\n```json\n{\n  \"@context\": {\n    \"foaf\": \"http://xmlns.com/foaf/0.1/\"\n  },\n  \"@graph\": [\n    {\"@id\": \"ex:1\", \"@type\": \"foaf:Person\", \"foaf:name\": \"Alice\", \"foaf:age\": 30},\n    {\"@id\": \"ex:2\", \"@type\": \"foaf:Person\", \"foaf:name\": \"Bob\", \"foaf:age\": 25}\n  ]\n}\n```\n\n**TOON-LD:**\n```\n@context:\n  foaf: http://xmlns.com/foaf/0.1/\n@graph[2]{@id,@type,foaf:age,foaf:name}:\n  ex:1, foaf:Person, 30, Alice\n  ex:2, foaf:Person, 25, Bob\n```\n\nNotice how object keys appear once in the header instead of repeating for each object.\n\n## How TOON-LD Extends TOON\n\nJust as JSON-LD extends JSON by adding semantic meaning to certain key names (those starting with `@`), TOON-LD extends TOON the same way:\n\n- **No new syntax**: TOON-LD uses only standard TOON syntax (objects, arrays, tabular format)\n- **Semantic interpretation**: Keys like `@context`, `@id`, `@type` have special JSON-LD meaning\n- **Full compatibility**: Any TOON parser can parse TOON-LD documents\n- **Value nodes**: Language tags and datatypes use tabular format for efficiency\n\nExample value node with language tag:\n```\ntitle[2]{@value,@language}:\n  The Hobbit,en\n  Der Hobbit,de\n```\n\nThis is standard TOON tabular syntax that base TOON parsers handle natively, while TOON-LD processors interpret it as JSON-LD value nodes.\n\n## Installation\n\n### Rust\n```toml\n[dependencies]\ntoon-ld = \"0.2\"\n```\n\n### CLI\n```bash\ncargo install toon-cli\n```\n\n### Python\n```bash\npip install toon-ld\n```\n\n### JavaScript/TypeScript\n```bash\nnpm install toon-ld\n```\n\n## Quick Start\n\n### CLI\n```bash\n# Convert JSON-LD to TOON-LD\ntoon-ld convert -i data.jsonld -o data.toon\n\n# Convert back to JSON-LD\ntoon-ld convert -i data.toon -o data.jsonld\n\n# Run benchmark\ntoon-ld benchmark --max-records 10000\n```\n\n### Rust\n```rust\nuse toon_ld::{encode, decode};\n\nlet json_ld = r#\"{\"@context\": {\"foaf\": \"http://xmlns.com/foaf/0.1/\"}, \"foaf:name\": \"Alice\"}\"#;\nlet toon = encode(json_ld)?;\nlet back = decode(\u0026toon)?;\n```\n\n### Python\n```python\nimport toon_ld\n\njson_ld = '{\"@context\": {\"foaf\": \"http://xmlns.com/foaf/0.1/\"}, \"foaf:name\": \"Alice\"}'\ntoon_str = toon_ld.encode(json_ld)\njson_str = toon_ld.decode(toon_str)\n```\n\n### JavaScript\n```javascript\nimport { encode, decode, parse, stringify } from 'toon-ld';\n\n// String conversion\nconst jsonLd = '{\"@context\": {\"foaf\": \"http://xmlns.com/foaf/0.1/\"}, \"foaf:name\": \"Alice\"}';\nconst toon = encode(jsonLd);\nconst json = decode(toon);\n\n// Object helpers\nconst data = parse(toon); // Returns JS Object\nconsole.log(data['foaf:name']); // \"Alice\"\n\nconst toonStr = stringify({ \"foaf:name\": \"Bob\" }); // Takes JS Object\n```\n \n\n## Key Concepts\n\n### Tabular Arrays\nArrays of objects share a header with field names, followed by CSV-like rows:\n```\n@context:\n  foaf: http://xmlns.com/foaf/0.1/\n  vcard: http://www.w3.org/2006/vcard/ns#\nfoaf:knows[3]{foaf:name,foaf:age,vcard:locality}:\n  Alice, 30, null\n  Bob, null, Portland\n  Carol, 28, Seattle\n```\n\n### Value Nodes\nLanguage tags and datatypes use standard TOON object or tabular syntax:\n```\n@context:\n  dc: http://purl.org/dc/terms/\n  schema: http://schema.org/\n  xsd: http://www.w3.org/2001/XMLSchema#\ndc:title:\n  @value: Bonjour\n  @language: fr\nschema:datePublished:\n  @value: \"2024-01-15\"\n  @type: xsd:date\n```\n\nOr using tabular format for multiple values:\n```\ndc:titles[2]{@value,@language}:\n  Bonjour,fr\n  Hello,en\n```\n\n### Context Support\nAutomatic URI compaction using `@context`:\n```\n@context:\n  foaf: http://xmlns.com/foaf/0.1/\nfoaf:name: Alice\n```\n\n## Project Structure\n\n- `toon-core/` - Core Rust implementation\n- `toon-cli/` - Command-line tool\n- `toon-wasm/` - WebAssembly bindings (npm)\n- `toon-py/` - Python bindings (PyPI)\n\n## Building from Source\n\n```bash\n# Build all workspace members\ncargo build --release\n\n# Run tests\ncargo test --workspace\n\n# Build WASM package\ncd toon-wasm \u0026\u0026 wasm-pack build --target web\n\n# Build Python wheel\ncd toon-py \u0026\u0026 maturin build --release\n```\n\n## License\n\nMIT License - See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fargahsuknesib%2Ftoon-ld","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fargahsuknesib%2Ftoon-ld","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fargahsuknesib%2Ftoon-ld/lists"}