{"id":35120378,"url":"https://github.com/ignaciocolussi/simple_toon","last_synced_at":"2026-01-13T22:55:34.049Z","repository":{"id":330728643,"uuid":"1123755059","full_name":"ignaciocolussi/simple_toon","owner":"ignaciocolussi","description":"Python parser and serializer for TOON (Token-Oriented Object Notation) - Reduce LLM token usage by 30-60%","archived":false,"fork":false,"pushed_at":"2025-12-27T21:54:55.000Z","size":116,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-29T10:56:44.538Z","etag":null,"topics":["data-format","json","llm","parser","python","token-optimization","toon"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ignaciocolussi.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-27T14:53:05.000Z","updated_at":"2025-12-27T21:54:59.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ignaciocolussi/simple_toon","commit_stats":null,"previous_names":["ignaciocolussi/simple_toon"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ignaciocolussi/simple_toon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ignaciocolussi%2Fsimple_toon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ignaciocolussi%2Fsimple_toon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ignaciocolussi%2Fsimple_toon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ignaciocolussi%2Fsimple_toon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ignaciocolussi","download_url":"https://codeload.github.com/ignaciocolussi/simple_toon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ignaciocolussi%2Fsimple_toon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28400405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T14:36:09.778Z","status":"ssl_error","status_checked_at":"2026-01-13T14:35:19.697Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-format","json","llm","parser","python","token-optimization","toon"],"created_at":"2025-12-27T23:40:08.116Z","updated_at":"2026-01-13T22:55:34.035Z","avatar_url":"https://github.com/ignaciocolussi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple TOON Parser (Python)\n\n[![Tests](https://github.com/ignaciocolussi/simple_toon/workflows/Tests/badge.svg)](https://github.com/ignaciocolussi/simple_toon/actions)\n[![PyPI version](https://badge.fury.io/py/simple-toon.svg)](https://pypi.org/project/simple-toon/)\n[![Python versions](https://img.shields.io/pypi/pyversions/simple-toon.svg)](https://pypi.org/project/simple-toon/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nA Python parser and serializer for **TOON (Token-Oriented Object Notation)**, a compact data format designed to reduce LLM token consumption by 30-60% compared to JSON.\n\n## Installation\n\n```bash\npip install simple-toon\n```\n\n## Quick Start\n\n### Functional API (Recommended for simple use cases)\n\n```python\nfrom toon_parser import parse, stringify\n\n# Convert TOON to JSON\ntoon_data = \"\"\"\nusers[2]{id,name,active}:\n  1,Alice,true\n  2,Bob,false\n\"\"\"\njson_data = parse(toon_data)\n# Result: {\"users\": [{\"id\": 1, \"name\": \"Alice\", \"active\": true}, ...]}\n\n# Convert JSON to TOON\njson_obj = {\n    \"users\": [\n        {\"id\": 1, \"name\": \"Alice\", \"active\": True},\n        {\"id\": 2, \"name\": \"Bob\", \"active\": False}\n    ]\n}\ntoon_string = stringify(json_obj)\n```\n\n### Object-Oriented API (Recommended for complex applications)\n\n```python\nfrom toon_parser import ToonParser, ToonSerializer, ToonDocument\n\n# Create configured parser\nparser = ToonParser(advanced=True)\ndata = parser.parse(toon_string)\n\n# Create configured serializer\nserializer = ToonSerializer(advanced=True)\ntoon = serializer.stringify(data)\n\n# Work with documents\ndoc = ToonDocument.from_file(\"data.toon\")\nactive_users = doc.query(\"users\", lambda u: u[\"active\"])\ndoc.add_item(\"users\", {\"id\": 99, \"name\": \"New User\"})\ndoc.save(\"updated.toon\")\n```\n\n## Advanced Features\n\n### Nested Objects\n\nAutomatically flatten and unflatten nested objects:\n\n```python\nfrom toon_parser import stringify_advanced, parse_advanced\n\ndata = {\n    \"users\": [\n        {\"id\": 1, \"name\": \"Alice\", \"address\": {\"city\": \"NYC\", \"zip\": \"10001\"}},\n        {\"id\": 2, \"name\": \"Bob\", \"address\": {\"city\": \"LA\", \"zip\": \"90001\"}}\n    ]\n}\n\n# Serializes with dot notation: users[2]{id,name,address.city,address.zip}:\ntoon = stringify_advanced(data)\n\n# Parse restores nested structure\nresult = parse_advanced(toon)\n```\n\n### Multiple Arrays\n\nHandle multiple arrays in a single TOON document:\n\n```python\ndata = {\n    \"users\": [{\"id\": 1, \"name\": \"Alice\"}],\n    \"products\": [{\"sku\": \"A001\", \"price\": 19.99}]\n}\n\ntoon = stringify_advanced(data)\n# Both arrays in one document\nparsed = parse_advanced(toon)\n```\n\n### Streaming Parser \u0026 Serializer\n\nMemory-efficient operations for large datasets:\n\n```python\nfrom toon_parser import stream_parse, StreamingSerializer\n\n# Streaming parser (read large files)\nfor array_name, items in stream_parse(large_toon_data):\n    print(f\"Processing {array_name}: {len(items)} items\")\n    for item in items:\n        process(item)  # Process one at a time\n\n# Streaming serializer (write large files)\nwith StreamingSerializer(\"output.toon\") as writer:\n    writer.begin_array(\"users\", [\"id\", \"name\", \"email\"])\n\n    for user in database.query_users():  # Stream from DB\n        writer.write_row([user.id, user.name, user.email])\n\n    writer.end_array()\n```\n\n### Custom Configuration\n\n```python\nfrom toon_parser import ToonConfig, stringify_advanced\n\nconfig = ToonConfig(\n    separator=\"_\",      # Use underscore instead of dot\n    indent_size=4,      # 4-space indentation\n    max_nesting_depth=5 # Maximum nesting levels\n)\n\ntoon = stringify_advanced(data, config)\n```\n\n### Schema Validation\n\nDefine and validate data schemas:\n\n```python\nfrom toon_parser import Field, FieldType, Schema, infer_schema\n\n# Define schema manually\nschema = Schema(\"users\", [\n    Field(\"id\", FieldType.INTEGER),\n    Field(\"name\", FieldType.STRING),\n    Field(\"email\", FieldType.STRING, pattern=r\"^[\\w\\.-]+@[\\w\\.-]+\\.\\w+$\"),\n    Field(\"age\", FieldType.INTEGER, min_value=0, max_value=120)\n])\n\n# Validate data\nschema.validate(data)\n\n# Or infer schema from example data\nschema = infer_schema(sample_data, \"users\")\n```\n\n### File I/O\n\nRead and write TOON files with optional validation:\n\n```python\nfrom toon_parser import read_toon, write_toon, convert_json_to_toon\n\n# Write TOON file with validation\nwrite_toon(data, \"output.toon\", advanced=True, schema=schema)\n\n# Read TOON file\ndata = read_toon(\"input.toon\", advanced=True)\n\n# Convert between formats\nconvert_json_to_toon(\"input.json\", \"output.toon\")\n\n# Batch convert directory\nfrom toon_parser import batch_convert\nbatch_convert(\"json_files/\", \"toon_files/\", from_format=\"json\", to_format=\"toon\")\n\n# Get file statistics\nfrom toon_parser import get_file_stats\nstats = get_file_stats(\"data.toon\")\nprint(f\"Total items: {stats['total_items']}\")\n```\n\n## What is TOON?\n\nTOON is a token-efficient serialization format optimized for LLM input. It combines:\n- YAML-style indentation for nested objects\n- CSV-style tabular layout for uniform arrays\n- Explicit schema declarations with `[N]{field1,field2}` headers\n\n## Performance\n\n- **30-60% fewer tokens** than JSON (up to 63% with nested objects)\n- **Lossless, deterministic** round-trip conversion\n- Optimized for uniform arrays (logs, user lists, analytics events)\n- Streaming parser for memory-efficient processing of large files\n\n### Benchmarks\n\n| Dataset | JSON Size | TOON Size | Savings |\n|---------|-----------|-----------|---------|\n| Simple arrays (50 items) | 3,536 chars | 1,362 chars | **61.5%** |\n| Nested objects (50 items) | 7,220 chars | 2,639 chars | **63.4%** |\n| Event data (10 items) | 845 bytes | 235 bytes | **72.2%** |\n| Multiple arrays | Varies | Varies | 30-60% |\n\n## API Reference\n\n### Functional API\n\n**Basic Functions:**\n- `parse(toon: str) -\u003e Any` - Parse TOON to JSON\n- `stringify(obj: Any) -\u003e str` - Serialize JSON to TOON\n\n**Advanced Functions:**\n- `parse_advanced(toon: str, config: ToonConfig) -\u003e Any` - Parse with nested object support\n- `stringify_advanced(obj: Any, config: ToonConfig) -\u003e str` - Serialize with nested objects\n- `stream_parse(toon: str) -\u003e Iterator` - Memory-efficient streaming parser\n\n**Schema Validation:**\n- `Schema(array_name, fields)` - Define validation schema\n- `Field(name, field_type, **options)` - Define field with constraints\n- `infer_schema(data, array_name)` - Auto-generate schema from data\n- `MultiSchema(schemas)` - Validate multiple arrays\n\n**File I/O:**\n- `read_toon(path, advanced, schema)` - Read and validate TOON file\n- `write_toon(data, path, advanced, schema)` - Write and validate TOON file\n- `convert_json_to_toon(json_path, toon_path)` - Convert JSON → TOON\n- `convert_toon_to_json(toon_path, json_path)` - Convert TOON → JSON\n- `batch_convert(input_dir, output_dir)` - Batch convert files\n- `get_file_stats(path)` - Analyze file statistics\n\n**Streaming:**\n- `StreamingSerializer(output)` - Stream write large TOON files\n- `streaming_serializer(output)` - Context manager for streaming\n- `stream_from_database(query_func, ...)` - Stream from database to TOON\n\n### Object-Oriented API\n\n**Classes:**\n- `ToonParser(advanced, config, schema)` - Stateful parser\n- `ToonSerializer(advanced, config, schema)` - Stateful serializer\n- `ToonDocument(data)` - Document object model with query/manipulation methods\n- `ToonConverter(advanced, config)` - Format converter with statistics\n\n## Examples\n\nSee the example files for detailed usage:\n- `example.py` - Basic parsing and serialization (functional API)\n- `example_advanced.py` - Nested objects, multiple arrays, configuration\n- `example_schema_io.py` - Schema validation and file I/O\n- `example_oo_streaming.py` - Object-oriented API and streaming serializer\n\n## Development\n\n```bash\n# Install development dependencies\npip install -e \".[dev]\"\n\n# Run tests\npytest\n\n# Run tests with coverage\npytest --cov=toon_parser\n\n# Format code\nblack toon_parser/ tests/\n\n# Lint\nruff check toon_parser/ tests/\n\n# Type check\nmypy toon_parser/\n```\n\n## Contributing\n\nWe welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n**Quick start for contributors:**\n1. Fork the repository\n2. Create a feature branch\n3. Install dev dependencies: `pip install -e \".[dev]\"`\n4. Install pre-commit hooks: `pre-commit install`\n5. Make your changes and run tests\n6. Submit a pull request\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fignaciocolussi%2Fsimple_toon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fignaciocolussi%2Fsimple_toon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fignaciocolussi%2Fsimple_toon/lists"}