{"id":51509153,"url":"https://github.com/dagoste/aton-format","last_synced_at":"2026-07-08T04:01:41.304Z","repository":{"id":364640460,"uuid":"1103162989","full_name":"dagoSte/aton-format","owner":"dagoSte","description":"ATON Format - Python implementation. Token-efficient data serialization for LLM applications.","archived":false,"fork":false,"pushed_at":"2025-11-25T04:39:25.000Z","size":93,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-13T22:17:46.543Z","etag":null,"topics":["ai","encoder-decoder","gpt","llm","llms","token","tokenization"],"latest_commit_sha":null,"homepage":"https://www.atonformat.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dagoSte.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-24T14:08:26.000Z","updated_at":"2025-11-25T13:26:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dagoSte/aton-format","commit_stats":null,"previous_names":["dagoste/aton-format"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/dagoSte/aton-format","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagoSte%2Faton-format","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagoSte%2Faton-format/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagoSte%2Faton-format/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagoSte%2Faton-format/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dagoSte","download_url":"https://codeload.github.com/dagoSte/aton-format/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagoSte%2Faton-format/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35251017,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","encoder-decoder","gpt","llm","llms","token","tokenization"],"created_at":"2026-07-08T04:01:39.882Z","updated_at":"2026-07-08T04:01:41.297Z","avatar_url":"https://github.com/dagoSte.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ATON V2.0 - Adaptive Token-Oriented Notation\n\n[![Version](https://img.shields.io/badge/version-2.0.1-blue.svg)](https://github.com/dagoSte/aton-format)\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n[![Python](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/)\n[![PyPI](https://img.shields.io/pypi/v/aton-format.svg)](https://pypi.org/project/aton-format/)\n\n**The data serialization format designed for Large Language Models and AI applications.**\n\nATON achieves **56% token reduction** compared to JSON while providing type safety, native relationships, and human readability. Perfect for RAG systems, multi-agent architectures, and any LLM-powered application.\n\nDocumentation: \nhttps://www.atonformat.com/documentation.html\n---\n\n## Core Features\n\n| Feature | Description |\n|---------|-------------|\n| ✅ **56% Token Reduction** | Dramatically lower your LLM API costs |\n| ✅ **Type Safety** | Explicit schema with type definitions (`@schema[field:type]`) |\n| ✅ **Human Readable** | Easy to read, write, and debug by both humans and AI |\n| ✅ **Native Relationships** | Reference entities directly with `→` syntax |\n| ✅ **Smart Defaults** | Omit repetitive values automatically with `@defaults` |\n| ✅ **Zero Data Loss** | Perfect round-trip: `decode(encode(data)) === data` |\n\n---\n\n## What's New in V2.0\n\n### Advanced Compression System\n- **4 Compression Modes**: Fast, Balanced, Ultra, and Adaptive\n- **Intelligent Algorithms**: Dictionary, Delta encoding, Pattern recognition\n- **AI-Driven Selection**: Adaptive mode analyzes data and chooses optimal strategy\n\n### SQL-like Query Language\n- **Full AST Parser**: Not simplified regex, real production parser\n- **Complete Operators**: `=`, `!=`, `\u003c`, `\u003e`, `\u003c=`, `\u003e=`, `IN`, `LIKE`, `BETWEEN`\n- **Complex Logic**: `AND`, `OR`, `NOT` with nested conditions\n- **Advanced Features**: `SELECT`, `ORDER BY`, `LIMIT`, `OFFSET`\n\n### Streaming Support\n- **Memory Efficient**: Process millions of records with constant memory\n- **Schema Caching**: First chunk includes schema, subsequent chunks only data\n- **Progress Tracking**: Monitor encoding progress\n- **Configurable Chunks**: Adjust chunk size for your use case\n\n### Production-Grade Quality\n- **Custom Exceptions**: Complete error hierarchy\n- **Full Validation**: Input validation, type checking, boundary conditions\n- **Type Safety**: 100% type hints coverage\n- **Zero Compromises**: Professional code quality throughout\n\n---\n\n## Performance\n\n### Token Reduction vs JSON\n\n| Dataset | JSON Tokens | ATON Tokens | Reduction |\n|---------|-------------|-------------|-----------|\n| Employee Records (1K) | 12,450 | 5,280 | **57.6%** |\n| Product Catalog (10K) | 145,200 | 64,800 | **55.4%** |\n| Transaction Log (100K) | 1,856,000 | 815,000 | **56.1%** |\n| Real-time Agents | 42,000 | 18,500 | **56.0%** |\n\n### Compression Speed\n\n| Mode | Records/sec | Latency | Best For |\n|------|-------------|---------|----------|\n| FAST | ~50,000 | \u003c1ms | Real-time applications |\n| BALANCED | ~35,000 | \u003c2ms | General purpose (recommended) |\n| ULTRA | ~20,000 | \u003c5ms | Batch processing, storage |\n| ADAPTIVE | ~30,000 | \u003c3ms | Mixed workloads |\n\n---\n\n## Real-World Cost Savings\n\n### Use Cases\n\n| Application | Records/Hour | Annual API Savings |\n|-------------|--------------|-------------------|\n| Multi-Agent Systems | 1,000 updates | **$609,120** |\n| E-commerce Platform | 100K products | **$182,500** |\n| Analytics Dashboard | Millions of events | **$1,460,000** |\n| Customer Support (RAG) | Thousands of chats | **$584,930** |\n| Financial Trading | Real-time ticks | **$2,920,000** |\n| Healthcare Records | Patient data | **$730,000** |\n\n---\n\n## Installation\n\n### Python\n\n```bash\npip install aton-format\n```\n\n### JavaScript\n\n```bash\nnpm install aton-format\n# or\nyarn add aton-format\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/dagoSte/aton-format.git\ncd aton-format\npip install -e .\n```\n\n---\n\n## Quick Start\n\n### Basic Usage (Python)\n\n```python\nfrom aton_format import ATONEncoder, ATONDecoder, CompressionMode\n\n# Initialize encoder with compression\nencoder = ATONEncoder(\n    compression=CompressionMode.BALANCED,  # or FAST, ULTRA, ADAPTIVE\n    optimize=True\n)\n\n# Your data\ndata = {\n    \"employees\": [\n        {\"id\": 1, \"name\": \"Alice\", \"salary\": 95000, \"active\": True},\n        {\"id\": 2, \"name\": \"Bob\", \"salary\": 92000, \"active\": True},\n        {\"id\": 3, \"name\": \"Carol\", \"salary\": 110000, \"active\": False}\n    ]\n}\n\n# Encode to ATON (50-60% fewer tokens!)\naton_text = encoder.encode(data)\nprint(aton_text)\n\n# Output:\n# @schema[id:int, name:str, salary:int, active:bool]\n# @defaults[active=true]\n#\n# employees(3):\n#   1, \"Alice\", 95000, null\n#   2, \"Bob\", 92000, null\n#   3, \"Carol\", 110000, false\n\n# Decode back to original\ndecoder = ATONDecoder()\noriginal = decoder.decode(aton_text)\n\nassert data == original  # Perfect round-trip!\n```\n\n### Basic Usage (JavaScript)\n\n```javascript\nconst ATON = require('aton-format');\n\n// Initialize encoder\nconst encoder = new ATON.Encoder({\n    compression: ATON.CompressionMode.BALANCED,\n    optimize: true\n});\n\n// Your data\nconst data = {\n    employees: [\n        {id: 1, name: \"Alice\", salary: 95000, active: true},\n        {id: 2, name: \"Bob\", salary: 92000, active: true},\n        {id: 3, name: \"Carol\", salary: 110000, active: false}\n    ]\n};\n\n// Encode to ATON\nconst atonText = encoder.encode(data);\nconsole.log(atonText);\n\n// Decode back\nconst decoder = new ATON.Decoder();\nconst original = decoder.decode(atonText);\n```\n\n---\n\n## Advanced Features\n\n### 1. Query Language\n\nFilter and transform data before encoding:\n\n```python\nfrom aton_format import ATONEncoder, CompressionMode\n\nencoder = ATONEncoder(\n    compression=CompressionMode.ADAPTIVE,\n    queryable=True\n)\n\n# Complex query with SQL-like syntax\nresult = encoder.encode_with_query(\n    data,\n    \"\"\"\n    employees WHERE\n        (salary \u003e 100000 AND role = 'Engineer')\n        OR department = 'Executive'\n    ORDER BY salary DESC\n    LIMIT 20\n    \"\"\"\n)\n\n# Only relevant records encoded!\n# Massive token savings for large datasets\n```\n\n### 2. Streaming for Large Datasets\n\nProcess millions of records with constant memory:\n\n```python\nfrom aton_format import ATONStreamEncoder, CompressionMode\n\n# Initialize stream encoder\nstream_encoder = ATONStreamEncoder(\n    chunk_size=1000,\n    compression=CompressionMode.ULTRA\n)\n\n# Large dataset\ndata = {\"transactions\": large_transaction_list}\n\n# Stream encode in chunks\nfor chunk in stream_encoder.stream_encode(data):\n    print(f\"Chunk {chunk['chunk_id']}/{chunk['total_chunks']}\")\n    print(f\"Records: {chunk['metadata']['records_in_chunk']}\")\n\n    # Process chunk (send to LLM, store, etc.)\n    process_chunk(chunk['data'])\n\n    # Memory stays constant!\n```\n\n### 3. Compression Modes\n\nChoose the right strategy for your use case:\n\n```python\nfrom aton_format import ATONEncoder, CompressionMode\n\n# Fast mode - optimized for speed\nencoder = ATONEncoder(compression=CompressionMode.FAST)\n\n# Balanced mode - optimal mix (RECOMMENDED)\nencoder = ATONEncoder(compression=CompressionMode.BALANCED)\n\n# Ultra mode - maximum compression\nencoder = ATONEncoder(compression=CompressionMode.ULTRA)\n\n# Adaptive mode - AI-driven selection\nencoder = ATONEncoder(compression=CompressionMode.ADAPTIVE)\n```\n\n### 4. Query Operators\n\nFull SQL-like query support:\n\n```python\n# Comparison operators\n\"products WHERE price \u003e 100\"\n\"employees WHERE age \u003c 30\"\n\"orders WHERE total \u003e= 1000\"\n\n# Special operators\n\"users WHERE status IN ('active', 'pending')\"\n\"products WHERE name LIKE '%Premium%'\"\n\"orders WHERE total BETWEEN 100 AND 500\"\n\n# Complex logic\n\"employees WHERE (role = 'Engineer' OR role = 'Manager') AND salary \u003e 80000\"\n\n# Field selection\n\"SELECT name, email FROM users WHERE active = true\"\n\n# Sorting and pagination\n\"products ORDER BY rating DESC LIMIT 20 OFFSET 40\"\n```\n\n---\n\n## Documentation\n\n- **Complete Documentation**: [https://www.atonformat.com/documentation.html](https://www.atonformat.com/documentation.html)\n- **Technical Whitepaper**: [https://www.atonformat.com/whitepaper.html](https://www.atonformat.com/whitepaper.html)\n- **API Reference**: Full API docs with examples\n- **GitHub Repository**: [https://github.com/dagoSte/aton-format](https://github.com/dagoSte/aton-format)\n- **PyPI Package**: [https://pypi.org/project/aton-format/](https://pypi.org/project/aton-format/)\n\n---\n\n## Use Cases\n\n### 1. Multi-Agent Systems\n```python\n# Real-time agent orchestration\nencoder = ATONEncoder(compression=CompressionMode.FAST)\n\n# Encode agent states (56% fewer tokens)\nagent_state = encoder.encode({\"agents\": get_all_agents()})\n\n# Query specific agents\nactive_agents = encoder.encode_with_query(\n    {\"agents\": agents},\n    \"agents WHERE status = 'active' AND priority = 'high'\"\n)\n```\n\n### 2. E-commerce Product Search\n```python\n# Large product catalog\nproducts = load_product_catalog()  # 100K products\n\n# Filter and send only relevant products\nresults = encoder.encode_with_query(\n    {\"products\": products},\n    \"\"\"\n    products WHERE\n        category IN ('Electronics', 'Computers')\n        AND price BETWEEN 100 AND 500\n        AND name LIKE '%Premium%'\n    ORDER BY rating DESC\n    LIMIT 50\n    \"\"\"\n)\n# Only 50 relevant products sent to LLM!\n```\n\n### 3. Analytics Dashboard\n```python\n# Stream large datasets\nstream_encoder = ATONStreamEncoder(chunk_size=5000)\n\nfor chunk in stream_encoder.stream_encode(analytics_data):\n    # Process each chunk independently\n    # Memory stays constant even for millions of events\n    analyze_chunk(chunk['data'])\n```\n\n### 4. Customer Support (RAG)\n```python\n# Conversation history with filtering\nrecent_chats = encoder.encode_with_query(\n    {\"conversations\": all_conversations},\n    \"\"\"\n    conversations WHERE\n        created_date \u003e '2024-11-01'\n        AND sentiment != 'negative'\n    ORDER BY created_date DESC\n    LIMIT 100\n    \"\"\"\n)\n```\n\n---\n\n## Error Handling\n\n```python\nfrom aton_format import (\n    ATONEncoder,\n    ATONEncodingError,\n    ATONDecodingError,\n    ATONQueryError\n)\n\ntry:\n    encoder = ATONEncoder(validate=True)\n    result = encoder.encode(data)\n\nexcept ATONEncodingError as e:\n    print(f\"Encoding failed: {e}\")\n    # Handle encoding errors\n\nexcept ATONQueryError as e:\n    print(f\"Query failed: {e}\")\n    # Handle query errors\n\nexcept Exception as e:\n    print(f\"Unexpected error: {e}\")\n    # Handle unexpected errors\n```\n\n---\n\n## Best Practices\n\n### 1. Choose the Right Compression Mode\n\n```python\n# Real-time applications: Use FAST\nencoder = ATONEncoder(compression=CompressionMode.FAST)\n\n# General purpose: Use BALANCED (recommended)\nencoder = ATONEncoder(compression=CompressionMode.BALANCED)\n\n# Batch processing: Use ULTRA\nencoder = ATONEncoder(compression=CompressionMode.ULTRA)\n\n# Mixed workloads: Use ADAPTIVE\nencoder = ATONEncoder(compression=CompressionMode.ADAPTIVE)\n```\n\n### 2. Use Streaming for Large Datasets\n\n```python\n# DON'T: Load everything in memory\ndata = load_million_records()\nencoded = encoder.encode(data)  # Memory spike!\n\n# DO: Use streaming\nstream_encoder = ATONStreamEncoder(chunk_size=1000)\nfor chunk in stream_encoder.stream_encode(data):\n    process_chunk(chunk)  # Constant memory!\n```\n\n### 3. Leverage Query Language\n\n```python\n# DON'T: Send all data then filter\nall_data = encoder.encode(large_dataset)\n# LLM has to process everything...\n\n# DO: Filter before encoding\nfiltered = encoder.encode_with_query(\n    large_dataset,\n    \"employees WHERE salary \u003e 100000 LIMIT 100\"\n)\n# Only 100 relevant records sent!\n```\n\n### 4. Always Validate\n\n```python\n# Enable validation in production\nencoder = ATONEncoder(validate=True)\ndecoder = ATONDecoder(validate=True)\n\n# Test round-trip\nencoded = encoder.encode(data)\ndecoded = decoder.decode(encoded)\nassert data == decoded\n```\n\n---\n\n## Benchmarks\n\nRun benchmarks on your data:\n\n```python\nimport time\nfrom aton_format import ATONEncoder, CompressionMode\n\nmodes = [\n    CompressionMode.FAST,\n    CompressionMode.BALANCED,\n    CompressionMode.ULTRA,\n    CompressionMode.ADAPTIVE\n]\n\nfor mode in modes:\n    encoder = ATONEncoder(compression=mode)\n\n    start = time.time()\n    result = encoder.encode(data)\n    duration = time.time() - start\n\n    print(f\"{mode.value}:\")\n    print(f\"  Time: {duration:.3f}s\")\n    print(f\"  Size: {len(result)} chars\")\n    print()\n```\n\n---\n\n## Migration from V1\n\nATON V2 is **fully backward compatible** with V1:\n\n```python\n# V1 code (still works!)\nencoder = ATONEncoder()\nresult = encoder.encode(data)\n\n# V2 enhanced\nencoder = ATONEncoder(\n    compression=CompressionMode.ADAPTIVE,  # NEW\n    queryable=True,                        # NEW\n    validate=True\n)\n\n# Use new features\nresult = encoder.encode_with_query(      # NEW\n    data,\n    \"employees WHERE salary \u003e 100000\"\n)\n```\n\n---\n\n## Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n### Development Setup\n\n```bash\n# Clone repository\ngit clone https://github.com/dagoSte/aton-format.git\ncd aton-format\n\n# Install in development mode\npip install -e .[dev]\n\n# Run tests\npytest tests/\n\n# Run linting\nflake8 src/\nmypy src/\n```\n\n---\n\n## License\n\nATON Format is released under the [MIT License](LICENSE).\n\n```\nMIT License\n\nCopyright (c) 2025 Stefano D'Agostino\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n```\n\n---\n\n## Acknowledgments\n\n- Inspired by the need for efficient LLM data serialization\n- Built with focus on production-grade quality\n- Community feedback helped shape V2 features\n\n---\n\n## Contact\n\n- **Author**: Stefano D'Agostino\n- **GitHub**: [@dagoSte](https://github.com/dagoSte)\n- **Email**: dago.stefano@gmail.com\n- **Website**: [https://www.atonformat.com](https://www.atonformat.com)\n\n---\n\n## Star History\n\nIf you find ATON useful, please consider giving it a star on GitHub!\n\n[![Star History Chart](https://api.star-history.com/svg?repos=dagoSte/aton-format\u0026type=Date)](https://star-history.com/#dagoSte/aton-format\u0026Date)\n\n---\n\n**Made with love by Stefano D'Agostino**\n\n**ATON V2.0 - Production-Grade Data Serialization for LLMs**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdagoste%2Faton-format","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdagoste%2Faton-format","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdagoste%2Faton-format/lists"}