{"id":49225441,"url":"https://github.com/tonicai/json-flattener","last_synced_at":"2026-04-24T07:01:58.174Z","repository":{"id":310038893,"uuid":"1038428588","full_name":"TonicAI/json-flattener","owner":"TonicAI","description":"A Java application that reads JSONL (JSON Lines) files and converts nested JSON structures into a flat representation with detailed structure markers.","archived":false,"fork":false,"pushed_at":"2025-08-15T09:34:04.000Z","size":74,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-15T11:22:48.374Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TonicAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-15T07:17:45.000Z","updated_at":"2025-08-15T09:34:08.000Z","dependencies_parsed_at":"2025-08-15T11:23:11.060Z","dependency_job_id":"0e2236cf-bd40-4e85-8b68-327f0a1d0cea","html_url":"https://github.com/TonicAI/json-flattener","commit_stats":null,"previous_names":["tonicai/json-flattener"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/TonicAI/json-flattener","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TonicAI%2Fjson-flattener","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TonicAI%2Fjson-flattener/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TonicAI%2Fjson-flattener/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TonicAI%2Fjson-flattener/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TonicAI","download_url":"https://codeload.github.com/TonicAI/json-flattener/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TonicAI%2Fjson-flattener/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32212808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T03:15:14.334Z","status":"ssl_error","status_checked_at":"2026-04-24T03:15:11.608Z","response_time":64,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-24T07:01:57.102Z","updated_at":"2026-04-24T07:01:58.156Z","avatar_url":"https://github.com/TonicAI.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ostia - JSON Flattening Tool\n\nA Java application that converts nested JSON structures into a flat representation with detailed structure markers.\n\n## Two Modes of Operation\n\n### Mode 1: Transform Local Files\n\nProcess existing JSONL (JSON Lines) files on your filesystem.\n\n### Mode 2: Generate \u0026 Transform with Fabricate\n\nConnect to Fabricate API to generate synthetic data, then automatically flatten it.\n\n## Output Format\n\nThe tool converts JSON data into a structured format with the following markers:\n\n- **Objects**: Start with `\"value\": \"Structure\"` and end with `\"value\": \"EndStructure\"`\n- **Arrays**: Start with the array length as value and end with `\"value\": \"EndArray\"`\n- **Primitives**: Direct key-value pairs\n\n### Example\n\nInput JSONL:\n\n```json\n{\n  \"name\": \"John\",\n  \"location\": { \"city\": \"New York\" },\n  \"nicknames\": [\"Jon-boy\", \"Johnny\"]\n}\n```\n\nOutput:\n\n```json\n[\n  {\n    \"fields\": [\n      { \"key\": \"name\", \"value\": \"John\" },\n      { \"key\": \"location\", \"value\": \"Structure\" },\n      { \"key\": \"location.city\", \"value\": \"New York\" },\n      { \"key\": \"location.\", \"value\": \"EndStructure\" },\n      { \"key\": \"nicknames\", \"value\": \"2\" },\n      { \"key\": \"nicknames[0]\", \"value\": \"Jon-boy\" },\n      { \"key\": \"nicknames[1]\", \"value\": \"Johnny\" },\n      { \"key\": \"nicknames.\", \"value\": \"EndArray\" }\n    ]\n  }\n]\n```\n\n## Getting Started\n\n### Prerequisites\n\n- Java 17 or later\n- Maven 3.6 or later\n\n### Build the Project\n\n```bash\nmvn clean compile\n```\n\n### Package the Application\n\nCreate a JAR file with all dependencies:\n\n```bash\nmvn clean package\n```\n\n### Run Tests\n\n```bash\nmvn test\n```\n\n### Run the Packaged JAR\n\nAfter building, you can also run the application directly from the JAR file:\n\n```bash\njava -jar target/json-flattener-1.0.0.jar path/to/file.jsonl\n```\n\n## Quick Start\n\n**Transform a local file:**\n\n```bash\nmvn exec:java -Dexec.mainClass=\"ai.tonic.fabricate.tools.App\" -Dexec.args=\"path/to/file.jsonl\"\n\n# Or using the shorter command:\nmvn exec:java -Dexec.args=\"path/to/file.jsonl\"\n```\n\n**Generate data with Fabricate:**\n\n```bash\n# Set up .env file first (see Configuration section)\nmvn exec:java -Pfabricate\n```\n\n## Usage\n\n### Mode 1: Transform Local Files\n\nProcess existing JSONL files on your filesystem:\n\n```bash\nmvn exec:java -Dexec.mainClass=\"ai.tonic.fabricate.tools.App\" -Dexec.args=\"path/to/your/file.jsonl\"\n\n# Or using the shorter command:\nmvn exec:java -Dexec.args=\"path/to/your/file.jsonl\"\n```\n\n**Example:**\n\n```bash\nmvn exec:java -Dexec.args=\"/Users/john/data/customers.jsonl\"\n```\n\nThis mode requires:\n\n- ✅ A valid JSONL file path\n- ❌ No Fabricate API configuration needed\n\n### Mode 2: Generate \u0026 Transform with Fabricate\n\nGenerate synthetic data via Fabricate API and automatically flatten it:\n\n```bash\n# Uses entity from .env file\nmvn exec:java -Pfabricate\n```\n\nThis mode requires:\n\n- ✅ Fabricate API configuration (see Configuration section below)\n- ✅ Valid API key and workspace access\n- ❌ No local files needed\n\n## Configuration (Mode 2 Only)\n\n⚠️ **Note**: Configuration is only required for **Mode 2** (Fabricate API integration). **Mode 1** (local file processing) doesn't need any configuration.\n\nThe application supports loading configuration from environment variables or a `.env` file for Fabricate API integration.\n\n### Environment Variables\n\nCreate a `.env` file in the project root or set these environment variables:\n\n```bash\n# Required\nFABRICATE_API_KEY=sk-your-api-key-here\nWORKSPACE=your-workspace-name\nDATABASE=your-database-name\nENTITY=users\n\n# Optional\nFABRICATE_URI_BASE=http://localhost:3000  # defaults to https://fabricate.tonic.ai\n```\n\n### Setup Instructions\n\n1. **Copy the example configuration:**\n\n   ```bash\n   cp env.example .env\n   ```\n\n2. **Edit `.env` with your actual values:**\n\n   ```bash\n   FABRICATE_API_KEY=sk-abc123xyz...\n   WORKSPACE=my-workspace\n   DATABASE=my-database\n   ENTITY=users\n   ```\n\n3. **Run the Fabricate integration:**\n\n   ```bash\n   mvn exec:java -Pfabricate\n   ```\n\n## Project Structure\n\n```\n\u003croot\u003e/\n├── src/main/java/ai/tonic/fabricate/tools/\n│   ├── App.java                  # Main application logic (Mode 1)\n│   ├── FabricateExample.java     # Fabricate integration (Mode 2)\n│   ├── JsonFlattener.java        # Core flattening logic\n│   ├── FabricateClient.java      # Fabricate API client\n│   └── EnvConfig.java            # Environment configuration\n├── data/\n│   └── example.jsonl             # Sample input file\n├── target/                       # Maven build directory\n├── pom.xml                       # Maven project configuration\n└── README.md                     # This file\n```\n\n## Dependencies\n\n- **Jackson Databind**: For JSON parsing and processing\n- **OkHttp**: HTTP client for Fabricate API calls\n- **Dotenv Java**: Environment variable loading from .env files\n- **JUnit**: For testing (test framework)\n- **Guava**: Utility libraries\n\n## Input File Format\n\nThe application expects JSONL (JSON Lines) format where each line contains a valid JSON object:\n\n```jsonl\n{\"field1\": \"value1\", \"nested\": {\"field2\": \"value2\"}}\n{\"field1\": \"value3\", \"array\": [\"item1\", \"item2\"]}\n```\n\n## Development\n\n### Core Components\n\n- **`App.java`**: Mode 1 entry point (local file processing)\n- **`FabricateExample.java`**: Mode 2 entry point (Fabricate API integration)\n- **`JsonFlattener.java`**: Core flattening logic (used by both modes)\n- **`FabricateClient.java`**: Fabricate API communication (Mode 2 only)\n- **`EnvConfig.java`**: Environment configuration (Mode 2 only)\n\n### Key Methods (JsonFlattener)\n\n- `processJsonlFile()`: Reads and processes JSONL files\n- `flattenJsonNode()`: Converts JSON nodes to flattened structure\n- `flattenNode()`: Recursively processes nested objects and arrays\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonicai%2Fjson-flattener","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftonicai%2Fjson-flattener","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonicai%2Fjson-flattener/lists"}