{"id":25174921,"url":"https://github.com/mongrov/timon","last_synced_at":"2025-05-06T21:10:11.030Z","repository":{"id":257811660,"uuid":"867991423","full_name":"mongrov/timon","owner":"mongrov","description":"Efficient local storage and Amazon S3-compatible data synchronization for time-series data,leveraging Parquet for storage and DataFusion for querying, all wrapped in a simple and intuitive API","archived":false,"fork":false,"pushed_at":"2025-05-05T21:08:35.000Z","size":231,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-05-06T21:10:00.263Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mongrov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-05T07:23:23.000Z","updated_at":"2025-05-05T21:08:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"b2fb64f5-a7ff-446e-8ca2-c581bcfb5510","html_url":"https://github.com/mongrov/timon","commit_stats":null,"previous_names":["mongrov/timon"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongrov%2Ftimon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongrov%2Ftimon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongrov%2Ftimon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongrov%2Ftimon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mongrov","download_url":"https://codeload.github.com/mongrov/timon/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252769425,"owners_count":21801378,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-09T12:28:15.929Z","updated_at":"2025-05-06T21:10:11.023Z","avatar_url":"https://github.com/mongrov.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Timon File \u0026 S3-Compatible Storage API\n\nThis API provides a set of functions for managing databases and tables in both local file storage and S3-compatible storage. It supports creating databases and tables, inserting data, querying using SQL, and more.\n\n## Table of Contents\n\n# Mobile based API's\n1. [File Storage Functions](#file-storage-functions)\n2. [S3-Compatible Storage Functions](#s3-compatible-storage-functions)\n3. [Function Descriptions](#function-descriptions)\n\n# Utility CLI\n1. [Get The Latest Utility Build](#get-the-latest-utility-build)\n2. [How To Run The Utility](#how-to-run-the-utility)\n---\n\n## File Storage Functions\n\nThese functions manage databases and tables stored locally on the file system. Data can be inserted, queried, and organized using SQL-like operations.\n\n```kotlin\n// Initialize Timon with a local storage path\nexternal fun initTimon(storagePath: String, bucketInterval: Number, userName: String): String\n\n// Create a new database\nexternal fun createDatabase(dbName: String): String\n\n// Create a new table within a specific database\nexternal fun createTable(dbName: String, tableName: String): String\n\n// List all available databases\nexternal fun listDatabases(): String\n\n// List all tables within a specific database\nexternal fun listTables(dbName: String): String\n\n// Delete a specific database\nexternal fun deleteDatabase(dbName: String): String\n\n// Delete a specific table within a database\nexternal fun deleteTable(dbName: String, tableName: String): String\n\n// Insert data into a table in JSON format\nexternal fun insert(dbName: String, tableName: String, jsonData: String): String\n\n// Query a database with SQL query\nexternal fun query(dbName: String, sqlQuery: String): String\n```\n\n## S3-Compatible Storage Functions\n\nThese functions manage data stored in an S3-compatible bucket, allowing for querying and saving daily data as Parquet files.\n\n```kotlin\n// Initialize S3-compatible storage with endpoint and credentials\nexternal fun initBucket(bucket_endpoint: String, bucket_name: String, access_key_id: String, secret_access_key: String, bucket_region: String): String\n\n// Sink daily data to Parquet format in the bucket\nexternal fun cloudSinkParquet(dbName: String, tableName: String): String\n\n// Fetch data from a given user and save it locally\nexternal fun cloudFetchParquet(userName: String, dbName: String, tableName: String, dateRange: Map\u003cString, String\u003e): String\n```\n\n## Get The Latest Utility Build\n\n### Build the Binary\nRun the following command to build the utility with the necessary features:  \n\n#### **Cross-Compile the Binary**\nRust provides tools to cross-compile your code for different platforms. This involves building the binary for a platform different from your current one.\n\n#### Example for Windows:\nOn Linux or macOS, you can compile for Windows:\n```bash\nrustup target add x86_64-pc-windows-gnu\ncargo build --features dev_cli --release --target x86_64-pc-windows-gnu\n```\n\n#### Example for macOS:\nOn Linux, you can compile for macOS:\n```bash\nrustup target add x86_64-apple-darwin\ncargo build --features dev_cli --release --target x86_64-apple-darwin\n```\n\n#### **Build Natively on Each Platform**\nIf cross-compilation is not feasible, you can build the binary on each target platform natively. This ensures compatibility.\n\n#### On macOS:\n```bash\ncargo build --release\n```\n\n#### On Windows:\n```powershell\ncargo build --release\n```\n\n#### **Use Cross (Simplified Cross-Compiling)**\nThe [`cross`](https://github.com/cross-rs/cross) tool simplifies cross-compiling by providing pre-configured Docker containers for various targets. It automatically handles dependencies and toolchains.\n\n#### Install `cross`:\n```bash\ncargo install cross\n```\n\n```bash\ncross build --release --target x86_64-pc-windows-gnu\ncross build --release --target x86_64-apple-darwin\n```\n\n\n#### **Consider Using Rust's MUSL for Static Linking (Linux Only)**\nIf targeting Linux systems with no shared libraries, you can build a statically linked binary using MUSL:\n```bash\nrustup target add x86_64-unknown-linux-musl\ncargo build --release --target x86_64-unknown-linux-musl\n```\nThis produces a binary that works on most Linux distributions.\n\n### Summary\n- Use **cross-compilation** to build for other platforms without a native environment.\n- Use **`cross`** for easier cross-compilation.\n- If you have access to all platforms, build natively on each.\n\n---\n\n## How To Run The Utility\n\n### Available Commands\n\n#### 1. Convert JSON to Parquet\nTo convert a JSON file to a Parquet file, use the following command:  \n```bash\n./tsdb_timon convert \u003cjson_file_path\u003e \u003cparquet_file_path\u003e\n```\n\n**Example:**  \n```bash\n./tsdb_timon convert test_input.json test_output.parquet\n```\n\n#### 2. Execute SQL Query on Parquet\nRun an SQL query against the Parquet file:  \n```bash\n./tsdb_timon query \u003cparquet_file_path\u003e \"\u003csql_query\u003e\"\n```\n\n**Example:**  \n```bash\n./tsdb_timon query test_output.parquet \"SELECT * FROM timon\"\n```\n\n### Notes:\n- The table name is always set to **`timon`**. Ensure all SQL queries reference the `timon` table explicitly.\n- Replace `\u003cjson_file_path\u003e`, `\u003cparquet_file_path\u003e`, and `\u003csql_query\u003e` with your respective input file paths and query.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongrov%2Ftimon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmongrov%2Ftimon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongrov%2Ftimon/lists"}