{"id":32271054,"url":"https://github.com/tocreator/tostore","last_synced_at":"2026-03-08T18:04:32.221Z","repository":{"id":267048224,"uuid":"900119412","full_name":"tocreator/tostore","owner":"tocreator","description":" A high-performance relational database for Dart with multi-space architecture. Features smart caching, file/local storage, SQL \u0026 key-value persistent store.","archived":false,"fork":false,"pushed_at":"2026-02-26T16:34:41.000Z","size":18768,"stargazers_count":40,"open_issues_count":8,"forks_count":3,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-02-26T22:52:42.170Z","etag":null,"topics":["cache","data","database","db","schema","sql","storage"],"latest_commit_sha":null,"homepage":"https://pub.dev/packages/tostore","language":"Dart","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tocreator.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-07T23:01:40.000Z","updated_at":"2026-02-26T16:40:42.000Z","dependencies_parsed_at":"2025-01-16T09:27:03.048Z","dependency_job_id":"1b813923-4c5e-4ca7-bfdd-5f3f2714fdc9","html_url":"https://github.com/tocreator/tostore","commit_stats":null,"previous_names":["tocreator/tostore"],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/tocreator/tostore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tocreator%2Ftostore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tocreator%2Ftostore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tocreator%2Ftostore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tocreator%2Ftostore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tocreator","download_url":"https://codeload.github.com/tocreator/tostore/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tocreator%2Ftostore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30229589,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T19:01:10.287Z","status":"ssl_error","status_checked_at":"2026-03-07T18:59:58.103Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","data","database","db","schema","sql","storage"],"created_at":"2025-10-22T22:45:27.635Z","updated_at":"2026-03-08T18:04:32.214Z","avatar_url":"https://github.com/tocreator.png","language":"Dart","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"doc/resource/logo-tostore.svg\" width=\"400\" alt=\"Tostore\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"doc/resource/divider.svg\" width=\"600\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pub.dev/packages/tostore\"\u003e\u003cimg src=\"https://img.shields.io/pub/v/tostore.svg\" alt=\"pub package\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pub.dev/packages/tostore/score\"\u003e\u003cimg src=\"https://img.shields.io/pub/points/tostore.svg\" alt=\"Pub Points\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pub.dev/packages/tostore/likes\"\u003e\u003cimg src=\"https://img.shields.io/pub/likes/tostore.svg\" alt=\"Pub Likes\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pub.dev/packages/tostore\"\u003e\u003cimg src=\"https://img.shields.io/pub/dm/tostore.svg\" alt=\"Monthly Downloads\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://opensource.org/licenses/Apache-2.0\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-Apache_2.0-blue.svg\" alt=\"License\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pub.dev/packages/tostore\"\u003e\u003cimg src=\"https://img.shields.io/badge/Platform-Multi--Platform-02569B?logo=dart\" alt=\"Platform\"\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Architecture-Neural--Distributed-orange\" alt=\"Architecture\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  English | \n  \u003ca href=\"doc/translations/README.zh-CN.md\"\u003e简体中文\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.ja.md\"\u003e日本語\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.ko.md\"\u003e한국어\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.es.md\"\u003eEspañol\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.pt-BR.md\"\u003ePortuguês (Brasil)\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.ru.md\"\u003eРусский\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.de.md\"\u003eDeutsch\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.fr.md\"\u003eFrançais\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.it.md\"\u003eItaliano\u003c/a\u003e | \n  \u003ca href=\"doc/translations/README.tr.md\"\u003eTürkçe\u003c/a\u003e\n\u003c/p\u003e\n\n\n\n\n## Why Choose Tostore?\n\nTostore is the only high-performance storage engine for distributed vector databases in the Dart/Flutter ecosystem. Utilizing a neural-network-like architecture, it features intelligent interconnectivity and collaboration between nodes, supporting infinite horizontal scaling. It builds a flexible data topology network and provides precise schema change identification, encryption protection, and multi-space data isolation. Tostore fully leverages multi-core CPUs for extreme parallel processing and naturally supports cross-platform collaboration from mobile edge devices to the cloud. With various distributed primary key algorithms, it provides a powerful data foundation for scenarios such as immersive AR/VR fusion, multi-modal interaction, spatial computing, generative AI, and semantic vector space modeling.\n\nAs generative AI and spatial computing shift the center of gravity toward the edge, terminal devices are evolving from mere content displays into cores for local generation, environmental perception, and real-time decision-making. Traditional single-file embedded databases are limited by their architectural design, often struggling to support the immediate response requirements of intelligent applications when facing high-concurrency writes, massive vector retrieval, and cloud-edge collaborative generation. Tostore is born for edge devices, empowering them with distributed storage capabilities sufficient to support complex local AI generation and large-scale data flow, truly achieving deep collaboration between the cloud and the edge.\n\n**Power failure and crash resistant**: Even in the event of an unexpected power outage or application crash, data can be automatically recovered, achieving true zero loss. When a data operation responds, the data has already been safely saved, eliminating the risk of data loss.\n\n**Breaking performance limits**: Performance tests show that even with 100 million+ records, a typical smartphone can maintain constant retrieval performance regardless of data scale, delivering an experience far exceeding that of traditional databases.\n\n\n\n\n...... From fingertips to cloud applications, Tostore helps you release data computing power and empower the future ......\n\n\n\n\n## Key Features\n\n- 🌐 **Seamless All-Platform Support**\n  - Run the same code across all platforms, from mobile apps to cloud servers.\n  - Intelligently adapt to different platform storage backends (IndexedDB, File System, etc.).\n  - Unified API interface for worry-free cross-platform data synchronization.\n  - Seamless data flow from edge devices to cloud servers.\n  - Local vector computation on edge devices, reducing network latency and cloud dependency.\n\n- 🧠 **Neural-Network-Like Distributed Architecture**\n  - Neural-network-like interconnected node topology for efficient data flow organization.\n  - High-performance data partitioning mechanism for true distributed processing.\n  - Intelligent dynamic workload balancing to maximize resource utilization.\n  - Infinite horizontal scaling of nodes to easily build complex data networks.\n\n- ⚡ **Ultimate Parallel Processing**\n  - True parallel read/write using Isolates, running at full speed on multi-core CPUs.\n  - Intelligent resource scheduling automatically balances load to maximize multi-core performance.\n  - Collaborative multi-node computing network doubles task processing efficiency.\n  - Resource-aware scheduling framework automatically optimizes execution plans to avoid resource contention.\n  - Streaming query interface handles massive datasets with ease.\n\n- 🔑 **Diverse Distributed Primary Key Algorithms**\n  - Sequential Increment Algorithm - Freely adjust random step sizes to hide business scale.\n  - Timestamp-Based Algorithm - The best choice for high-concurrency scenarios.\n  - Date-Prefix Algorithm - Perfect support for time-range data display.\n  - Short Code Algorithm - Generates short, human-readable unique identifiers.\n\n- 🔄 **Intelligent Schema Migration \u0026 Data Integrity**\n  - Precisely identifies renamed table fields with zero data loss.\n  - Millisecond-level automatic detection of schema changes and data migration.\n  - Zero-downtime upgrades, seamless to the business.\n  - Safe migration strategies for complex structure changes.\n  - Automatic foreign key constraint validation with cascading support ensures referential integrity.\n\n- 🛡️ **Enterprise-Grade Security \u0026 Durability**\n  - Dual Protection Mechanism: Real-time logging of data changes ensures nothing is ever lost.\n  - Automatic Crash Recovery: Automatically resumes unfinished operations after power failure or crash.\n  - Data Consistency Guarantee: Operations either succeed entirely or roll back completely.\n  - Atomic Computational Updates: Expression system supports complex calculations, executed atomically to avoid concurrency conflicts.\n  - Instant Safe Flushing: Data is safely saved when the operation succeeds, no waiting required.\n  - High-strength ChaCha20Poly1305 encryption protects sensitive data.\n  - End-to-end encryption for security throughout storage and transmission.\n\n- 🚀 **Intelligent Caching \u0026 Retrieval Performance**\n  - Multi-level intelligent caching mechanism for blazing-fast data retrieval.\n  - Caching strategies deeply integrated with the storage engine.\n  - Adaptive scaling maintains stable performance as data scale grows.\n  - Real-time data change notifications with automatic query result updates.\n\n- 🔄 **Intelligent Data Workflow**\n  - Multi-space architecture provides data isolation along with global sharing.\n  - Intelligent workload distribution across computing nodes.\n  - Provides a solid foundation for large-scale data training and analysis.\n\n\n## Installation\n\n\u003e [!IMPORTANT]\n\u003e **Upgrading from v2.x?** Please read the [v3.x Upgrade Guide](doc/UPGRADE_GUIDE_v3.md) for critical migration steps and breaking changes.\n\nAdd `tostore` as a dependency in your `pubspec.yaml`:\n\n```yaml\ndependencies:\n  tostore: any # Please use the latest version\n```\n\n## Quick Start\n\n\u003e [!IMPORTANT]\n\u003e **Defining table schema is the first step**: Before performing CRUD operations, you must define the table schema. The specific definition method depends on your scenario:\n\u003e - **Mobile/Desktop**: Recommended [Static Definition](#integration-for-frequent-startup-scenarios).\n\u003e - **Server-side**: Recommended [Dynamic Creation](#server-side-integration).\n\n```dart\n// 1. Initialize the database\nfinal db = await ToStore.open();\n\n// 2. Insert data\nawait db.insert('users', {\n  'username': 'John',\n  'email': 'john@example.com',\n  'age': 25,\n});\n\n// 3. Chained queries ([query operators](#query-operators) support =, !=, \u003e, \u003c, LIKE, IN, etc.)\nfinal users = await db.query('users')\n    .where('age', '\u003e', 20)\n    .where('username', 'like', '%John%')\n    .orderByDesc('age')\n    .limit(20);\n\n// 4. Update and Delete\nawait db.update('users', {'age': 26}).where('username', '=', 'John');\nawait db.delete('users').where('username', '=', 'John');\n\n// 5. Real-time Listening (UI updates automatically when data changes)\ndb.query('users').where('age', '\u003e', 18).watch().listen((users) {\n  print('Matching users updated: $users');\n});\n```\n\n### Key-Value Storage (KV)\nSuitable for scenarios that do not require defining structured tables. It's simple, practical, and includes a built-in high-performance KV store for configuration, status, and other scattered data. Data in different Spaces is naturally isolated but can be set for global sharing.\n\n```dart\n// 1. Set key-value pairs (Supports String, int, bool, double, Map, List, etc.)\nawait db.setValue('theme', 'dark');\nawait db.setValue('login_attempts', 3);\n\n// 2. Get data\nfinal theme = await db.getValue('theme'); // 'dark'\n\n// 3. Remove data\nawait db.removeValue('theme');\n\n// 4. Global key-value (Shared across Spaces)\n// Default KV data is space-specific. Use isGlobal: true for global sharing.\nawait db.setValue('app_version', '1.0.0', isGlobal: true);\nfinal version = await db.getValue('app_version', isGlobal: true);\n```\n\n\n\n## Integration for Frequent Startup Scenarios\n\n📱 **Example**: [mobile_quickstart.dart](example/lib/mobile_quickstart.dart)\n\n```dart\n// Schema definition suitable for frequent startup scenarios like mobile and desktop apps.\n// Precisely identifies schema changes and auto-migrates data with zero code maintenance.\nfinal db = await ToStore.open(\n  schemas: [\n    const TableSchema(\n            name: 'global_settings',\n            isGlobal: true,  // Global table accessible to all spaces\n            fields: [],\n    ),\n    const TableSchema(\n      name: 'users', // Table name\n      tableId: \"users\",  // Unique identifier for 100% rename detection\n      primaryKeyConfig: PrimaryKeyConfig(\n        name: 'id',       // Primary key name\n      ),\n      fields: [        // Field definitions (excluding primary key)\n        FieldSchema(\n          name: 'username', \n          type: DataType.text, \n          nullable: false, \n          unique: true, // Automatically creates a unique index\n          fieldId: 'username',\n        ),\n        FieldSchema(\n          name: 'email', \n          type: DataType.text, \n          nullable: false, \n          unique: true // Automatically creates a unique index\n        ),\n        FieldSchema(\n          name: 'last_login', \n          type: DataType.datetime,\n          createIndex: true // Automatically creates an index\n        ),\n      ],\n      // Composite index example\n      indexes: [\n        IndexSchema(fields: ['username', 'last_login']),\n      ],\n    ),\n    // Foreign key constraint example\n    TableSchema(\n      name: 'posts',\n      primaryKeyConfig: const PrimaryKeyConfig(name: 'id'),\n      fields: [\n        const FieldSchema(name: 'title', type: DataType.text, nullable: false),\n        const FieldSchema(name: 'user_id', type: DataType.integer, nullable: false),\n        const FieldSchema(name: 'content', type: DataType.text),\n      ],\n      foreignKeys: [\n        ForeignKeySchema(\n          name: 'fk_posts_user',\n          fields: ['user_id'],              // Current table fields\n          referencedTable: 'users',         // Referenced table\n          referencedFields: ['id'],         // Referenced fields\n          onDelete: ForeignKeyCascadeAction.cascade,  // Cascade delete\n          onUpdate: ForeignKeyCascadeAction.cascade,  // Cascade update\n        ),\n      ],\n    ),\n  ],\n);\n\n// Multi-space architecture - perfect isolation of different users' data\nawait db.switchSpace(spaceName: 'user_123');\n```\n\n### Keeping login state and logout (active space)\n\nMulti-space works well for **per-user data**: use one space per user and switch when the user logs in. To keep the current user across app restarts and to support logout, use **active space** and **close** options.\n\n- **Keeping login state**: When the user switches to their space, save it as the active space so the next launch opens in that space by default (no need to \"open default then switch\").\n- **Logout**: When the user logs out, close the database with `keepActiveSpace: false` so the next launch does not auto-open in the previous user's space.\n\n```dart\n\n// After login: switch to this user's space and remember it for next launch (keep login state)\nawait db.switchSpace(spaceName: 'user_${userId}', keepActive: true);\n\n// Optional: open strictly in default (e.g. login screen only) — do not use stored active space\n// final db = await ToStore.open(..., applyActiveSpaceOnDefault: false);\n\n// On logout: close and clear active space so next launch uses default space (logout)\nawait db.close(keepActiveSpace: false);\n```\n\n\n## Server-Side Integration\n\n🖥️ **Example**: [server_quickstart.dart](example/lib/server_quickstart.dart)\n\n```dart\nfinal db = await ToStore.open();\n\n// Bulk schema creation at runtime - suitable for continuous running scenarios\nawait db.createTables([\n  // 3D Spatial Feature Vector storage\n  const TableSchema(\n    name: 'spatial_embeddings',\n    primaryKeyConfig: PrimaryKeyConfig(\n      name: 'id',\n      type: PrimaryKeyType.timestampBased,   // Timestamp PK for high-concurrency writes\n    ),\n    fields: [\n      FieldSchema(\n        name: 'video_name',\n        type: DataType.text,\n        nullable: false,\n      ),\n      FieldSchema(\n        name: 'spatial_features',\n        type: DataType.vector,                // Vector storage type\n        vectorConfig: VectorFieldConfig(\n          dimensions: 1024,                   // High-dimensional vector\n          precision: VectorPrecision.float32, \n        ),\n      ),\n    ],\n    indexes: [\n      IndexSchema(\n        fields: ['video_name'],\n        unique: true,\n      ),\n      IndexSchema(\n        type: IndexType.vector,              // Vector index\n        fields: ['spatial_features'],\n        vectorConfig: VectorIndexConfig(\n          indexType: VectorIndexType.ngh,    // NGH algorithm for efficient ANN\n          distanceMetric: VectorDistanceMetric.cosine,\n          parameters: {\n            'M': 16,\n            'efConstruction': 200,\n          },\n        ),\n      ),\n    ],\n  ),\n  // Other tables...\n]);\n\n// Online Schema Updates - Seamless to the business\nfinal taskId = await db.updateSchema('users')\n  .renameTable('users_new')                // Rename table\n  .modifyField(\n    'username',\n    minLength: 5,\n    maxLength: 20,\n    unique: true\n  )                                        // Modify field attributes\n  .renameField('old_name', 'new_name')     // Rename field\n  .removeField('deprecated_field')         // Remove field\n  .addField('created_at', type: DataType.datetime)  // Add field\n  .removeIndex(fields: ['age'])            // Remove index\n  .setPrimaryKeyConfig(                    // Change PK config\n    const PrimaryKeyConfig(type: PrimaryKeyType.shortCode)\n  );\n    \n// Monitor migration progress\nfinal status = await db.queryMigrationTaskStatus(taskId);\nprint('Migration Progress: ${status?.progressPercentage}%');\n\n\n// Manual Query Cache Management (Server-side)\n// For queries on primary keys or indexed fields (Equality, IN queries), \n// performance is already extreme and manual cache management is usually unnecessary.\n\n// Manually cache a query result for 5 minutes.\nfinal activeUsers = await db.query('users')\n    .where('is_active', '=', true)\n    .useQueryCache(const Duration(minutes: 5));\n\n// Invalidate specific cache when data changes to ensure consistency.\nawait db.query('users')\n    .where('is_active', '=', true)\n    .clearQueryCache();\n\n// Explicitly disable cache for queries requiring real-time data.\nfinal freshUserData = await db.query('users')\n    .where('is_active', '=', true)\n    .noQueryCache();\n```\n\n\n\n## Advanced Usage\n\nTostore provides a rich set of advanced features for complex business requirements:\n\n### Table-level TTL (automatic time-based expiration)\n\nFor logs, events, and other time-series style data that should expire automatically, you can define table-level TTL via `ttlConfig`. Expired data is cleaned up in the background in small batches, without requiring manual iteration in your business code:\n\n```dart\nconst TableSchema(\n  name: 'event_logs',\n  fields: [\n    FieldSchema(\n      name: 'created_at',\n      type: DataType.datetime,\n      nullable: false,\n      createIndex: true,\n      defaultValueType: DefaultValueType.currentTimestamp,\n    ),\n  ],\n  ttlConfig: TableTtlConfig(\n    ttlMs: 7 * 24 * 60 * 60 * 1000, // keep 7 days\n    // When sourceField is omitted, the engine uses the write time\n    // and manages the required index automatically.\n    // Optional: when you provide a custom sourceField, it must:\n    // 1) have type DataType.datetime\n    // 2) be non-nullable (nullable: false)\n    // 3) use DefaultValueType.currentTimestamp as defaultValueType\n    // sourceField: 'created_at',\n  ),\n);\n```\n\n### Nested Queries \u0026 Custom Filtering\nSupports infinite nesting of conditions and flexible custom functions.\n\n```dart\n// Condition nesting: (type = 'app' OR (id \u003e= 123 OR fans \u003e= 200))\nfinal idCondition = QueryCondition().where('id', '\u003e=', 123).or().where('fans', '\u003e=', 200);\n\nfinal result = await db.query('users')\n    .condition(\n        QueryCondition().whereEqual('type', 'app').or().condition(idCondition)\n    )\n    .limit(20);\n\n// Custom condition function\nfinal customResult = await db.query('users')\n    .whereCustom((record) =\u003e record['tags']?.contains('recommend') ?? false)\n    .limit(20);\n```\n\n### Intelligent Upsert\nUpdate if a row exists (by primary key or unique key in data), otherwise insert. No where clause; conflict target is derived from data.\n\n```dart\n// By primary key\nfinal result = await db.upsert('users', {\n  'id': 1,\n  'username': 'john',\n  'email': 'john@example.com',\n});\n\n// By unique key (record must contain all fields of one unique index + required fields)\nawait db.upsert('users', {\n  'username': 'john',\n  'email': 'john@example.com',\n  'age': 26,\n});\n\n// Batch upsert\nfinal batchResult = await db.batchUpsert('users', [\n  {'username': 'a', 'email': 'a@example.com'},\n  {'username': 'b', 'email': 'b@example.com'},\n], allowPartialErrors: true);\n```\n\n\n### Joins \u0026 Field Selection\n```dart\nfinal orders = await db.query('orders')\n    .select(['orders.id', 'users.name as user_name'])\n    .join('users', 'orders.user_id', '=', 'users.id')\n    .where('orders.amount', '\u003e', 1000)\n    .limit(20);\n```\n\n### Streaming \u0026 Aggregation\n```dart\n// Count records\nfinal count = await db.query('users').count();\n\n// Check if a table has been defined in the current database (space-agnostic).\n// Note: this does NOT indicate whether the table has data.\nfinal usersTableDefined = await db.tableExists('users');\n\n// Efficient existence check based on conditions (no full record loading).\nfinal emailExists = await db.query('users')\n    .where('email', '=', 'test@example.com')\n    .exists();\n\n// Aggregate functions\nfinal totalAge = await db.query('users').where('age', '\u003e', 18).sum('age');\nfinal avgAge = await db.query('users').avg('age');\nfinal maxAge = await db.query('users').max('age');\nfinal minAge = await db.query('users').min('age');\n\n// Group and filter\nfinal result = await db.query('orders')\n    .select(['status', Agg.sum('amount', alias: 'total')])\n    .groupBy(['status'])\n    .having(Agg.sum('amount'), '\u003e', 1000)\n    .limit(20);\n\n// Distinct query\nfinal uniqueCities = await db.query('users').distinct(['city']);\n\n// Streaming query (suitable for massive data)\ndb.streamQuery('users').listen((data) =\u003e print(data));\n```\n\n\n\n### Querying \u0026 Efficient Pagination\n\n\u003e [!TIP]\n\u003e **Use `limit` for better performance**: It is highly recommended to always specify a `limit`. If omitted, the engine defaults to a limit of 1000 records. While the core engine is fast, fetching and serializing 1000 records can cause unnecessary latency in UI-sensitive applications.\n\nTostore offers dual-mode pagination support to fit different data scales and performance needs:\n\n#### 1. Offset Mode\nSuitable for small datasets (e.g., under 10k records) or when specific page jumping is required.\n\n```dart\nfinal result = await db.query('users')\n    .orderByDesc('created_at')\n    .offset(40) // Skip first 40\n    .limit(20); // Take 20\n```\n\u003e [!TIP]\n\u003e When `offset` is very large, the database must scan and discard many records, leading to linear performance degradation. Use **Cursor Mode** for deep paging.\n\n#### 2. High-Performance Cursor Mode\n**Recommended for massive data and infinite scroll scenarios**. Utilizes `nextCursor` for O(1) performance, ensuring constant query speed regardless of page depth.\n\n\u003e [!IMPORTANT]\n\u003e If sorting by an unindexed field or for certain complex queries, the engine may fall back to a full table scan and return a `null` cursor (meaning pagination for that specific query is not yet supported).\n\n```dart\n// Page 1\nfinal page1 = await db.query('users')\n    .orderByDesc('id')\n    .limit(20);\n\n// Fetch next page using the cursor\nif (page1.nextCursor != null) {\n  final page2 = await db.query('users')\n      .orderByDesc('id')\n      .limit(20)\n      .cursor(page1.nextCursor); // Seek directly to the position\n}\n\n// Efficiently move backwards with prevCursor\nfinal prevPage = await db.query('users')\n    .limit(20)\n    .cursor(page2.prevCursor);\n```\n\n| Feature | Offset Mode | Cursor Mode |\n| :--- | :--- | :--- |\n| **Query Performance** | Decreases as page increases | **Constant (O(1))** |\n| **Complexity** | Small data, page jumping | **Massive data, infinite scroll** |\n| **Consistency** | Data changes can cause skips | **Avoids duplicates/omissions from data changes** |\n\n\n\n### Query Operators\n\nAll `where(field, operator, value)` conditions use the following operators (case-insensitive):\n\n| Operator | Description | Example / Value type |\n| :--- | :--- | :--- |\n| `=` | Equal | `where('status', '=', 'active')` |\n| `!=`, `\u003c\u003e` | Not equal | `where('role', '!=', 'guest')` |\n| `\u003e` | Greater than | `where('age', '\u003e', 18)` |\n| `\u003e=` | Greater than or equal | `where('score', '\u003e=', 60)` |\n| `\u003c` | Less than | `where('price', '\u003c', 100)` |\n| `\u003c=` | Less than or equal | `where('quantity', '\u003c=', 10)` |\n| `IN` | Value in list | `where('id', 'IN', ['a','b','c'])` — value: `List` |\n| `NOT IN` | Value not in list | `where('status', 'NOT IN', ['banned'])` — value: `List` |\n| `BETWEEN` | Between start and end (inclusive) | `where('age', 'BETWEEN', [18, 65])` — value: `[start, end]` |\n| `LIKE` | Pattern match (`%` any, `_` single char) | `where('name', 'LIKE', '%John%')` — value: `String` |\n| `NOT LIKE` | Pattern not match | `where('email', 'NOT LIKE', '%@test.com')` — value: `String` |\n| `IS` | Is null | `where('deleted_at', 'IS', null)` — value: `null` |\n| `IS NOT` | Is not null | `where('email', 'IS NOT', null)` — value: `null` |\n\n### Semantic query methods (recommended)\n\nPrefer semantic methods to avoid typing operator strings and get better IDE support:\n\n```dart\n// Comparison\ndb.query('users').whereEqual('username', 'John');\ndb.query('users').whereNotEqual('role', 'guest');\ndb.query('users').whereGreaterThan('age', 18);\ndb.query('users').whereGreaterThanOrEqualTo('score', 60);\ndb.query('users').whereLessThan('price', 100);\ndb.query('users').whereLessThanOrEqualTo('quantity', 10);\n\n// Membership \u0026 range\ndb.query('users').whereIn('id', ['id1', 'id2']);\ndb.query('users').whereNotIn('status', ['banned', 'pending']);\ndb.query('users').whereBetween('age', 18, 65);\n\n// Null checks\ndb.query('users').whereNull('deleted_at');\ndb.query('users').whereNotNull('email');\n\n// Pattern match\ndb.query('users').whereLike('name', '%John%');\ndb.query('users').whereNotLike('email', '%@temp.');\ndb.query('users').whereContains('bio', 'flutter');   // LIKE '%flutter%'\ndb.query('users').whereNotContains('title', 'draft');\n\n// Equivalent to: .where('age', '\u003e', 18).where('name', 'like', '%John%')\nfinal users = await db.query('users')\n    .whereGreaterThan('age', 18)\n    .whereLike('username', '%John%')\n    .orderByDesc('age')\n    .limit(20);\n```\n\n## Distributed Architecture\n\n```dart\n// Configure Distributed Nodes\nfinal db = await ToStore.open(\n  config: DataStoreConfig(\n    distributedNodeConfig: const DistributedNodeConfig(\n      enableDistributed: true,\n      clusterId: 1,\n      centralServerUrl: 'http://127.0.0.1:8080',\n      accessToken: 'b7628a4f9b4d269b98649129'\n    )\n  )\n);\n\n// High-Performance Batch Insertion\nawait db.batchInsert('vector_data', [\n  {'vector_name': 'face_2365', 'timestamp': DateTime.now()},\n  {'vector_name': 'face_2366', 'timestamp': DateTime.now()},\n  // ... Records inserted efficiently in bulk\n]);\n\n// Stream large datasets - Constant memory usage\nawait for (final record in db.streamQuery('vector_data')\n  .where('vector_name', '=', 'face_2366')\n  .where('timestamp', '\u003e=', DateTime.now().subtract(Duration(days: 30)))\n  .stream) {\n  // Efficiently process even TB-scale data without high memory usage\n  print(record);\n}\n```\n\n## Primary Key Examples\n\nTostore provides various distributed primary key algorithms for different scenarios:\n\n- **Sequential** (PrimaryKeyType.sequential): 238978991\n- **Timestamp-Based** (PrimaryKeyType.timestampBased): 1306866018836946\n- **Date-Prefix** (PrimaryKeyType.datePrefixed): 20250530182215887631\n- **Short Code** (PrimaryKeyType.shortCode): 9eXrF0qeXZ\n\n```dart\n// Sequential Primary Key configuration example\nawait db.createTables([\n  const TableSchema(\n    name: 'users',\n    primaryKeyConfig: PrimaryKeyConfig(\n      type: PrimaryKeyType.sequential,\n      sequentialConfig: SequentialIdConfig(\n        initialValue: 10000,\n        increment: 50,\n        useRandomIncrement: true, // Hide business volume\n      ),\n    ),\n    fields: [/* Field definitions */]\n  ),\n]);\n```\n\n\n## Expression Atomic Operations\n\nThe expression system provides type-safe atomic field updates. All calculations are executed atomically at the database level to avoid concurrency conflicts:\n\n```dart\n// Simple Increment: balance = balance + 100\nawait db.update('accounts', {\n  'balance': Expr.field('balance') + Expr.value(100),\n}).where('id', '=', accountId);\n\n// Complex Calculation: total = price * quantity + tax\nawait db.update('orders', {\n  'total': Expr.field('price') * Expr.field('quantity') + Expr.field('tax'),\n}).where('id', '=', orderId);\n\n// Nested Parentheses: finalPrice = ((price * quantity) + tax) * (1 - discount)\nawait db.update('orders', {\n  'finalPrice': ((Expr.field('price') * Expr.field('quantity')) + Expr.field('tax')) * \n                 (Expr.value(1) - Expr.field('discount')),\n}).where('id', '=', orderId);\n\n// Using Functions: price = min(price, maxPrice)\nawait db.update('products', {\n  'price': Expr.min(Expr.field('price'), Expr.field('maxPrice')),\n}).where('id', '=', productId);\n\n// Timestamp: updatedAt = now()\nawait db.update('users', {\n  'updatedAt': Expr.now(),\n}).where('id', '=', userId);\n```\n\n**Conditional expressions (e.g. for upsert)**: Use `Expr.isUpdate()` / `Expr.isInsert()` with `Expr.ifElse` or `Expr.when` so that expressions run only on update or only on insert:\n\n```dart\n// Upsert: increment on update, set to 1 on insert (insert branch uses plain value; only update evaluates expression)\nawait db.upsert('counters', {\n  'key': 'visits',\n  'count': Expr.ifElse(\n    Expr.isUpdate(),\n    Expr.field('count') + Expr.value(1),\n    1,  // insert branch: plain value, not evaluated by insert\n  ),\n});\n\n// Same with Expr.when (single-branch, otherwise defaults to null)\nawait db.upsert('orders', {\n  'id': orderId,\n  'updatedAt': Expr.when(Expr.isUpdate(), Expr.now(), otherwise: Expr.now()),\n});\n```\n\n## Transactions\n\nTransactions ensure the atomicity of multiple operations—either all succeed or all roll back, guaranteeing data consistency.\n\n**Transaction Features**:\n- Atomic execution of multiple operations.\n- Automatic recovery of unfinished operations after a crash.\n- Data is safely saved upon successful commit.\n\n```dart\n// Basic Transaction - Atomic commit of multiple operations\nfinal txResult = await db.transaction(() async {\n  // Insert User\n  await db.insert('users', {\n    'username': 'john',\n    'email': 'john@example.com',\n    'fans': 100,\n  });\n  \n  // Atomic update using expressions\n  await db.update('users', {\n    'fans': Expr.field('fans') + Expr.value(50),\n  }).where('username', '=', 'john');\n  \n  // If any operation fails, all changes roll back automatically.\n});\n\nif (txResult.isSuccess) {\n  print('Transaction committed successfully');\n} else {\n  print('Transaction rolled back: ${txResult.error?.message}');\n}\n\n// Automatic rollback on error\nfinal txResult2 = await db.transaction(() async {\n  await db.insert('users', {\n    'username': 'jane',\n    'email': 'jane@example.com',\n  });\n  throw Exception('Business logic error'); // Triggers rollback\n}, rollbackOnError: true);\n```\n\n### Pure Memory Mode\n\nFor scenarios like data caching, temporary computation, or diskless environments without persisting to disk, you can initialize a pure in-memory database using ToStore.memory(). In this mode, all data (including schemas, indexes, and key-value pairs) is kept strictly in memory.\n\n```dart\n// Initialize database in pure memory mode\nfinal db = await ToStore.memory(\n  schemas: [],\n);\n\n// All operations (CRUD and search) are instantly executed in memory\nawait db.insert('temp_cache', {'key': 'session_1', 'value': {'user': 'admin'}});\n\n// Note: Data created in memory mode is completely lost upon application closure or restart.\n```\n\n## Security Configuration\n\n**Data Security Mechanisms**:\n- Dual protection mechanisms ensure data is never lost.\n- Automatic crash recovery of unfinished operations.\n- Instant safe persistence upon operation success.\n- High-strength encryption protects sensitive data.\n\n\u003e [!WARNING]\n\u003e **Key Management**: **`encodingKey`** can be changed freely; the engine will automatically migrate data when it changes, so you need not worry about data loss. **`encryptionKey`** must not be changed arbitrarily—changing it will make old data unreadable unless a migration is performed. Do not hardcode sensitive keys; fetch them from a secure server.\n\n```dart\nfinal db = await ToStore.open(\n  config: DataStoreConfig(\n    encryptionConfig: EncryptionConfig(\n      // Algorithms supported: none, xorObfuscation, chacha20Poly1305, aes256Gcm\n      encryptionType: EncryptionType.chacha20Poly1305, \n      \n      // Encoding Key (can be changed freely; data will be auto-migrated)\n      encodingKey: 'Your-32-Byte-Long-Encoding-Key...', \n      \n      // Encryption Key for critical data (do not change arbitrarily; old data becomes unreadable unless migrated)\n      encryptionKey: 'Your-Secure-Encryption-Key...',\n      \n      // Device Binding (Path-based)\n      // When enabled, keys are bound to path and device characteristics.\n      // Boosts security against database file copying but data retrieval \n      // depends on app installation path and device persistence.\n      deviceBinding: false, \n    ),\n    // Write-Ahead Logging (WAL) enabled by default\n    enableJournal: true, \n    // Force disk flush on commit for maximum durability (set to false for performance)\n    persistRecoveryOnCommit: true,\n  ),\n);\n```\n\n### Value-level encryption (ToCrypto)\n\nFull-database encryption above encrypts all table and index data and can affect overall performance. To encrypt only sensitive fields, use **ToCrypto**: it is independent of the database (no db instance required). You encode or decode values yourself before writing or after reading; the key is managed entirely by your app. Output is Base64, suitable for JSON or TEXT columns.\n\n- **key** (required): `String` or `Uint8List`. If not 32 bytes, a 32-byte key is derived via SHA-256.\n- **type** (optional): Encryption type, [ToCryptoType]: [ToCryptoType.chacha20Poly1305] or [ToCryptoType.aes256Gcm]. Default [ToCryptoType.chacha20Poly1305]. Omit for default.\n- **aad** (optional): Additional Authenticated Data — `Uint8List`. If you pass it at encode, you must pass the same bytes at decode (e.g. table name + field name for context binding). Omit for simple use.\n\n```dart\n\nconst key = 'my-secret-key';\n// Encode: plain → Base64 cipher (store in DB or JSON)\nfinal cipher = ToCrypto.encode('sensitive data', key: key);\n// Decode when reading\nfinal plain = ToCrypto.decode(cipher, key: key);\n\n// Optional: bind cipher to context with aad (same aad at encode and decode)\nfinal aad = Uint8List.fromList(utf8.encode('users:id_number'));\nfinal cipher2 = ToCrypto.encode('secret', key: key, aad: aad);\nfinal plain2 = ToCrypto.decode(cipher2, key: key, aad: aad);\n```\n\n## Performance \u0026 Experience\n\n### Performance Specs\n\n- **Startup Speed**: Instant startup and data display even with 100M+ records on average smartphones.\n- **Query Performance**: Scale-independent, consistently blazing-fast retrieval at any data volume.\n- **Data Safety**: ACID transaction guarantees + crash recovery for zero data loss.\n\n### Recommendations\n\n- 📱 **Example Project**: A complete Flutter app example is provided in the `example` directory.\n- 🚀 **Production**: Use Release mode for performance far exceeding Debug mode.\n- ✅ **Standard Tests**: All core functionalities have passed standard integration tests.\n\n### Demo Videos\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"doc/media/basic-demo.gif\" alt=\"Tostore basic performance demo\" width=\"320\" /\u003e\n  \u003c/p\u003e\n\n- **Basic performance demo** (\u003ca href=\"doc/media/basic-demo.mp4?raw=1\" target=\"_blank\" rel=\"noopener\"\u003ebasic-demo.mp4\u003c/a\u003e): GIF preview may be cropped; click the video to view the full demo. Shows that even on an ordinary mobile device with 100M+ records, startup, paging, and search performance remain stable and smooth. As long as storage is sufficient, edge devices can sustain TB/PB-scale datasets while keeping interactive latency consistently low.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"doc/media/disaster-recovery.gif\" alt=\"Tostore disaster recovery stress test\" width=\"320\" /\u003e\n  \u003c/p\u003e\n\n- **Disaster recovery stress test** (\u003ca href=\"doc/media/disaster-recovery.mp4?raw=1\" target=\"_blank\" rel=\"noopener\"\u003edisaster-recovery.mp4\u003c/a\u003e): Intentionally kills the process during intensive write workloads to simulate crashes and power failures. Even when tens of thousands of operations are interrupted, Tostore recovers extremely fast on a typical phone and does not impact startup or data availability.\n\n\n\n\nIf Tostore helps you, please give us a ⭐️\n\n\n\n\n## Roadmap\n\nTostore is actively developing features to further enhance AI-era data infrastructure:\n\n- **High-Dimensional Vectors**: Adding vector retrieval and semantic search algorithms.\n- **Multi-modal Data**: Providing end-to-end processing from raw data to feature vectors.\n- **Graph Data Structures**: Supporting efficient storage and querying of knowledge graphs and complex relational networks.\n\n\n\n\n\n\u003e **Recommendation**: Mobile developers might also consider the [Toway Framework](https://github.com/tocreator/toway), a full-stack solution that automates data requests, loading, storage, caching, and display.\n\n\n\n\n## More Resources\n\n- 📖 **Documentation**: [Wiki](https://github.com/tocreator/tostore)\n- 📢 **Feedback**: [GitHub Issues](https://github.com/tocreator/tostore/issues)\n- 💬 **Discussion**: [GitHub Discussions](https://github.com/tocreator/tostore/discussions)\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftocreator%2Ftostore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftocreator%2Ftostore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftocreator%2Ftostore/lists"}