Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/metarhia/globalstorage
Distributed Data Warehouse 🌍
https://github.com/metarhia/globalstorage
big-data cloud cluster database db dbms globalstorage highload in-memory js jstp memory metadata metarhia node nodejs realtime schema storage
Last synced: about 1 month ago
JSON representation
Distributed Data Warehouse 🌍
- Host: GitHub
- URL: https://github.com/metarhia/globalstorage
- Owner: metarhia
- License: mit
- Created: 2016-03-27T23:55:43.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2023-01-03T15:17:10.000Z (over 1 year ago)
- Last Synced: 2024-06-03T02:36:05.361Z (3 months ago)
- Topics: big-data, cloud, cluster, database, db, dbms, globalstorage, highload, in-memory, js, jstp, memory, metadata, metarhia, node, nodejs, realtime, schema, storage
- Language: JavaScript
- Homepage: https://metarhia.com
- Size: 2.32 MB
- Stars: 60
- Watchers: 20
- Forks: 6
- Open Issues: 84
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# GlobalStorage
[](https://travis-ci.org/metarhia/globalstorage)
[](https://badge.fury.io/js/globalstorage)
[](https://www.npmjs.com/package/globalstorage)
[](https://www.npmjs.com/package/globalstorage)
## The Concept
This is a distributed DBMS for technological stack [Metarhia](https://github.com/metarhia/Metarhia) and it is built with following assumptions:
- GS is designed to be built-in DBMS, to work inside [Impress Applications Server](https://github.com/metarhia/impress); it is needed to avoid or minimize interprocess communication to access DB;
- GS is compatible with JSTP [JavaScript Transfer Protocol](https://github.com/metarhia/jstp), all data slould be stored, stansmitted, handled and placed in RAM in the same format;
- All data structures can be reduced to array representation to redice size by removing key names, we can do that with the help of metadata schemas and we can dynamicaly build prototypes from schemas and assign them to arrays, so getters/setters will convert access by name (hash) to assess by position (array);
- Maximum memory usage, read-ahead and lazy-write, minimizing data conversion;
- Using metadata everywhere, special declarative format for subject domein representation including fields, relations, and indices so we can automatically build a storage scheme in the relational database, memory structures and structure for the database, different the GUI, API server, etc.
- The same API for client-side runtime and server-side runtime:
- server-side storage engine;
- client-side storage engine (multiple implementations for different platforms including mobile, desktop and browser);
- sharding for distributed storage of large data amounts, geo-distribution, save backup copies, access load balancing;
- allows user to exchange data in P2P mode;
- Syncronization between client and server in realtime (close to realtime) and in lazy mode; so applications can work in online and offline (with locally stored data); having bidirectional data sync and hieratchical versioning like git have;
- Global data structures unification for applications working with [Metarhia](https://github.com/metarhia/Metarhia) technological stack: [GlobalStorage](https://github.com/metarhia/globalgtorage), [Impress](https://github.com/metarhia/impress), [JSTP](https://github.com/metarhia/jstp) and [Console](https://github.com/metarhia/console) through moderated distributed metadata repositories;
- Ability to work with non-unified data structures (custom schemas), specific to certain subject domain;
- GlobalStorage provides DAC (data access layer) abstraction, it substitutes ORM but it does not necessarily make maping on the relational model (though RDBMS are also supported);
- Data structures have global distributed identification system, so data can be inserted anywhere and will not bring ID conflicts;
- Data reliability is provided by distributed storage facilities, so each data structure should have master server and multiple backup and cache servers; using those servers GlobalStorage supports addressing, versioning and branching.
## Metamodel Definition Language
Using this domain specific language we will describe subject domain in declarative format. To build GUI, API, business-loguic, data structures dynamically in runtime. For example we can build JavaScript prototype and assign it to positional array to access fields by name, so arrays will work like objects.
Example:
```js
{
code: { type: 'string', primary: true },
name: {
caption: 'City',
type: 'string',
size: 32,
nullable: false,
index: { unique: false },
master: { dataset: 'Cities', key: 'name' }
},
birth: 'Date',
city: 'string',
addresses: {
type: { array: 'Address' }
},
gender: {
type: 'char',
lookup: { dictionary: { M: 'Male', F: 'Female' } }
},
age: function() {
var difference = new Date() - this.birth;
return Math.floor(difference / 31536000000);
}
}
```
Data types:
- Built-in JavaScript types: string, number, boolean, Date, etc.
- Global Storage types: id, uid, tree, ip, etc.
- RDBMS data types: char, int, real, text, money, time, date...
- Links to other data structures in GlobalStorage
- Links to other data structures in Application
## JavaScript Query Language
JSQL is a query language for data structures manipulation. JSQL have syntax for:
filter, projection, dataset join and set operations. We have a separate
repository for examples and specification:
[metarhia/JSQL](https://github.com/metarhia/JSQL). Current Implementation can
be found in [`lib/transformations.js`](lib/transformations.js).
## Contributors
See github for full [contributors list](https://github.com/metarhia/globalstorage/graphs/contributors)
## API
### gs(provider, options)
- `provider`: [``][string] provider name
- `options`: [``][object]
- `serverSuffix`: [``][uint64] optional
- `serverBitmask`: [``][uint64] optional
- `systemSuffix`: [``][uint64] optional
- `systemBitmas`: [``][uint64] optional
Create provider
#### gs.schemaConfig
- [``][object] metaschema config
### class Cursor
#### Cursor.prototype.constructor(options)
#### Cursor.prototype.definition(schema, category)
- `schema`: [``][metaschema]
- `category`: [``][string] schema name
_Returns:_ [``][this]
Attach schema
#### Cursor.prototype.enableLogging(provider, ctx, args)
#### Cursor.prototype.copy()
_Returns:_ [``][cursor] new instance
Copy references to new dataset
#### Cursor.prototype.clone()
_Returns:_ [``][cursor] new instance
Clone all dataset objects
#### Cursor.prototype.enroll(jsql)
- `jsql`: [``][array] commands array
_Returns:_ [``][this]
Apply JSQL commands to dataset
#### Cursor.prototype.empty()
_Returns:_ [``][this]
Remove all instances from dataset
#### Cursor.prototype.from(arr)
- `arr`: [``][iterable]
_Returns:_ [``][cursor] new instance
Synchronous virtualization converts Array to Cursor
#### Cursor.prototype.map(fn)
_Returns:_ [``][this]
Lazy map
fn - [``][function], map function
#### Cursor.prototype.projection(fields)
- `fields`: [``][string]|[``][object] projection metadata
array of field names or object with structure:
`{ toKey: [ fromKey, functions... ] }`
_Returns:_ [``][this]
Declarative lazy projection
#### Cursor.prototype.filter(fn)
- `fn`: [``][function] filtering function
_Returns:_ [``][this]
Lazy functional filter
#### Cursor.prototype.select(query)
- `query`: [``][function] filtering expression
_Returns:_ [``][cursor] new instance
Declarative lazy filter
#### Cursor.prototype.distinct()
_Returns:_ [``][this]
Lazy functional distinct filter
#### Cursor.prototype.sort(fn)
- `fn`: [``][function] comparing function
_Returns:_ [``][this]
Lazy functional sort
#### Cursor.prototype.order(fields)
- `fields`: [``][string]|[``][string]
_Returns:_ [``][this]
Declarative lazy ascending sort
#### Cursor.prototype.desc(fields)
- `fields`: [``][string]|[``][string]
_Returns:_ [``][this]
Declarative lazy descending sort
#### Cursor.prototype.count(\[field\])
- `field`: [``][string] field to use for count, optional
_Returns:_ [``][this]
Calculate count
#### Cursor.prototype.sum(field)
- `field`: [``][string] field to use for sum
_Returns:_ [``][this]
Calculate sum
#### Cursor.prototype.avg(field)
- `field`: [``][string] field to use for avg
_Returns:_ [``][this]
Calculate avg
#### Cursor.prototype.max(field)
- `field`: [``][string] field to use for max
_Returns:_ [``][this]
Calculate max
#### Cursor.prototype.min(field)
- `field`: [``][string] field to use for min
_Returns:_ [``][this]
Calculate min
#### Cursor.prototype.col()
_Returns:_ [``][this]
Convert first column of dataset to Array
#### Cursor.prototype.row()
_Returns:_ [``][this]
Return first row from dataset
#### Cursor.prototype.one()
_Returns:_ [``][this]
Get single first record from dataset
#### Cursor.prototype.limit(count)
- `count`: [``][number]
_Returns:_ [``][this]
Get first n records from dataset
#### Cursor.prototype.offset(offset)
- `offset`: [``][number]
_Returns:_ [``][this]
Offset into the dataset
#### Cursor.prototype.union(cursor)
- `cursor`: [``][cursor]
_Returns:_ [``][this]
Calculate union and put results to this Cursor instance
#### Cursor.prototype.intersection(cursor)
- `cursor`: [``][cursor]
_Returns:_ [``][this]
Calculate intersection and put results to this Cursor instance
#### Cursor.prototype.difference(cursor)
- `cursor`: [``][cursor]
_Returns:_ [``][this]
Calculate difference and put results to this Cursor instance
#### Cursor.prototype.complement(cursor)
- `cursor`: [``][cursor]
_Returns:_ [``][this]
Calculate complement and put results to this Cursor instance
#### Cursor.prototype.selectToMemory(query)
#### async Cursor.prototype.continue(data)
- `data`: [``][array] rows to date
_Returns:_ [``][promise]
Continue computations via i.e. MemoryCursor or other cursor
to handle remaining operations unsupported by current cursor
#### async Cursor.prototype.fetch(\[permissionChecker\])
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Get results after applying consolidated jsql
### class StorageProvider
Abstract Storage Provider
#### StorageProvider.prototype.constructor(options)
- `options`: [``][object]
- `serverSuffix`: [``][uint64] optional
- `serverBitmask`: [``][uint64] optional
- `systemSuffix`: [``][uint64] optional
- `systemBitmas`: [``][uint64] optional
Create StorageProvider
#### async StorageProvider.prototype.open(options)
- `options`: [``][object]
- `schema`: [``][metaschema]
_Returns:_ [``][promise]
Open StorageProvider
#### async StorageProvider.prototype.close()
_Returns:_ [``][promise]
Close StorageProvider
#### async StorageProvider.prototype.setup(options)
_Returns:_ [``][promise]
Setup StorageProvider
#### StorageProvider.prototype.enableLogging(ctx)
#### StorageProvider.prototype.error(name, ...ctx)
- `name`: [``][string] error name that must be equal to one of the
values from the Action's Errors field
- `ctx`: [``][array]
Utility method to generate `` from inside the Action
#### async StorageProvider.prototype.takeId()
_Returns:_ [``][promise]
Generate globally unique id
#### async StorageProvider.prototype.get(id\[, permissionChecker\])
- `id`: [``][string] globally unique object id
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Get object from GlobalStorage
#### async StorageProvider.prototype.getDetails(category, id, fieldName\[, permissionChecker\])
- `category`: [``][string] category to get details in
- `id`: [``][string] object id
- `fieldName`: [``][string] field with the Many decorator
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Get details for many-to-many link from GlobalStorage
#### async StorageProvider.prototype.set(obj\[, permissionChecker\])
- `obj`: [``][object] to be stored
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Set object in GlobalStorage
#### async StorageProvider.prototype.create(category, obj\[, permissionChecker\])
- `category`: [``][string] category to store the object in
- `obj`: [``][object] to be stored
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Create object in GlobalStorage
#### async StorageProvider.prototype.update(category, query, patch\[, permissionChecker\])
- `category`: [``][string] category to update the records in
- `query`: [``][object] example: `{ Id }`
- `patch`: [``][object] fields to update
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Update object in GlobalStorage
#### async StorageProvider.prototype.delete(category, query\[, permissionChecker\])
- `category`: [``][string] category to delete the records from
- `query`: [``][object] example: `{ Id }`
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Delete object in GlobalStorage
#### async StorageProvider.prototype.linkDetails(category, field, fromId, toIds\[, permissionChecker\])
- `category`: [``][string] category with field having the Many decorator
- `field`: [``][string] field with the Many decorator
- `fromId`: [``][uint64] Id of the record in category specified in the
first argument
- `toIds`: [``][uint64]|[``][uint64] Id(s) of the record(s) in
category specified in the Many decorator of the specified field
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Link records with Many relation between them
#### async StorageProvider.prototype.unlinkDetails(category, field, fromId, toIds\[, permissionChecker\])
- `category`: [``][string] category with field having the Many decorator
- `field`: [``][string] field with the Many decorator
- `fromId`: [``][uint64] Id of the record in category specified in the
first argument
- `toIds`: [``][uint64]|[``][uint64] Id(s) of the record(s) in
category specified in the Many decorator of the specified field
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Unlink records with Many relation between them
#### StorageProvider.prototype.select(category, query\[, permissionChecker\])
- `category`: [``][string] category to select the records from
- `query`: [``][object] fields conditions
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][cursor]
Select objects from GlobalStorage
#### async StorageProvider.prototype.execute(category, action, actionArgs\[, permissionChecker\])
- `category`: [``][string]|[``][null] category name or null to
execute public action
- `action`: [``][string] action name
- `actionArgs`: [``][object]
- `context`: [``][object]
- `args`: [``][object]
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `action`: [``][string]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Execute an action
#### StorageProvider.prototype.log(op, args, ctx, response)
#### StorageProvider.prototype.getSystemSuffix(id)
- `id`: [``][uint64]
_Returns:_ [``][uint64]
Get system suffix for given id
#### StorageProvider.prototype.curSystem(id)
- `id`: [``][uint64]
_Returns:_ [``][boolean]
Check whether data with given id is stored on this system
#### StorageProvider.prototype.getServerSuffix(id)
- `id`: [``][uint64]
_Returns:_ [``][uint64]
Get server suffix for given id
#### StorageProvider.prototype.curServer(id)
- `id`: [``][uint64]
_Returns:_ [``][boolean]
Check whether data with given id is stored on this server
#### StorageProvider.prototype.getLocalId(id)
- `id`: [``][uint64]
_Returns:_ [``][uint64]
Get id without system and server suffix
#### StorageProvider.prototype.parseId(id)
- `id`: [``][uint64]
_Returns:_ [``][object]
- `systemSuffix`: [``][uint64] system suffix for given id
- `serverSuffix`: [``][uint64] server suffix for given id
- `localId`: [``][uint64] id without system and server suffix
Parse id
#### async StorageProvider.prototype.listApplications(\[filtererByRoles\])
- `filtererByRoles`: [``][function] optional
- `applications`: [``][string]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
List all available applications
#### async StorageProvider.prototype.listCategories(\[filtererByPermission\])
- `filtererByPermission`: [``][function] optional
- `categories`: [``][string]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
List all available categories
#### async StorageProvider.prototype.listActions(\[filtererByPermission\])
- `filtererByPermission`: [``][function] optional
- `actions`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
List all available actions
#### StorageProvider.prototype.createJstpApi()
#### async StorageProvider.prototype.getSchemaSources({ categoryList, actionLists, appList } = {})
#### async StorageProvider.prototype.getCategoryL10n(langTag, category)
#### async StorageProvider.prototype.getDomainsL10n(langTag)
#### async StorageProvider.prototype.getCommonL10n(langTag)
#### async StorageProvider.prototype.getFormL10n(langTag, category, form)
#### async StorageProvider.prototype.getActionL10n(langTag, category, action)
### class FsProvider extends [StorageProvider][storageprovider]
#### FsProvider.prototype.constructor(options // { path } where path is database location)
#### FsProvider.prototype.readStat(callback)
#### FsProvider.prototype.open(options, callback)
#### FsProvider.prototype.writeStat(callback)
#### FsProvider.prototype.close(callback)
#### FsProvider.prototype.takeId(callback)
#### FsProvider.prototype.get(id, callback)
#### FsProvider.prototype.create(obj, callback)
#### FsProvider.prototype.update(obj, callback)
#### FsProvider.prototype.delete(id, callback)
#### FsProvider.prototype.select(query, options, callback)
### class MemoryProvider extends [StorageProvider][storageprovider]
#### MemoryProvider.prototype.constructor(callback)
#### MemoryProvider.prototype.close(callback)
#### MemoryProvider.prototype.create(obj, callback)
### class PostgresProvider extends [StorageProvider][storageprovider]
#### PostgresProvider.prototype.constructor()
Create PostgresProvider
#### async PostgresProvider.prototype.open(options)
- `options`: [``][object] to be passed to pg
_Returns:_ [``][promise]
Open PostgresProvider
#### async PostgresProvider.prototype.close()
_Returns:_ [``][promise]
Close PostgresProvider
#### async PostgresProvider.prototype.setup(options)
- `options`: [``][object]
- `maxIdCount`: [``][number]
- `refillPercent`: [``][number]
_Returns:_ [``][promise]
Setup StorageProvider
#### async PostgresProvider.prototype.takeId(client)
- `client`: [``][pg.pool]|[``][pg.client]
_Returns:_ [``][promise]
Generate globally unique id
#### async PostgresProvider.prototype.getCategoryById(id)
#### async PostgresProvider.prototype.beginTx(\[options\])
- `options`: [``][object] transaction options
- `isolationLevel`: [``][string] 'committed', 'repeatable', or
'serializable'
- `readOnly`: [``][boolean]
- `deferrable`: [``][boolean]
_Returns:_ [``][promise]
Begin transaction, returns a Promise that resolves in an object containing
some of the methods of the current provider and also the methods `commit()`,
`rollback()`, and `release()`. For more detailed description of the options see
https://www.postgresql.org/docs/current/sql-set-transaction.html
#### async PostgresProvider.prototype.get(id\[, permissionChecker\])
- `id`: [``][string] globally unique object id
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Get object from GlobalStorage
#### async PostgresProvider.prototype.getDetails(category, id, fieldName\[, permissionChecker\])
- `category`: [``][string] category to get details in
- `id`: [``][string] object id
- `fieldName`: [``][string] field with the Many decorator
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Get details for many-to-many link from GlobalStorage
#### async PostgresProvider.prototype.set(obj\[, permissionChecker\])
- `obj`: [``][object] to be stored
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Set object in GlobalStorage
#### async PostgresProvider.prototype.create(category, obj\[, permissionChecker\])
- `category`: [``][string] category to store the object in
- `obj`: [``][object] to be stored
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Create object in GlobalStorage
#### async PostgresProvider.prototype.update(category, query, patch\[, permissionChecker\])
- `category`: [``][string] category to update the records in
- `query`: [``][object] example: `{ Id }`
- `patch`: [``][object] fields to update
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Update object in GlobalStorage
#### async PostgresProvider.prototype.delete(category, query\[, permissionChecker\])
- `category`: [``][string] category to delete the records from
- `query`: [``][object] example: `{ Id }`
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Delete object in GlobalStorage
#### async PostgresProvider.prototype.linkDetails(category, field, fromId, toIds\[, permissionChecker\])
- `category`: [``][string] category with field having the Many decorator
- `field`: [``][string] field with the Many decorator
- `fromId`: [``][uint64] Id of the record in category specified in the
first argument
- `toIds`: [``][uint64]|[``][uint64] Id(s) of the record(s) in
category specified in the Many decorator of the specified field
- `permissionChecker`: [``][function] optional
- `category`: [``][string]
- `options`: [``][object]
- _Returns:_ [``][promise]
_Returns:_ [``][promise]
Link records with Many relation between them
#### async PostgresProvider.prototype.unlinkDetails(category, field, fromId, toIds\[, permissionChecker\])
- `category`: [`