Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/anywhichway/thunderclap

A key-value, indexed JSON, and graph database plus function oriented server designed for Cloudflare
https://github.com/anywhichway/thunderclap

Last synced: about 2 months ago
JSON representation

A key-value, indexed JSON, and graph database plus function oriented server designed for Cloudflare

Awesome Lists containing this project

README 

        


# thunderclap



Thunderclap is a key-value, indexed JSON, and graph database plus function oriented server designed specifically for Cloudflare. It runs on top of the

Cloudflare KV store. Its [query capability](#joqular) is supported by [JOQULAR](https://medium.com/@anywhichway/joqular-high-powered-javascript-pattern-matching-273a0d77eab5) 

(JavaScript Object Query Language), which is similar to, but more extensive than, the query language associated with MongoDB. 

In addition to having more predicates than MongoDB, JOQULAR extends pattern matching to object properties, e.g.



```javascript

// match all objects with properties starting with the letter "a" containing the value 1

db.query({Object:{[/a.*/]:{$eq: 1}}}) 

```



Thunderclap uses a [JavaScript client](#javascript) client to support:



1) [Special storage for Infinity, NaN, Dates](#special-storage)



2) [Built in User, Edge, Position, and Coordinate Classes](#built-in-classes)



3) [role based access control mechanisms](#access-control)



4) [inline analytics and hooks](#analytics)



5) [triggers](#triggers)



6) [custom functions (with access control)](#functions)



7) [full text indexing and search](#full-text) in addition to [regular property indexing](#indexing)



8) [schema based or schemaless operation](#schema)



9) [an admin UI](#admin)



A [URL fetch (CURL)](#curl) capability is also supported.



Like MongoDB, Thunderclap is open-sourced under the Server Side Public License. This means licencees are free to use and 

modify the code for internal applications or public applications that are not primarily a means of providing Thunderclap

as a hosted service. In order to provide Thunderclap as a hosted service you must either secure a commercial license from 

AnyWhichWay or make all your source code available, including the source of non-derivative works that support the 

monitoring and operation of Thunderclap as a service.



# Important Notes



Thunderclap is currently in ALPHA because:



1) Workers KV from Cloudflare recently came out of beta and is missing a few key features that are "patched" by 

Thunderclap.



2) Security measures are incomplete and have not yet been vetted by a third party.



3) Although there are many unit tests, application level functional testing has been limited.



4) The source code could do with a lot more comments.



5) Project structure does not currently have a clean separation between what people might want to change

for their own use vs submit as a pull request. In general, changes to file in the `src` directory are

candidates for pull requests and with the exception of this README those outside are not.



6) It has not been performance tuned.



7) It is highly likely you will need to re-create your NAMESPACES with every new ALPHA release.



8) APIs are not yet stable.



9) It could do with contributors!



# Installation and Deployment



Clone the repository https://www.github.com/anywhichway/thunderclap.



Run `npm install`.



## Production



NOTE: While the software is in ALPHA state, you should probably not deploy to a production Cloudflare server that

is not behind Cloudflare's paid Access management interface.



When Thunderclap is running in production, it will be available at `thunderclap.`. When it is running in 

[development mode](#development), it will be available at `-thunderclap.`. You can choose

the `dev-host-prefix`.



You can deploy and use Thunderclap immediately after creating and populating a `thunderclap.json` configuration file,

creating a KV namespace, and establishing a CNAME alias `thunderclap` in the Cloudflare DNS control panel. This can 

just point to your root, you do not need a distinct IP address, Cloudflare's smart routers will send requests to the 

Thunderclap Cloudflare Worker.



Create a Cloudflare Workers KV namespace using the Cloudflare Workers control panel. By convention, the following 

name form is recommended so that the name parallels the name of thw worker script generated by Thunderclap.



`thunderclap--`



e.g. `thunderclap-mydomain-com` is the KV namespace and script name associated with `thunderclap.mydomain.com`



You will need to populate a file `thunderclap.json` with many of your Cloudflare ids or keys. Copy the file 

`thunderclap.template.json` to `thunderclap.json` and replace the placeholder values. This will contain secret keys,

so you may want to move it out of your project directory to avoid having it checked-in. The `thunderclap` script in

`webpack.config.js` assumes the file is up one level in the directory tree. If you do leave it in the project

directory .gitignore is configured to not check it in. But, you will need to edit `webpack.config.js` so that it can

find `thunderclap.json`.



The .gitignore is also set to ignore `dbo.js`, which contains the default `dbo` password and `keys.js` which contains

a function definition for iterating over keys on the server that requires special credentials. None of this data is 

built into the browser software.



You will need to place the file `db.json` at the root of your web server's public directory and `thunderclap.js` 

in your normal JavaScript resources directory. For convenience, `db.json` and `thunderclap.js` are located in the `docs` 

subdirectory of the repository so you can host them using GitHub Pages if you wish.



You can use Thunderclap without making any modifications by setting the `mode` in `thunderclap.json` to `production`

and running `npm run thunderclap`. This will deploy the Thunderclap worker and a route. Don't forget to deploy `db.json`

and `thunderclap.js` also.



See the files in `docs` and `docs/test` for examples of using Thunderclap.





# Data Manipulation [top](#top)



Data in Thunderclap can be manipulated using a JavaScript client or CURL.



## JavaScript Client [top](#top)



```javascript



const endpoint = "https://thunderclap.mydomain.com/db.json",

	username = "<get from somewhere>", 

	password = "<get from somewhere>",

	// there is a default user `dbo`, with a default password `dbo`.

	db = new Thunderclap({endpoint,user:{username,password}});



```



`boolean async addRoles(string userName,Array [string role,...]=[])` - Assigns roles to the named user. Returns `true` on success. By default,

only a user with the role `dbo` can call this function.



`undefined async clear(string prefix="")` - Deletes items whose keys start with `prefix`. By default it can only be 

called by a user with the `dbo` role. See the section on [Access Control](#access-control) to change this.



`string async changePassword(string userName,string password,string oldPassword)` - Changes the password from

`oldPassword` to `password` for the user with name `userName` and old password `oldPassword`. If `password` is not 

provided, a random 10 character password is generated and returned. If the currently authenticated user has the role 

`dbo` and is not the user for whom the password was being changed, `oldPassword` can be omitted. If a password

reset process has been inititiated with `resetPassword`, then `oldPassword` should be the temporary password or mobile

code.



`User async createUser(string userName,string password,object extras={},reAuth)` - Creates a user. The password is 

stored on the server as an SHA-256 hash and salt. `createUser` can be called even if Thunderclap is started without 

a username and password. If this is done and `createUser` succeeds, the Thunderclap instance is bound to the new user 

for immediate authenticated use. The `extras` argument can be any additional data to be stored in the user object except roles. 

If `reAuth` is truthy and the Thunderclap instance was already authenticated, the Thunderclap instance will also be re-bound 

to the new user. You can implement access control and account creation logic on the server to prevent the creation of 

un-authorized accounts. See the section on [Access Control](#access-control).



`boolean async deleteUser(string userName)` - Deletes a user. Returns `true` if user was deleted or did not exist. By default,

only a user with the role `dbo` can call this function.



`Array async entries(string prefix="",{number batchSize,string cursor})` - Returns an array or arrays for keys,

values and optionally expirations of data with keys that start with `prefix`.  By default,

only a user with the role `dbo` can call this function. Expirations are in Unix epoch milliseconds,

e.g. `entries("Person@")` might return:



```javascript

[["Person@jxmc9cc1kswqak4ga",{"name":"joe"},1562147669820],["Person@jxmcnqkx9irjhrz4p",{"name":"joe"}]]

```



Entries can be used in a loop just like `keys` below.



`Array entry(string key)` - Returns the entry for a key as a two or three element array or `undefined`. By default,

only a user with the role `dbo` can call this function. For example, `entry("Person@jxmc9cc1kswqak4ga")` 

might return:



```javascript

["Person@jxmc9cc1kswqak4ga",{"name":"joe"},1562147669820]

``` 



`Edge async get(string|Array path)` - Returns an `Edge` in a graph data store. The `path` can be an Array or a dot delimited string, e.g.

`["people","joe"]` is the same as "people.joe".



`User async getUser(string userName)` - Returns the User with the `userName` or `undefined`. By default can only be executed 

by a user with the role `dbo` or the named user itself.



`any async getItem(string key)` - Gets the value at `key`. Returns `undefined` if no value exists.



`boolean async hasKey(string key)` - Returns `true` if `key` exists.



`Array async keys(prefix="",{number batchSize=1000,string cursor,boolean expanded})` - Returns an Array of the next `batchSize`

keys in database than match the `prefix` every time it is called. By convention, the last value in the array is the cursor.

By default it can only be called by a user with the `dbo` role. A loop can be used to process all keys:



```javascript

	let cursor;

	do {

		keys = await mythunder.keys("",{cursor});

		cursor = keys.pop();

		keys.forEach((key) => dosomething(key));

	} while(cursor && keys.length>0)

```



`any async putItem(object instance,options={})` - Adds a unique id on property "#" of `instance`, if one does not exist, 

indexes the object and stores it with `setItem` using the id as the key. In most cases, the unique id will be of the form 

`@xxxxxxxxxxxxx`. The `options` can be one of: `{expiration: secondsSinceEpoch}` or `{expirationTtl: secondsFromNow}`.



`boolean async removeItem(string|object keyOrObject)` - Removes the keyOrObject. If the argument is an indexed object 

or a key that resolves to an indexed object, the index entries are also removed from the database so long as the user has 

the appropriate privileges. If the key exists but can't be removed, the function returns `false`. If the key does not exist

or removal succeeds, the function returns `true`.



`boolean async removeRoles(string userName,Array [string role,...]=[])` - Removes roles from the named user. Returns `true` on success. By default,

only a user with the role `dbo` can call this function.



`string async resetPassword(string userName,string method="email"||"mobile")` - COMING SOON. Initiates a password reset process for 

the `userName`. `method` defaults to "email". The `User` object stored in the database must have an `email` or `mobile` property. The "mobile" 

option requires Neutrino keys in `thunderclap.json` and "email" requires Mailgun keys. You must build a UI that calls `changePassword` 

with the code sent to the user as the `oldPassword` argument. Mobile codes are good for 5 minutes. Email codes are good for 30 minutes. The

current password on the user is not changed until `changePassword` is called. By default `resetPassword` can only be executed by a `dbo` or

the current user for for themself.



`boolean async sendMail({to:Array [string emailAddress,...],{cc:Array},{bcc:Array},{subject:string},{body:string})` - 

Send's email. Requires providing Mailgun keys in `thunderclap.json`. The `from` for the e-mail is always the currently authenticated user's e-mail. 

This function should be access controlled in `secure.js`. By default, only users with the role `dbo` can send mail. Developers might

consider adding a role `mailsenders`. Returns `true` if mail was successfuly sent, `false` if there was no e-mail address for the

autheticated user, and error text with a 500 HTTP status if there was some other cause of failure.



`any async setItem(string key,any value,options={})` - Sets the `key` to `value`. If the `value` is an object it is 

NOT indexed. Options can one of: `{expiration: secondsSinceEpoch}` or `{expirationTtl: secondsFromNow}`.



`Array async query(object {:JOQULARPattern},{boolean partial,number limit)` - Query the database using `JOQULARPattern`. 

If `partial` is truthy, then only those properties used in the query will actuall be returned. The `limit` defaults internally to

1000 items. See [JOQULAR](#joqular) below.



`boolean async unique(object|string cnameOrIdOrObject,property,value)` - Returns true if the `value` on `property` is or will be unique for

the provided `cnameOrIdOrObject`. If a string class name is provided and true is returned, the the `value` will be unique for the class name

and property when added to the database. If a full object id, e.g. `Object@jy34s5bz1fkqseh8j` is provided, then either the value will

be unique when the object is added or the object exists and has the unique value. Passing in an actuall object just has its id pulled

for use in the call to the server.



`User async updateUser(string userName,properties={})` - COMING SOON. Update the user with the provided properties. The currently 

autheticated user must be a `dbo` or the target user. To delete values, use a value of `undefined` for a property. The properties

`role`, `password`, `hash`, and `salt` properties are ignored. By default `resetPassword` can only be executed by a `dbo` or

the current user for for themself.



`any async value(string|Array path [,any value [,object options={}])` - With no optional arguments, returns the value stored at 

the edge found at `path` in a graph data store. The `path` can be an Array or a dot delimited string, e.g. `["people","joe"]` 

is the same as "people.joe". With the optional argument `value`, sets the value at the edge. The `options` can be one 

of: `{expiration: secondsSinceEpoch}` or `{expirationTtl: secondsFromNow}`.



`Array async values(string prefix="",{number batchSize,string cursor})` - Returns all the data associated with keys that

start with `prefix`. By default it can only be called by a user with the `dbo` role. It can be used in a loop just like 

`keys` above.





## CURL Requests [top](#top)



To be written





# Special Storage [top](#top)



Most JavaScript data stores do not support special values like `undefined`, `Infinity` and `NaN`. Thunderclap 

serializes these as special strings, e.g. `@Infinity`. However, this is transparent to API calls via the JavaScript

client and should only be of concern to those who are customizing or extending Thuderclap.



The Thunderclap client also serializes dates as `Date@` and restores them to full-fledged dates after

transport.





# Built-in Classes [top](#top)



## User [top](#top)



Thunderclap provides a basic `User` class accessable via `new Thunderclap.User({string userName,object roles={user:true}})` 

and `createUser(string userName,string password,[object extras],[boolean reauth])`. Developers are free to add other 

properties and values to the constructor argument or `extras` object. Additional role keys may also be added to the roles 

sub-object. The only built-in roles are `user` and `dbo`. See Access Control(#access-control) for more detail. To actually 

create a user and store it in the database use `createUser`.



## Edge [top](#top)



The `Edge` object is used to support graph database options. The graph API is similar to, but no identical to, the 

GunDB graph API. It has a number of methods:



`Edge async add(string|Array path,any data[,options={}])` - Adds data to a Set at `path`.



`number async delete(string|Array path)` - Deletes the sub-graph, if any, at `path`. Returns the number of nodes deleted.



`Edge async get(string|Array path)` - Gets the sub-edge at `path`.



`object async put(object data)` - Explodes the object into a sub-graph on the current Edge.



`Edge async remove(string|Array path,any data)` - Removes data from a Set at `path`.



`any async value(string|Array path [,any value [,object options={}])` - Get's or sets the value associated with the Edge.



## Coordinates [top](#top)



For convenience, Thunderclap exposes a Coordinates object with the same properties as the [JavaScript browser standard

interface](https://developer.mozilla.org/en-US/docs/Web/API/Coordinates) `{latitude,longitude,altitude,accuracy,altitudeAccuracy,heading,speed}`. Coordinates can be created directly with:



```javascript

new Thunderclap.Coordinates({Coordinates coords,number timestamp});

```

There is also an asynchronous `Thunderclap.Coordinates.create([Coordinates coords])`. If the

optional argument is not provided, then the browser `navigator.geolocation.getCurrentPosition` will be called

to get the values to create the Coordinates. This makes it easy to deploy clients that automatically collect and store

location data.



## Position [top](#top)



For convenience, Thunderclap exposes a Position object with the same properties as the [JavaScript browser standard

interface](https://developer.mozilla.org/en-US/docs/Web/API/Position) `coords` and `timestamp`. Positions can be created 

directly with:



```javascript

new Thunderclap.Position({Coordinates coords,number timestamp});

```



There is also an asynchronous `Thunderclap.Position.create([{Coordinates coords,number timestamp}])`. If the

optional argument is not provided, then the browser 

[navigator.geolocation.getCurrentPosition](https://developer.mozilla.org/en-US/docs/Web/API/Geolocation/getCurrentPosition) 

will be called to get the values to create the Position. This makes it easy to deploy clients that automatically collect and store

location data.





# JOQULAR [top](#top)



Thunderclap supports a subset of JOQULAR. It is simlar to the MongoDB query language but more extensive. You can see many

examples in the unit test file `test/index.js`. 



Here is a basic query that returns all Users of age 21 or greater in zipcode 98101:



```javascript

const db = new Thunderclap({endpoint,user:{username:"",password:""}}),

	results = await db.query({User:{age:{$gte: 21},address:{zipcode:98101}}});

```



Thuderclap also suppport pattern matching on property names:



```javascript

db.query({Object:{[/a.*/]:{$eq: 1}}}) // match all Objects with properties starting with the letter "a" containing the value 1

```



You may have noted above, the top level property names in a query should be class names. In MongoDB, these would 

be collections. Using different top level keys you can query multiple "collections" at the sam etime. If you use the

top leve wild card key `_`, all "collections" will be searched.



The supported patterns are described below. All examples assume these two objects exist in the database:



```javascript

const o1 = {

		#:"User@jxxtlym2fxbmg0pno",

		userName:"joe",

		age:21,

		email: "[email protected]",

		SSN: "555-55-5555",

		registeredIP: "127.0.0.1",

		address:{city:"Seattle",zipcode:"98101"},

		registered:"Tue, 15 Jan 2019 05:00:00 GMT",

		favoritePhrase:"to be or not to be, that is the question"

	},

	o2 = {

		#:"User@jxxviym2fxbmg0pcr",

		userName:"mary",

		age:20,

		address:{city:"Bainbridge Island",zipcode:"98110"},

		registered:"Tue, 15 Jan 2019 10:00:00 GMT",

		favoritePhrase:"premum non nocere"

	};



```



The supported patterns include the below. (If a pattern is not documented, it may not have been tested yet. See the

unit test file `docs/test/index.js` to confirm.):



## Math and String Comparisons [top](#top)



`{$lt: number|string value}` - A value in a property is less than the one provided, e.g. `{age:{$lt:21}}` matches o2.



`{$lte: number|string value}` - A value in a property is less or equal the one provided, e.g. `{age:{$lte:21}}` matches o1 and o2.



`{$eq: number|string value}` - A value in a property is relaxed equal the one provided, e.g. `{age:{$eq:21}}` and `{age:{$eq:"21"}}` match o1.



`{$eeq: number|string value}` - A value in a property is exactly equal the one provided, e.g. `{age:{$eeq:21}}` matches o1 but and `{age:{$eeq:"21"}}` does not.



`{$neq: number|string value}` - A value in a property is relaxed equal the one provided, e.g. `{age:{$neq:21}}` matches o2.



`{$gte: number|string value}` - A value in a property is greater than or equal the one provided, e.g. `{age:{$gte:20}}` matches o1 and o2.



`{$gt: number|string value}` - A value in a property is greater than the one provided, e.g. `{age:{$gt:20}}` matches o1 and o2.



## String Tests



`{$startsWith: string value}` - A value in a property starts with the one provided.



`{$endsWith: string value}` - A value in a property ends with the one provided.



## Logical Operators [top](#top)



`{$and: Array}` -  Ands multiple conditions, e.g. `{age:{$and:[{$gt:20},{$lt: 30}]}` matches o1 and o2. Typically not required

because this produces the same result, `{age:{$gt:20,$lt: 30}}`.



`{$not: JOQULARExpression}` - Negates the contained condition, e.g. `{age:{$not:{$gt:20}}}` matches o2.



`{$or: Array}` - Ors multiple conditions, e.g. `{age:{$or:[{$eq:20},{$eq: 21}]}` matches o1 and o2. Allow repeating

the same predicate for a single property. The nested form is also supported, `{age:{$eq:20,$or:{$eq: 21}}}`



`{$xor: Array}` - Exclusive ors multiple conditions.



## Date and Time [top](#top)



The full range of methods available for extracting parts from Date on a native JavaScript Date instance are also 

available as predicates:



`{$date: number dayOfMonth}` - `{$date: 14}` matches o1 in EST.



`{$day: number dayOfWeek}` - `{$day: 2}` matches o1 in EST.



`{$fullYear: number fourDigitYear}` - `{$fullYear: 2019}` matches o1 in EST.



`{$hours: number hours}` - `{$hours: 5}` matches o1 in EST.



`{$milliseconds: number ms}` - `{$milliseconds: 0}` matches o1.



`{$minutes: number minutes}` - `{$minutes: 0}` matches o1.



`{$month: number month}` - `{$month: 0}` matches o1.



`{$seconds: number seconds}` - `{$seconds: 0}` matches o1.



`{$time: number time}` - `{$time: 1547528400000}` matches o1.



`{$UTCDate: number dayofMonth}` - `{$date: 14}` matches o1.



`{$UTCDay: number dayOfWeek}` -  `{$day: 2}` matches o1.



`{$UTCFullYear: number fourDigitYear}` - `{$fullYear: 2019}` matches o1.



`{$UTCHours: number hours}` - `{$hours: 5}` matches o1.



`{$UTCMilliseconds: number ms}` - `{$milliseconds: 0}` matches o1.



`{$UTCMinutes: number minutes}` - `{$minutes: 0}` matches o1.



`{$UTCMonth: number month}` - `{$month: 0}` matches o1.



`{$UTCSeconds: number seconds}` - `{$seconds: 0}` matches o1.



`{$year: number 2digitYear}` - `${year: 19}` matches o1.



## Membership [top](#top)



`{$in: Array|string container}` - A value in a property is in the provided container, e.g. `{age:{$in:[20,21,22]}}` matches o1 and o2.



`{$nin: Array|string container}`  - A value in a property is not in the provided container, e.g. `{age:{$nin:[21,22,23]}}` matches o2.



`{$includes: boolean|number|string included}` - A value in a property (Array or string) includes `included`.



`{$intersects: Array|string container}` - A value in a property (Array or string) intersects `container`.



`$disjoint` -



## Ranges [top](#top)



`{$between: Array [number|string bound1,number|string bound2,inclusive]}` - A value in a property in between the two provided 

limits. The limits can be in any order, e.g. `{age:{$between:[19,21]}}` or `{age:{$between:[21,19]}}` matches o2. Optionally, 

the limits can be inclusive, e.g. `{age:{$between:[19,21,true]}}` matches o1 and o2.



`{$outside: Array [number|string bound1, number|string bound2]}` - A value in a property in outside the two provided limits. 

The limits can be in any order, e.g. `{age:{$outside:[19,20]}}` or `{age:{$between:[20,19]}}` matches o1.



`{$near: Array [number target,number|string absoluteOrPercent]}` - A value in a property is near the provided number either 

from an absolute or percentage perspective, e.g. `{age:{$near:[21,1]}}` matches both o1 and o2 as does 

`{age:{$near:[21,"5%"]}}` since 1 is 4.7% of 21.



## Regular Expression [top](#top)



`{$matches: RegExp|string pattern}` - A value in a property matches the provided regular expression. The regular expression 

can be a string that looks like a regular expression or an actual regular expression, e.g. `{userName:{$matches:/a.*/}}`

or `{userName:{$matches:"/a.*/"}}`



## Special Tests [top](#top)



`{$instanceof: string className}` - A value in a property is an instanceof the class denoted by the string argument. The 

class must  be registered on the server. Currently this includes Object, Array, Data, User, Schema, Position, and Coordinates.

You can add more classes by modifying the file `classes.js`.



`{$isa: string className}` - A value in a property is of the class provided by the string argument. Note, this is not 

an `instanceof` test, it does not walk the inheritance tree.



Note that other special tests typically take `true` as an argument. This is an artifact of JSON format that does not allow

empty properties. Passing anything else will cause them to fail. You may occassionaly want to pass `false` to match

things that do not satisfy the test.



`{$isCreditCard: boolean value}` - A value in a property is a valid credit card based on a regular expression and Luhn algorithm.



`{$isEmail: boolean value}` - A value in a property is a valid e-mail address by format, e.g. `{email:{$isEmail: true}}`. Note:

e-mail addresses are remarkably hard to validate without actually trying to send and e-mail. This will address

all reasonable cases.



`{$isEven: boolean value}` - A value in a property is even, e.g. `{age:{$isEven: true}}` matches o2.



`{$isFloat: boolean value}` - A value in a property is a float, e.g. `{age:{$isFloat: true}}` will not match either o1 or o2. Note,

0 and 0.0 are both treated as 0 by JavaScript, so 0 will never satisfy $isFloat.



`{$isIPAddress: boolean value}` - A value in a property is a dot delimited IP address, e.g. `{registeredIP:{$isIPAddress: true}}



`{$isInt: boolean value}` - A value in a property is a dot delimited IP address, e.g. `{registeredIP:{$isIPAddress: true}} matches o1.



`{$isNaN: boolean value}` - A value in a property is a not a number, e.g. `{address:{zipcode:{$isNaN: true}}} matches o1. Note, $isNaN

will fail when there is no value since it is no known whether the target is a number or not.



`{$isOdd: boolean value}` - A value in a property is odd, e.g. `{age:{$isOdd: true}}` matches o1.



`{$isSSN: boolean value}` - A value in a property looks like a Social Security Number, e.g. `{SSN:{$isSSN: true}}` matches o1. Note,

unlike `$isCreditCard` no validation is done beyond textual format.



## Text Search [top](#top)



`{$echoes: string soundALike}` - A value in a property sounds like the provided value, e.g. `{userName:{$echoes: "jo"}}` matches o1.





`{$search: string searchPhrase}` - Does a full text trigram based search, e.g. `{favoritePhrase:{$search:"question"}}` matches o1. 

If no second argument is provided, the search is fuzzy at 80%, e.g. `{favoritePhrase:{$search:"questin"}}` also

matches o1 whereas `{favoritePhrase:{$search:["questin",.99]}}`, which requires a 99% match does not. The search

phrase can contain multiple space separated words.



## Special Predicates



`{$_:any value}` - Matches any property that has `value`.



`{"$.":[string functionName,...args]}` - Calls the `functionName` on the value in the property, e.g. `{name:{$startsWith:"ma"}}`

is the same as `{name:{"$.":["startsWith","ma"]}}`. By default, queries that use this predicate will return a HTTP status code 403

for users that do not have a `dbo` role.



`{"$.":[...args]}` - Similar to `$.`, except the function name is part of the property. By default, queries that

use this predicate will return a HTTP status code 403 for users that do not have a `dbo` role.





# Access Control [top](#top)



The Thunderclap security mechanisms support the application of role based read and write access rules for functions,

objects, properties, keys, and edges. 



If a user is not authorized read access to an object, key value or edge, it will not be returned. If a user is not 

authorized access to a particular property, the property will be stripped from the object before the

object is returned. Additionally, a query for an object using properties to which a user does not have access

will automatically drop the properties from the selection process to prevent data leakage through inference.



If a user is not authorized write access to specific properties on an object, update attempts will 

fall back to partial updates on just those properties for which write access is allowed. If write access to a

key, entire object, or edge is not allowed, the write will simply fail and return `undefined`.



At the moment, by default, all keys, objects, and properties are available for read and write unless specifically

controlled in the `secure.js` file in the root of the Thunderclap repository. A future release will support defaulting

to prevent read and write unless specifically permitted.



If the user is not authorized to execute a function a 403 status will be returned.



The default `secure.js` file is show below.



```javacript

(function() {

	module.exports = {

		"Function@": {

			securedTestFunction: { // for testing purposes

				execute: [] // no execution allowed

			},

			addRoles: { // only dbo can add roles to a user

				execute: ["dbo"]

			},

			clear: { // only dbo can clear

				execute: ["dbo"]

			},

			deleteUser: {

				execute: ["dbo"]

			},

			entries: { // only dbo can list entries

				execute: ["dbo"]

			},

			entry: {

				execute: ["dbo"]

			},

			keys: { // only dbo can list keys

				execute: ["dbo"]

			},

			removeRoles: {

				execute: ["dbo"]

			},

			resetPassword: { // only user themself or dbo can start a password reset

				execute({argumentsList,user}) {

					return user.roles.dbo || argumentsList[0]===user.userName

				}

			},

			sendMail: { // only dbo can send mail

				execute: ["dbo"]

			},

			updateUser: {  // only user themself or dbo can update user properties

				execute({argumentsList,user}) {

					return user.roles.dbo || argumentsList[0]===user.userName

				}

			},

			values: { // only dbo can list values

				execute: ["dbo"]

			}

		},

		"User@": { // key to control, use @ for classes

			

			// read: ["",...], // array or map of roles to allow get, not specifying means all have get

			// write: {:true}, // array or map of roles to allow set, not specifying means all have set

			// a filter function can also be used

			// action with be "get" or "set", not returning anything will result in denial

			// not specifying a filter function will allow all get and set, unless controlled above

			// a function with the same call signature can also be used as a property value above

			filter({action,user,data,request}) {

				// very restrictive, don't return a user record unless requested by the dbo or data subject

				if(user.roles.dbo || user.userName===data.userName) {

					return data;

				}

			},

			keys: { // only applies to objects

				roles: {

					// only dbo's and data subject can get roles

					get({action,user,object,key,request}) { return user.roles.dbo || object.userName===user.userName; }, 

				},

				hash: {

					// only dbo's can get password hashes

					read: ["dbo"],

					// only the dbo and data subject can set a hash

					set({action,user,object,key,request}) { return user.roles.dbo || object.userName===user.userName; },

				},

				salt: {

					// example of alternate control form, only dbo's can get password salts

					read: {

						dbo: true

					},

					// only the dbo and data subject can set a salt

					set({action,user,object,key,request}) { return user.roles.dbo || object.userName===user.userName; },

				},

				name({action,user,data,request}) { return data; } // example, same as no access control

			}

			/* keys could also be a function

			keys({action,user,data,key,request})

			 */

		},

		securedTestReadKey: { // for testing purposes

			read: [] // no gets allowed

		},

		securedTestWriteKey: { // for testing purposes

			write: [] // no sets allowed

		},

		[/\!.*/]: { // prevent direct index access by anyone other than a dbo, changing this may create a data inference leak

			read: ["dbo"],

			write: ["dbo"]

		}

		/* Edges are just nested keys or wildcards, e.g.

		people: {

			_: { // matches any sub-edge

				secretPhrase: { // matches secrePhrase edge

					read(...) { ... },

					write(...) { ... }

				}

			}

		}

		*/

	}

}).call(this);

```



Roles can also be established in a tree that is automatically applied at runtime. See the file `roles.js`.



When Thunderclap is first initialized, a special user `User@dbo` with the user name `dbo` the role `dbo` and the

dbo password defined in `thunderclap.json` is created. It also has the unique id `User@dbo`.



You can create additional accounts with the `createUser` method and change passwords with the `changePassword` 

method.





# Inline Analytics & Hooks [top](#top)



Inline analytics and hooks are facilitated by the use of JOQULAR patterns or edge specficications and tranform or hook 

calls in the file `when.js`. The transforms and hooks can be invoked from the browser, a service worker, or in the cloud. 

They are not currently access controlled in the browser or a service worker. In the cloud, transforms are invoked after 

it is determined primary key access is allowed but before data property access is assesed and the data is written. 

This security is applied to the transformed data. Hooks are called after the data is written. If you need to transform 

something and call a hook, but not write to the database either call the hook as the last action in the transform and 

return nothing, or use a before trigger.



Below is an example.



```javascript

(function() {

	module.exports = {

		client: [

			{

				when: {testWhenBrowser:{$eq:true}},

				transform({data,pattern,user,request}) {

					Object.keys(data).forEach((key) => { if(!pattern[key]) delete data[key]; });

					return data;

				},

				call({data,pattern,user,request}) {

					

				}

			}

		],

		worker: [

			// not yet implemented

		],

		cloud: [

			{

				when: {testWhen:{$eq:true}},

				transform({data,pattern,user,request}) {

					Object.keys(data).forEach((key) => { if(!pattern[key]) delete data[key]; });

					return data;

				},

				call({data,pattern,user,request,db}) {

					

				}

			}

		]

	}

}).call(this);



```





# Triggers [top](#top)



Triggers can get invoked before and after key value or indexed object properties change or get deleted. The triggers are configured

in the file `on.js`. Any asynchronous triggers will be awaited. `before` triggers must return truthy for execution to

continue, i.e. a before on set that returns false will result in the set aborting. `before` triggers are fired immediately

before security checks. Triggers are not access controlled.



Triggers can be executed in the browser, a service worker, or the cloud.



```javascript

(function() {

	module.exports = {

		client: {

			

		},

		worker: {



		},

		cloud: {

			"User@": {

				read({value,key,user,request}) {

					; // called after get

				},

				write({value,key,oldValue,user,request}) {

					// if value!==oldValue it is a change

					// if oldValue===undefined it is new

					// if value===undefined it is delete

					; // called after set

				},

				execute({value,key,args,user,request}) {

					; // called after execute, value is the result, key is the function name

				},

				keys: {

					password: {

						write({value,key,oldValue,user,request}) {

							; // called after set

						}

					}

				}

			}

			/* Edges are just nested keys or wildcards, e.g.

			people: {

				_: { // matches any sub-edge

					secretPhrase: { // matches secrePhrase edge

						read(...) { ... },

						write(...) { ... }

					}

				}

			}

			*/

		}	

	}

}).call(this);

```



# Functions [top](#top)



Exposing server functions to the JavaScript client in the browser is simple. Just define the functions in 

the file `functions.js`. Any asynchronous functions will be awaited.



```javascript

(function() {

	module.exports = {

		client: { // added only to the client and invoked only there

		

		},

		worker: {

		

		}

		cloud: { // added to the client, but invoked on the server

			securedTestFunction() {

				return "If you see this, there may be a security leak";

			},

			getDate() {

				return new Date();

			}

		}

	}

}).call(this);

```



The functions will automatically become available in the admin client `docs/thunderclap`. Function execution in the cloud can

be access controlled in `secure.js`.





# Indexing [top](#top)



All properties of objects inserted using `putItem` are indexed for direct match, with the exception of properties 

containing strings over 64 characters in length. Strings longer than 64 characters can be matched use `$search`.

Objects that are just a value to `setItem` are not indexed. The index partitioned per class, but searches can be conducted

across all classes by the use of the wildcard key `_`.



The root index node can be accessed via `keys("!")`. Direct access is restricted to users with the role `dbo`.



Indexes in Thunderclap consume very little RAM, they are primarily composed of specially formed and partitioned keys 

pointing to just the value `1`. This means the performance of Thunderclap is heavily dependent on the performance of 

the Cloudflare KV with respect to iterating keys. It also means that Thunderclap can have an unlimited number of objects 

indexed. The largest object that can be stored is 2MB (the same as Cloudflare KV).





## Full Text Indexing [top](#top)



Any strings containg spaces are automatically added to a full-text index based on [trigrams](https://en.wikipedia.org/wiki/Trigram_search)

after stop words such as `and`, `but`, `or` have been removed. These can be searched using the [`{$search: }`](#search) pattern.





# Schema [top](#top)



The use of schema is optional with Thunderclap. They can be used to validate data in all tiers of an application: browser,

worker, or cloud. The built in classes `User`, `Position`, and `Coordinates` all have schema. If schema are present, they

are automatically used to validate data prior to insert in the cloud. They can be optionally applied in the browser. 



Below is an example for `User`. Note, by convention Schema are attached to classes as a static property.



```javascript

User.schema = {

	userName: {required:true, type: "string", unique:true},

	roles: {type: "object"}

}

```



The following constraints are supported:



`matches:RegExp` - Checks to see if a property value matches the provided regular expression.



`noindex:boolean` - If present and `truthy`, prevents indexing of a property.



`oneof:Array` - Checks to see if a property value is in the provided array.



`required:boolean` - Ensures the property has a value.



`unique:boolean` - Does a database look-up to ensure no other entity of the same class has the same property value. (Not yet implemented).



`validate:function` - Calls a custom validation function with the signature `(object object,string key,any value,Array errors,Thunderclap db)`. 

The function is responsible for pushing any errors into the provided errors array.





# Development [top](#top)



If you wish to modify Thunderclap, you must subscribe to the Cloudflare Argo tunneling service on the domain where 

you wish to use Thunderclap.



Create a Cloudflare Workers KV namespace using the Cloudflare Workers control panel. By convention the following 

name form is recommended so that the name parrallels the name of the worker script generated by the Thunderclap.



`-thunderclap--`



e.g. `myname-thunderclap-mydomain-com` is the KV namespace and script name associated with `myname-thunderclap.mydomain.com`



You do not need a CNAME record for your dev host, Argo manages this for you.



Run the thunderclap script:



`npm run thunderclap`



If the 'mode' in 'thunderclap.json` is set to `development`, then in addition to deploying the worker script to

`-thunderclap--` with a route, a local web server is started with an Argo tunnel 

to access `-thunderclap.` via your web browser.



If you access `https://-thunderclap./test/` via your web browser, the unit test file 

will load. 



When in dev mode, files are watched by webpack and any changes cause a re-bundling and deployment of the worker script

to Cloudflare.





## Admin UI [top](#top)



When in development mode, there is a primitive UI for making one-off requests at 

`https://-thunderclap./thunderclap.html`. This UI exposes all of the functions

available via the [Javascript](#javascript) client.



# History and Roadmap [top](#top)



Many of the concepts in Thunderclap were first explored in ReasonDB. ReasonDB development has been suspended for now, 

but many features found in ReasonDB will make their way into Thunderclap if interest is shown in the software. 



# Change Log (reverse chronological order) [top](#top)



2019-07-24 v0.0.33a Added graph `add` and `remove` for set operations on values.



2019-07-24 v0.0.32a Minor documentation fixes.



2019-07-23 v0.0.31a Slight performance inmprovements. Fixed broken `$search`.



2019-07-22 v0.0.30a Security now works on graph paths.



2019-07-22 v0.0.29a Started adding graph database capability. Not yet tied to triggers, security, etc. Reworked triggers, 

security, so that they will work across all of key-value, JSON, and graph storage. If you are using any, they will 

need substantive re-work.



2019-07-16 v0.0.28a Multiple classes can now be queried at the same time.



2019-07-15 v0.0.27a Added many user management functions.



2019-07-14 v0.0.26a Modified indexing and query approach to use classes at top level, i.e. `{:}`

instead of ``. NAMESPACES must be recreated. The ability to query across classes will be re-introduced in 

a subsequent release using {_:}. This change will improve performance is real-world cases by further

partitioning keys and also making unique key look-up/verification much faster.



2019-07-13 v0.0.25a Added `unique(cnameOrIdOrObject,property,value`).



2019-07-13 v0.0.24a Fixed `$instanceof` and added `$isa`. Eliminated gloabal leak in unit test for Schema validation.



2019-07-12 v0.0.23a Full text search repaired. Optimized inserts and deletes. NAMESPACES must be recreated.



2019-07-12 v0.0.22a Ehanced documentation. Completely re-worked indexing to allow for more object storage. Full text search

currently broken. NAMESPACES must be re-created.



2019-07-11 v0.0.21a Ehanced documentation.



2019-07-10 v0.0.20a Ehanced documentation. Added `Position` and `Coordinates`.



2019-07-09 v0.0.19a Server was throwing errors on date predicates. Fixed. Added support for un-indexing nested objects.

Unindexing full-text not yet implemented. Added a short term cache to improve performance. Unit tests for removeItem are

failing as a result. Not sure why.



2019-07-06 v0.0.18a Added nested object indexing (unindex does not yet work).



2019-07-04 v0.0.17a Added full text indexing with `{$search: string terms}` or `{$search: [string terms, number pctMatch]}`



2019-07-03 v0.0.16a Added `changePassword(userName,password,oldPassword)`.



2019-07-02 v0.0.15a Added `clear(prefix)`, `entries(prefix,options)`, `hasKey(key)`, `values(prefix,options)`.

All are limited to dbo access. Reverted to two level index for now to address performance. Limits number of 

entries per index due to 128MB limit of Cloudflare Workers.



2019-06-30 v0.0.14a Ehanced triggers and functions to allow browser, service worker, or cloud execution. 

Added `when` capability. Service worker support will operate once service workers are generated during the

build process.



2019-06-26 v0.0.13a Indexing optimized to reeuce RAM usage. Substantive performance drop.



2019-06-26 v0.0.12a Indexing now extends to 3 levels to provide more data spread. Sub-objects still not 

indexed as direct paths. Added support for expiring keys and listing keys.



2019-06-25 v0.0.11a Code optimizations and bug fixes.



2019-06-24 v0.0.10a Custom function support added.



2019-06-22 v0.0.9a Triggers on put, update, remove.



2019-06-22 v0.0.8a Triggers now working for `putItem`.



2019-06-22 v0.0.7a Added JOQULAR pattern `$near:[target,range]`. Range can be a number, in which case it is 

added/substracted or a string ending in the `%` sign, in which case the percentage is add/substracted. Added 

stress tests up to 1000 items. Started support for RegExp as acl keys. Enhanced doucmentation.



2019-06-21 v0.0.6a Documentation improvements.



2019-06-21 v0.0.5a ACL improvements. More of unit tests.



2019-06-20 v0.0.4a Added a large number of unit tests