Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/evinism/mistql
A query / expression language for performing computations on JSON-like structures. Tuned for clientside ML feature extraction.
https://github.com/evinism/mistql
expression-language feature-engineering feature-extraction hacktoberfest javascript json machine-learning mistql python query typescript
Last synced: about 2 months ago
JSON representation
A query / expression language for performing computations on JSON-like structures. Tuned for clientside ML feature extraction.
- Host: GitHub
- URL: https://github.com/evinism/mistql
- Owner: evinism
- License: mit
- Created: 2021-08-04T21:14:11.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-01-21T04:41:28.000Z (over 1 year ago)
- Last Synced: 2024-07-16T18:12:29.911Z (2 months ago)
- Topics: expression-language, feature-engineering, feature-extraction, hacktoberfest, javascript, json, machine-learning, mistql, python, query, typescript
- Language: TypeScript
- Homepage: https://mistql.com
- Size: 3.19 MB
- Stars: 349
- Watchers: 8
- Forks: 16
- Open Issues: 21
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/evinism/mistql/blob/main/LICENSE)
[](https://www.npmjs.com/package/mistql)
[](https://pypi.org/project/mistql/)
MistQL is a query language for JSON-like structures, built for embedding across multiple domains. It supports
logic for querying and manipulating JSON-like data in a simple, readable manner. MistQL serves as a powerful
common expression language across multiple implementations.
For more detailed usage information, please visit MistQL's docs site.
[Join the Discord!!](https://discord.gg/YupxqvE5Jk)
### Links
- [MistQL's doc site](https://www.mistql.com/)
- [Getting Started](https://www.mistql.com/docs/intro)
- [Try it out!](https://www.mistql.com/tryitout)
- [Functions](https://www.mistql.com/docs/reference/functions)
- [Reference](https://www.mistql.com/docs/reference/overview)
# At A Glance
MistQL is an embedded query language.
```js
import mistql from 'mistql';
const animals = [
{name: 'charlie', species: 'dog'},
{name: 'mulberry', species: 'cat'},
]
const cats = mistql.query('@ | filter species == "dog" | count')
```
The primary power of MistQL comes from its strong cross-platform behavior semantics. For example, the following have the same behavior:
```py
# Python
import mistql
import json
query = 'events | groupby type | keys'
print(mistql.query(query, json.loads(data)))
```
```js
// JavaScript
import mistql from 'mistql'
const query = 'events | groupby type | keys';
console.log(mistql.query(query, JSON.parse(data)))
```
# Developing MistQL
Contributions to MistQL are very welcome!
As MistQL is a small project, there are no formatting requirements for either issues or pull requests.
If you're planning on making a new implementation, [ping the discord](https://discord.gg/YupxqvE5Jk) and we'll coordinate!
For an example PR that adds a function to MistQL, refer to [this PR](https://github.com/evinism/mistql/pull/175) as an example of a new function with moderate complexity.
### Code workflow
Code contributions to MistQL should roughly follow standard open source workflows:
1. Fork the project
2. Make code changes on your fork of the project.
3. (if necessary) Pull upstream to bring in new changes
4. Submit a pull request to MistQL's `main` branch.
5. (if necessary) Implement changes requested by maintainers.
6. Wait for the branch to be accepted and merged by maintainers!
### MistQL standard
No MistQL standard yet exists, but we're aiming for the `0.5.0` release of `mistql` as a standardizable language. After the `0.5.0` release, we will create a language specification, separate from any implementation.
In the meantime, we're actually pretty close.
We have a [Lark grammar](https://github.com/evinism/mistql/blob/main/py/mistql/grammar.lark)
which defines the language's syntax. This is likely the final grammar that will be formalized
into ABNF, although it is possible that we may need to fix minor issues before `0.5.0`. I
expect this to barely change, if at all.
Additionally, our language-independent test suite is rather extensive and forms the de-facto
standard of behaviors, as shared by both Python and JavaScript. While not strictly formalized,
the tests and the docs together form a cohesive body of behaviors, that, except for a few
minuitae, is of sufficient detail to be standardized.
## Directory Structure
MistQL's directory structure is a monorepo, currently consisting of these main directories:
1. `/docs`: Documentation Site (hosted at [mistql.com](https://www.mistql.com/))
2. `/js`: MistQL's browser implementation (e.g. `mistql` on npm).
3. `/py`: MistQL's python implementation (e.g. `mistql` on pypi).
4. `/shared`: Shared assets between all implementation. Contains the language-independent test suite.
## Developing for the docs site
Docs are built via a fairly standard [Docusaurus 2](https://docusaurus.io/) implementation. Please follow Docusaurus's docs for developing for the Docs site.
## Developing for `mistql` on npm
`mistql` is written exclusively using typescript. Additionally, `mistql` uses `yarn` for dependency management, versioning, and uploading. JS-specific tests are stored alongside their implementation, using the suffix `.spec.ts`. Tests that describe the language itself are written in a language agnostic JSON format in the `/shared` directory. Writing tests for all feature additions and bug fixes is strongly encouraged.
For all major improvements, it is strongly encouraged to run `yarn bundlesize` to estimate gzipped impact of MistQL on a browser. MistQL for the browser should, in general, remain relatively close to 5kb.
The directory structure is relatively flat, except for the single `src/builtins` folder, which contains the implementation of all of MistQL's internal functions.
## Developing for `mistql` on pypi
`mistql` is a fairly standard python package managed with [poetry](https://python-poetry.org/).
Tests can be run using pytest, e.g. `poetry run pytest` from within the `/py` directory.