Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pocesar/apify-query-dataset
Use MongoDB query language style to search and generate a new subset of your datasets
https://github.com/pocesar/apify-query-dataset
Last synced: 17 days ago
JSON representation
Use MongoDB query language style to search and generate a new subset of your datasets
- Host: GitHub
- URL: https://github.com/pocesar/apify-query-dataset
- Owner: pocesar
- License: apache-2.0
- Created: 2020-03-22T01:24:35.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2023-03-04T07:29:00.000Z (over 1 year ago)
- Last Synced: 2024-10-18T06:28:33.162Z (27 days ago)
- Language: JavaScript
- Size: 604 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Query Dataset
Query your existing datasets, map and generate a subset of your data
Uses MongoDB-like query style, for extended documentation, check [MongoDB query documentation](http://docs.mongodb.org/manual/reference/operator/query/)
It uses [sift](https://www.npmjs.com/package/sift) module for matching, means you can use as query input.
## Example
Take this dataset, for example:
```json
[
{
"name": "Name 1",
"anotherValue": 1
},
{
"name": "Name 2",
"anotherValue": 2
},
{
"name": "",
"anotherValue": 3
}
]
```
You want to query only items that have a name that isn't empty, so you use the following `INPUT`:
```json
{
"datasetId": "YOUR_DATASET_ID",
"query": {
"name": { "$ne": "" }
}
}
```
`$ne` means "not equal" in MongoDB, so you'll receive "Name 1" and "Name 2" items.
Now say you want to rename the "name" field to something else:
```json
{
"datasetId": "YOUR_DATASET_ID",
"query": {
"name": { "$ne": "" }
},
"filterMap": "({ item }) => { item.name = item.name.replace('Name ', ''); item.extra = true; return item; }"
}
```
Your generated dataset is now:
```json
[
{
"name": "1",
"anotherValue": 1,
"extra": true,
},
{
"name": "2",
"anotherValue": 2,
"extra": true,
}
]
```
## filterMap and customOperationSetup
The `filterMap` parameter exists to do even more complex checks. `filterMap` is run in a limited context, and those are the variables available inside your function:
* `sift`: the [sift](https://www.npmjs.com/package/sift) module, so you can create a filter on-the-fly
* `console.log`: tied to the 'outside' `console.log` and outputs information to the actor log
* `item`: the current dataset item
* `index`: the current filtered index
* `total`: total items available in the dataset
* `filter`: the created filter from `query` parameter
* `datasetIndex`: the current position in the dataset index
The `customOperationSetup` is mostly useful to prepare a [custom operation](https://www.npmjs.com/package/sift#custom-operations) using `sift`:
```js
() => ({
$gtDate(params, ownerQuery, options) {
const timestamp = new Date(params).getTime();
return createEqualsOperation(
value => new Date(value).getTime() > timestamp, // 'value' here is the date from the field you provide
ownerQuery,
options
);
}
})
```
then use directly inside your `query` ("2020-01-01" is passed as param to `params`):
```json
{
"query": {
"lastModified": { "$gtDate": "2020-01-01" }
}
}
```
Most of the time, you won't need to use `customOperationSetup`, since the built-in operators can do a lot by themselves, but they are provided for completeness.
## Expected Comsumption
The memory requirements should be really low, but you need at least 128MB, the dataset items aren't loaded all at once in memory, but depending on the shape of your query, you may need more. The more query parameters you provide, more memory and CPU are required, subsequently your query finishes faster.
## Limitations
Some types aren't allowed in JSON, such as `Date` and `RegExp`. The workaround is to define a query without those types, then inside the `filterMap`, you return either null or undefined for dates or RegExp that don't match.
E.g.:
```json
{
"datasetId": "YOUR_DATASET_ID",
"query": {
},
"filterMap": "({ item }) => { if (new Date(item.someDateField).getTime() < new Date(2019, 10, 20)) { return item } }"
}
```
Or you can use the `customOperationSetup` and provide your advanced operator for native types.
## License
Apache-2.0