Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/nextapps-de/bulksearch
Lightweight and read-write optimized full text search library.
https://github.com/nextapps-de/bulksearch
full-text-search fulltext-search javascript javascript-library node-module nodejs open-source search search-algorithm search-engine search-in-text searching searching-algorithms standalone web
Last synced: 1 day ago
JSON representation
Lightweight and read-write optimized full text search library.
- Host: GitHub
- URL: https://github.com/nextapps-de/bulksearch
- Owner: nextapps-de
- License: apache-2.0
- Created: 2018-02-18T11:49:12.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-06-02T21:55:54.000Z (over 3 years ago)
- Last Synced: 2024-04-26T02:03:11.612Z (7 months ago)
- Topics: full-text-search, fulltext-search, javascript, javascript-library, node-module, nodejs, open-source, search, search-algorithm, search-engine, search-in-text, searching, searching-algorithms, standalone, web
- Language: HTML
- Homepage:
- Size: 261 KB
- Stars: 112
- Watchers: 4
- Forks: 7
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# BulkSearch
### Lightweight and read-write optimized full text search library.
When it comes to the __overall speed__, BulkSearch outperforms every searching library out there and also provides flexible search capabilities like multi-word matching, phonetic transformations or partial matching. It is essentially based on how a HDD manages files in a filesystem. Adding, updating or removing items are as fast as searching for them, but also consumes some additional memory. When your index doesn't need to be updated frequently then FlexSearch may be a better choice. BulkSearch also provides you a asynchronous processing model to perform queries in the background.
Benchmark:
- Comparison: https://jsperf.com/compare-search-libraries
- Detailed: https://jsperf.com/bulksearch
Supported Platforms:
- Browser
- Node.js
Supported Module Definitions:
- AMD (RequireJS)
- CommonJS (Node.js)
- Closure (Xone)
- Global (Browser)
All Features:
- Partial Words
- Multiple Words
- Flexible Word Order
- Phonetic Search
- Limit Results
- Pagination
- Caching
- Asynchronous Mode
- Custom Matchers
- Custom Encoders
## Installation
##### HTML / Javascript
```html
...
```
__Note:__ Use _bulksearch.min.js_ for production and _bulksearch.js_ for development.
Use latest from CDN:
```html
```
##### Node.js
```npm
npm install bulksearch
```
In your code include as follows:
```javascript
var BulkSearch = require("bulksearch");
```
Or pass in options when requiring:
```javascript
var index = require("bulksearch").create({/* options */});
```
__AMD__
```javascript
var BulkSearch = require("./bulksearch.js");
```
#### Compare BulkSearch vs. FlexSearch
Description
BulkSearch
FlexSearch
Access
Read-Write optimized index
Read-Memory optimized index
Memory
Large (~ 90 bytes per word)
Tiny (~ 2 bytes per word)
Usage
- Limited content
- Index updates continously
- Fastest possible search
- Rare updates on index
- Low memory capabilities
Limit Results
Yes
Yes
Pagination
Yes
No
## API Overview
Global methods:
- BulkSearch.__create__(\)
- BulkSearch.__addMatcher__({_KEY: VALUE_})
- BulkSearch.__register__(name, encoder)
- BulkSearch.__encode__(name, string)
Index methods:
- Index.__add__(id, string)
- Index.__search__(string, \, \)
- Index.__search__(string, \, \)
- Index.__search__(options, \)
- Index.__update__(id, string)
- Index.__remove__(id)
- Index.__reset__()
- Index.__destroy__()
- Index.__init__(\)
- Index.__optimize__()
- Index.__info__()
- Index.__addMatcher__({_KEY: VALUE_})
- Index.__encode__(string)
> BulkSearch.__create(\)__
```js
var index = new BulkSearch();
```
alternatively you can also use:
```js
var index = BulkSearch.create();
```
##### Create index with custom options
```js
var index = new BulkSearch({
// default values:
type: "integer",
encode: "icase",
boolean: "and",
size: 4000,
multi: false,
strict: false,
ordered: false,
paging: false,
async: false,
cache: false
});
```
__Read more:__ Phonetic Search, Phonetic Comparison, Improve Memory Usage
#### Add items to an index
> Index.__add(id, string)__
```js
index.add(10025, "John Doe");
```
#### Search items
> Index.__search(string|options, \, \)__
```js
index.search("John");
```
Limit the result:
```js
index.search("John", 10);
```
Perform queries asynchronously:
```js
index.search("John", function(result){
// array of results
});
```
Pass parameter as an object:
```js
index.search({
query: "John",
page: '1:1234',
limit: 10,
callback: function(result){
// async
}
});
```
#### Update item from an index
> Index.__update(id, string)__
```js
index.update(10025, "Road Runner");
```
#### Remove item from an index
> Index.__remove(id)__
```js
index.remove(10025);
```
#### Reset index
```js
index.reset();
```
#### Destroy index
```js
index.destroy();
```
#### Re-Initialize index
> Index.__init(\)__
__Note:__ Re-initialization will also destroy the old index!
Initialize (with same options):
```js
index.init();
```
Initialize with new options:
```js
index.init({
/* options */
});
```
#### Add custom matcher
> BulkSearch.__addMatcher({_REGEX: REPLACE_})__
Add global matchers for all instances:
```js
BulkSearch.addMatcher({
'ä': 'a', // replaces all 'ä' to 'a'
'ó': 'o',
'[ûúù]': 'u' // replaces multiple
});
```
Add private matchers for a specific instance:
```js
index.addMatcher({
'ä': 'a', // replaces all 'ä' to 'a'
'ó': 'o',
'[ûúù]': 'u' // replaces multiple
});
```
#### Add custom encoder
Define a private custom encoder during creation/initialization:
```js
var index = new BulkSearch({
encode: function(str){
// do something with str ...
return str;
}
});
```
##### Register a global encoder to be used by all instances
> BulkSearch.__register(name, encoder)__
```js
BulkSearch.register('whitespace', function(str){
return str.replace(/ /g, '');
});
```
Use global encoders:
```js
var index = new BulkSearch({ encode: 'whitespace' });
```
##### Call encoders directly
Private encoder:
```js
var encoded = index.encode("sample text");
```
Global encoder:
```js
var encoded = BulkSearch.encode("whitespace", "sample text");
```
##### Mixup/Extend multiple encoders
```js
BulkSearch.register('mixed', function(str){
str = this.encode("icase", str); // built-in
str = this.encode("whitespace", str); // custom
return str;
});
```
```js
BulkSearch.register('extended', function(str){
str = this.encode("custom", str);
// do something additional with str ...
return str;
});
```
```js
index.info();
```
Returns information about the index, e.g.:
```json
{
"bytes": 103600,
"chunks": 9,
"fragmentation": 0,
"fragments": 0,
"id": 0,
"length": 7798,
"matchers": 0,
"size": 10000,
"status": false
}
```
__Note:__ When the fragmentation value is about 50% or higher, your should consider using _cleanup()_.
#### Optimize / Cleanup index
Optimize an index will free all fragmented memory and also rebuilds the index by scoring.
```js
index.optimize();
```
### Pagination
__Note:__ Pagination can simply reduce query time by a factor of 100.
Enable pagination on initialization:
```js
var index = BulkSearch.create({ paging: true });
```
Perform query and pass a limit (items per page):
```js
index.search("John", 10);
```
The response will include a pagination object like this:
```json
{
"current": "0:0",
"prev": null,
"next": "1:16322",
"results": []
}
```
Explanation:
"current"
Includes the pointer to the current page.
"prev"
Includes the pointer to the previous page. Whenever this field has the value null there are no more previous pages available.
"next"
Includes the pointer to the next page. Whenever this field has the value null there are no more pages left.
"results"
Array of matched items.
Perform query and pass a pointer to a specific page:
```js
index.search("John", {
page: "1:16322", // pointer
limit: 10
});
```
Option
Values
Description
type
"byte"
"short"
"integer"
"float"
"string"
The data type of passed IDs has to be specified on creation. It is recommended to uses to most lowest possible data range here, e.g. use "short" when IDs are not higher than 65,535.
encode
false
"icase"
"simple"
"advanced"
"extra"
function(string):string
The encoding type. Choose one of the built-ins or pass a custom encoding function.
boolean
"and"
"or"
The applied boolean model when comparing multiple words. Note: When using "or" the first word is also compared with "and". Example: a query with 3 words, results has either: matched word 1 & 2 and matched word 1 & 3.
size
2500 - 10000
The size of chunks. It depends on content length which value fits best. Short content length (e.g. User names) are faster with a chunk size of 2,500. Bigger text runs faster with a chunk size of 10,000. Note: It is recommended to use a minimum chunk size of the maximum content length which has to be indexed to prevent fragmentation.
multi
true
false
Enable multi word processing.
ordered
true
false
Multiple words has to be the same order as the matched entry.
strict
true
false
Matches exactly needs to be started with the query.
cache
true
false
Enable caching.
Encoder
Description
False Positives
Compression Level
false
Turn off encoding
no
no
"icase"
Case in-sensitive encoding
no
no
"simple"
Phonetic normalizations
no
~ 3%
"advanced"
Phonetic normalizations + Literal transformations
no
~ 25%
"extra"
Phonetic normalizations + Soundex transformations
yes
~ 50%
Reference String: __"Björn-Phillipp Mayer"__
Query
ElasticSearch
BulkSearch (iCase)
BulkSearch (Simple)
BulkSearch (Adv.)
BulkSearch (Extra)
björn
yes
yes
yes
yes
yes
björ
no
yes
yes
yes
yes
bjorn
no
no
yes
yes
yes
bjoern
no
no
no
yes
yes
philipp
no
no
no
yes
yes
filip
no
no
no
yes
yes
björnphillip
no
no
yes
yes
yes
meier
no
no
no
yes
yes
björn meier
no
no
no
yes
yes
meier fhilip
no
no
no
yes
yes
byorn mair
no
no
no
no
yes
(false positives)
yes
no
no
no
yes
__Note:__ The data type of passed IDs has to be specified on creation. It is recommended to uses the most lowest possible data range here, e.g. use "short" when IDs are not higher than 65,535.
ID Type
Range of Values
Memory usage of every ~ 100,000 indexed words
Byte
0 - 255
4.5 Mb
Short
0 - 65,535
5.3 Mb
Integer
0 - 4,294,967,295
6.8 Mb
Float
0 - * (16 digits)
10 Mb
String
* (unlimited)
28.2 Mb
---
Author BulkSearch: Thomas Wilkerling
License: Apache 2.0 License