Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mbok/elasticsearch-linear-regression

A machine learning plugin for Elasticsearch providing aggregations to compute multiple linear regression on search results in real-time for predictive analytics.
https://github.com/mbok/elasticsearch-linear-regression

elasticsearch elasticsearch-plugin linear-regression machine-learning predictive-analytics

Last synced: 15 days ago
JSON representation

A machine learning plugin for Elasticsearch providing aggregations to compute multiple linear regression on search results in real-time for predictive analytics.

Host: GitHub
URL: https://github.com/mbok/elasticsearch-linear-regression
Owner: mbok
License: apache-2.0
Created: 2017-03-05T21:21:08.000Z (over 7 years ago)
Default Branch: master
Last Pushed: 2018-10-07T20:31:19.000Z (about 6 years ago)
Last Synced: 2024-08-01T15:07:56.957Z (3 months ago)
Topics: elasticsearch, elasticsearch-plugin, linear-regression, machine-learning, predictive-analytics
Language: Java
Homepage:
Size: 236 KB
Stars: 64
Watchers: 9
Forks: 21
Open Issues: 5
Metadata Files:
- Readme: README.adoc
- License: LICENSE

Awesome Lists containing this project

awesome - elasticsearch-linear-regression - A machine learning plugin for Elasticsearch providing aggregations to compute multiple linear regression on search results in real-time for predictive analytics. (Java)

README

# A multiple linear regression plugin for Elasticsearch

image:https://travis-ci.org/mbok/elasticsearch-linear-regression.svg?branch=master["Build Status", link="https://travis-ci.org/mbok/elasticsearch-linear-regression"]

Linear regression model has been a mainstay of statistics and machine learning
in the past decades and remains one of the most important tools in context of supervised learning algorithms.
It's a powerful technique for prediction of the value of a dependent variable `y` (called response variable) given the values of another independent
variables `x = (x~1~, x~2~,...,x~C~)` (called explanatory variables) based on a training data set. Prediction of the response variable with respect to the input values
for the explanatory variables is described by the linear hypothesis function ``h(x)`` with

image:http://latex.codecogs.com/gif.latex?h(x)%20=%20\theta_{0}%20+%20\sum_{j=1}^C%20\theta_{j}%20x_{j}[]

This plugin enhances Elasticsearch's query engine by two new aggregations, which utilize the index data during search
as training data for estimating a linear regression model in order to expose information like prediction of a value for the target variable,
anomaly detection and measuring the accuracy or rather predictiveness of the model.
Estimation is performed regarding the https://en.wikipedia.org/wiki/Ordinary_least_squares[OLS]
(ordinary least-squares) approach over the search result set.

## Aggregations
Both aggregations are numeric aggregations that estimate the linear regression coefficients
image:http://latex.codecogs.com/gif.latex?\theta_0,%20\theta_1,%20\theta_2,.%20.%20.,%20\theta_C%20[]
based on document results of a search query. Each search result
document is handled as an observation and the numerical fields as variables (explanatory and response)
for the linear model.

=== Aggregation for prediction

The `linreg_predict` aggregation computes the predicted outcome for the response variable
regarding the estimated model with respect to a set of given input values for the explanatory variables.

[horizontal]
`value`:: The predicted value for the response variable computed using the estimated linear hypothesis
function ``h(x)`` with `x` given by `C` input values for the explanatory variables
`x = [x~1~, x~2~,...,x~C~]`.
`coefficients`:: Estimated coefficients
image:http://latex.codecogs.com/gif.latex?\theta_0,%20\theta_1,%20\theta_2,%20\theta_3,.%20.%20.,%20\theta_C%20[]
of the linear linear hypothesis function ``h(x)``.

Assuming the data consists of documents representing sold house prices with features
like number of bedrooms, bathrooms and size etc. we can let predict or validate
the price for our house in Morro Bay with 2000 square feet, 4 bedrooms and 2 bathrooms by:

[source,js]
--------------------------------------------------
/houses/_search?size=0
{
"query": {
"match" : {
"location" : "Morro Bay"
}
},
"aggs": {
"house_prices": {
"linreg_predict": {
"fields": ["size", "bedrooms", "bathrooms", "price"],
"inputs": [2000, 4, 2]
}
}
}
}
--------------------------------------------------

<1> `fields` instructs this aggregation to use for the linear regression model the house feature fields `size`, `bedrooms` and `bathrooms`
as explanatory variables and the `price` field as the response variable. The size of the `fields` array is `C + 1`
with `C` entries for the explanatory variables and one entry for the response variable.
<2> `inputs` passes the feature values of our house we like to predict the price for. The numeric input values
have to be passed in array form in the order corresponding to the features listed in the `fields` attribute.
The size of the `inputs` array is `C` equivalent to the number of the explanatory variables.

And the following may be the response with the estimated price of around $ 581,458 for our house:

[source,js]
--------------------------------------------------
{
...
"aggregations": {
"my_house_price": {
"value": 581458.3087492324,
"coefficients": [
227990.63952712028,
248.92285661317254,
-68297.7720278421,
64406.52205356777
]
}
}
}
--------------------------------------------------

=== Aggregation for linear regression statistics

The `linreg_stats` aggregation computes statistics for the estimated linear regression model.

[horizontal]
`rss`:: Residual sum of squares as a measure of the discrepancy between the data and the estimated model.
The lower the `rss` number, the smaller the error of the prediction, and the better the model.
`mse`:: Mean squared error or rather `rss` divided by the number of documents consumed for model estimation.
`r2`:: Coefficient of determination, denoted R², as a statistical measure of how well the regression model
approximates the real data points. R² ranges from 0 to 1, where 1 indicates that the estimated hypothesis function perfectly fits the data.
(Available since 5.5.1.2)
`coefficients`:: Estimated coefficients
image:http://latex.codecogs.com/gif.latex?\theta_0,%20\theta_1,%20\theta_2,%20\theta_3,.%20.%20.,%20\theta_C%20[]
of the linear linear hypothesis function ``h(x)``.

Assuming the data consists of documents representing house prices we can compute statistics for
the estimated best fitting linear hypothesis function which predicts house prices based on number of
bedrooms, bathrooms and size with
[source,js]
--------------------------------------------------
/houses/_search?size=0
{
"aggs": {
"house_prices": {
"linreg_stats": {
"fields": ["bedrooms", "bathrooms", "size", "price"]
}
}
}
}
--------------------------------------------------

The aggregation type is `linreg_stats` and the `fields` setting defines the set of fields (as an array)
to be used for building the linear model. The first one to many fields stand for the explanatory variables
and the last for the response variable. The above request returns the following response:

[source,js]
--------------------------------------------------
{
...
"aggregations": {
"house_prices": {
"rss": 49523788338938.75,
"mse": 63410740510.80505,
"r2": 0.4788369924642064,
"coefficients": [
47553.1873756476,
-100544.07258945837,
45981.15827544975,
309.6013051477474
]
}
}
}
--------------------------------------------------

=== Data conditions
Due to algorithmic constraints both aggregations result an empty response, if

* the search result size is less or equal than the number of indicated explanatory variables,
* values of the explanatory variables in the search result set is linearly dependent (that means
that a column can be written as a linear combination of the other columns).

## Algorithm
This implementation is based on a new parallel, single-pass OLS estimation algorithm for multiple linear regression
(not yet published). By aggregating
over the data only once and in parallel the algorithm is ideally suited for large-scale, distributed data sets and
in this respect surpasses the majority of existing multi-pass analytical OLS estimators or iterative optimization algorithms.

The overall complexity of the implemented algorithm to estimate the regression coefficients is `O(N C² + C³)`, where
`N` denotes the size of the training data set (the number of documents in the search result set) and `C` the number
of the indicated explanatory variables (fields).

## Installation

### Elasticsearch 5.x
For installing this plugin please choose first the proper version under the compatible
matrix which matches your Elasticsearch version and use the download link for the following command.

[source]
----
./bin/elasticsearch-plugin install https://github.com/scaleborn/elasticsearch-linear-regression/releases/download/5.5.2.1/elasticsearch-linear-regression-5.5.2.1.zip
----
The plugin will be installed under the name "linear-regression".
Do not forget to restart the node after installing.

.Compatibility matrix
[frame="all"]
|===
| Plugin version | Elasticsearch version | Release date
| https://github.com/scaleborn/elasticsearch-linear-regression/releases/download/5.5.2.1/elasticsearch-linear-regression-5.5.2.1.zip[5.5.2.1] | 5.5.2 | Aug 29, 2017
| https://github.com/scaleborn/elasticsearch-linear-regression/releases/download/5.5.1.2/elasticsearch-linear-regression-5.5.1.2.zip[5.5.1.2] | 5.5.1 | Aug 29, 2017
| https://github.com/scaleborn/elasticsearch-linear-regression/releases/download/5.5.1.1/elasticsearch-linear-regression-5.5.1.1.zip[5.5.1.1] | 5.5.1 | Jul 27, 2017
| https://github.com/scaleborn/elasticsearch-linear-regression/releases/download/5.5.0.1/elasticsearch-linear-regression-5.5.0.1.zip[5.5.0.1] | 5.5.0 | Jul 18, 2017
| https://github.com/scaleborn/elasticsearch-linear-regression/releases/download/5.3.0.2/elasticsearch-linear-regression-5.3.0.2.zip[5.3.0.2] | 5.3.0 | Jul 16, 2017
| https://github.com/scaleborn/elasticsearch-linear-regression/releases/download/5.3.0.1/elasticsearch-linear-regression-5.3.0.1.zip[5.3.0.1] | 5.3.0 | Jun 30, 2017
|===

## Examples
### Predicting house prices
The idea is very simple. We have data in our Elasticsearch index representing
sold house prices in our region with some features like square footage of
the house, # of bathrooms, # of bedrooms etc. Now we want to find out which
price we have to pay for a house of our dreams.

In this example we use test data from: http://wiki.csc.calpoly.edu/datasets/attachment/wiki/Houses/RealEstate.csv?format=raw

To import the data into Elasticsearch we use logstash and this pipeline config
https://github.com/scaleborn/elasticsearch-linear-regression/tree/master/examples/houseprices/house-prices-import.conf[house-prices-import.conf]:
....
./bin/logstash -f house-prices-import.conf
....

The indexed documents will have this form:
[source,js]
--------------------------------------------------
{
"_index": "houses",
"_type": "prices",
"_id": "AV0zjVhTomRh2LZNgmfJ",
"_source": {
"bathrooms": 3,
"bedrooms": 4,
"size": 4168,
"mls": "140077",
"price": 1100000,
"location": "Morro Bay",
"price_sq_ft": 263.92,
"status": "Short Sale"
}
}
--------------------------------------------------

We can now query the index for houses in "Morro Bay" and let predict the price
for our dream house with respect to the desired features like 3 bedrooms,
2 bathrooms and at least 2000 square feet:
[source,js]
--------------------------------------------------
/houses/_search?size=0
{
"query": {
"match" : {
"location" : "Morro Bay"
}
},
"aggs": {
"dream_house_price": {
"linreg_predict": {
"fields": ["size", "bedrooms", "bathrooms", "price"],
"inputs": [2000, 3, 2]
}
}
}
}
--------------------------------------------------

Regarding the following prediction response we have to expect about
$ 650,000 to pay for the desired house in "Morro Bay".
[source,js]
--------------------------------------------------
{
"aggregations": {
"dream_house_price": {
"value": 649918.0709489314,
"coefficients": [
228318.6161854365,
249.02340193904183,
-68314.4830871133,
64248.05007337558
]
}
}
}
--------------------------------------------------

By using sub aggregations we are able to find out the estimated prices per location:
[source,js]
--------------------------------------------------
/houses/_search?size=0
{
"aggs": {
"locations": {
"terms": {
"field": "location.keyword",
"size": 15
},
"aggs": {
"dream_house_price": {
"linreg_predict": {
"fields": ["size", "bedrooms", "bathrooms", "price"],
"inputs": [2000, 3, 2]
}
}
}
}
}
}
--------------------------------------------------

The response uncovers that "Arroyo Grande" would be
the most expensive region for our dream house:

[source,js]
--------------------------------------------------
{
"aggregations": {
"locations": {
"buckets": [
{
"key": "Santa Maria-Orcutt",
"doc_count": 265,
"dream_house_price": {
"value": 256251.9105297585,
"coefficients": [
26437.192829649313,
81.19071633227178,
6825.9128627023265,
23477.773223729317
]
}
},
{
"key": "Paso Robles",
"doc_count": 85,
"dream_house_price": {
"value": 365620.0386191703,
"coefficients": [
42958.257094706176,
151.7000907380368,
6486.477078139843,
-98.91559301451247
]
}
},
...
{
"key": " Arroyo Grande",
"doc_count": 12,
"dream_house_price": {
"value": 1140196.791331573,
"coefficients": [
728566.7474390095,
1956.6474540196602,
-706891.620925945,
-690495.0006844609
]
}
}
...
]
}
}
}
--------------------------------------------------

## License

Licensed under the Apache License 2.0.