Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/ppy/osu-elastic-indexer

MySQL -> Elasticsearch score pump
https://github.com/ppy/osu-elastic-indexer

Last synced: 3 months ago
JSON representation

MySQL -> Elasticsearch score pump

Awesome Lists containing this project

README 

          
# ElasticIndexer



Component for loading [osu!](https://osu.ppy.sh) scores into Elasticsearch.



## Requirements



- .NET 6

- Elasticsearch 7

- Redis 6



## Elasticsearch 8 compatiblity



If using Elasticsearch 8, a minimum version of Elasticsearch 8.2 is required.



The following env needs to be set on the indexer:



```bash

ELASTIC_CLIENT_APIVERSIONING=true

```



and the following must be set in elasticsearch server configuration

`elasticsearch.yml`



```yml

xpack.security.enabled: false

```



or docker environment, e.g. in docker compose:

```yml

environment:

  xpack.security.enabled: false

```



This will enable http connections to elasticsearch and disable the https and authentication requirement, as well as, returning a compatible response to the client.



# Usage



## Schema



A string value is used to indicate the current schema version to be used.



When the queue processor is running, it will store the version it is processing in a set in Redis at `osu-queue:score-index:${prefix}active-schemas`.



If a queue processor is stops automatically due to a schema version change,

it will remove the version it is processing from the set of versions; it _will not_ be removed if the processor if stopped manually or from processor failures; this is to allow other services to continue pushing to those queues.



## Adding items to be indexed



Scores with `preserve`=`true` belonging to a user with `user_warnings`=`0` will be added to the index,

scores where any of the previous conditions are false will be removed from the index.



Push items to `osu-queue:score-index-${schema}`



## Switching to a new schema



Run `dotnet run schema set ${schema}` or set `osu-queue:score-index:${prefix}schema` directly in Redis



### Automatic index switching



If there is an already running indexer watching the queue for the new schema version,

it will automatically update the alias to point to the new index.

When the alias is updated, any index previously used by the alias will be closed.



The alias will not be updated if:

- the schema value does not change

- the indexer processing the queue for that version was not running before the change.



When the schema version changes, all indexers processing the queues for any other version will automatically stop.



# Configuration



Configuration is loaded from environment variables. No environment files are automatically loaded.



To read environment variables from an env file, you can prefix the command to run with `env $(cat {envfile})` replacing `{envfile}` with your env file, e.g.

> Note that this method of passing envvars does not support values with spaces.



    env $(cat .env) dotnet run



Additional envs can be set:



    env $(cat .env) SCHEMA=1 dotnet run



## Environment Variables



### BATCH_SIZE



Maximum number of items to handle/dequeue per batch. This affects the size of the `_bulk` request sent to Elasticsearch.



Defaults to `10000`.



### BUFFER_SIZE



Maximum number of `BATCH_SIZE * BUFFER_SIZE` items allowed inflight during queue processing.



Defaults to `5` (default of `50000` items).



### DB_HOST



Host for MySQL.



Defaults to `localhost`.



### DB_NAME



Database name.



Defaults to `osu`.



### DB_USER



Database username.



Defaults to `root`.



### DB_PASS



Database password.



### DD_AGENT_HOST



Host to submit DataDog/StatsD metrics to.



Defaults to `localhost`.



### DD_ENTITY_ID



Enables DataDog origin detection when running in a container. See [DataDog documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=kubernetes&code-lang=dotnet#origin-detection-over-udp).



### ES_INDEX_PREFIX



Optional prefix for the index names in elasticsearch.



### ES_HOST



Url to the Elasticsearch host.



Defaults to `http://localhost:9200`



### REDIS_HOST



Redis connection string; see [here](https://stackexchange.github.io/StackExchange.Redis/Configuration.html#configuration-options) for configuration options.



Defaults to `localhost`



### SCHEMA



Schema version for the queue; see [Schema](#schema).



# Commands



This documentation assumes `dotnet run` can be used;

in cases where `dotnet run` is not available, the assembly should be used, e.g. `dotnet osu.ElasticIndexer.dll`



## Watching a queue for new scores



Running `queue` will automatically create an index if an open index matching the requested `schema` does not exist.

If a matching open index exists, it will be reused.



    SCHEMA=${schema} dotnet run queue watch



e.g.



    SCHEMA=1 dotnet run queue watch



## Getting the current schema version



    dotnet run schema get



## Setting the schema version



    dotnet run schema set ${schema}



## Unsetting the schema version



This is used to unset the schema version for testing purposes.



    dotnet run schema clear



## Changing the alias to a new index



The index the alias points to can be changed manually:



    dotnet run schema alias ${schema}



will update the index alias to the latest index with schema ${schema} tag.



## List indices



To list all indices and their corresponding states (schema, aliased, open or closed)



    dotnet run index list



## Closing unused indices



This will close all score indices except the active one, unloading them from Elasticsearch's memory pool.



    dotnet run index close



A specific index can be closed by passing in index's name as an argument; e.g. the following will close `index_1`:



    dotnet run index close index_1



## Cleaning up closed indices



This will delete all closed indices and free up the storage space used by those indices.

The command will only delete an index if it is in the `closed` state.



    dotnet run index delete



Passing arguments to the command will delete the matching index:



    dotnet run index delete index_1



## Adding fake items to the queue



For testing purposes, we can add fake items to the queue:



    SCHEMA=1 dotnet run queue pump-fake



It should be noted that these items will not exist or match the ones in the database.



## Queuing a specific score for indexing



    SCHEMA=${schema} dotnet run queue pump-score ${id}



will queue the score with `${id}` for indexing; the score will be added or deleted as necessary, according to the value of `SoloScore.ShouldIndex`.



See [Queuing items for processing from another client](#queuing-items-for-processing-from-another-client)



## Adding existing database records to the queue



    SCHEMA=1 dotnet run queue pump-all



will read existing `solo_scores` in chunks and add them to the queue for indexing. Only scores with a corresponding `phpbb_users` entry will be queued.



Extra options:



`--from {id}`: `solo_scores.id` to start reading from



`--switch`: Sets the schema version after the last item is queued; it does not wait for the item to be indexed; this option is provided as a conveninence for testing.



## Listing known versions currently being processed



    dotnet run active-schemas list



will list the versions known to have queue processors listening on the queue.



## Manually add or remove known versions



For debugging purposes or to perform and manual maintenance or cleanups, the list of versions can be updated manually:



    dotnet run active-schemas add ${schema}

    dotnet run active-schemas remove ${schema}



# (Re)Populating an index



Populating an index is done by pushing score items to a queue.



# Docker



    docker build -t ${tagname} -f osu.ElasticIndexer/Dockerfile osu.ElasticIndexer



    docker run -e SCHEMA=1 -e "ES_HOST=http://host.docker.internal:9200" -e "ES_INDEX_PREFIX=docker." -e "REDIS_HOST=host.docker.internal" -e "DB_CONNECTION_STRING=Server=host.docker.internal;Database=osu;Uid=osuweb;SslMode=None;" ${tagname} ${cmd}



where `${cmd}` is the command to run, e.g. `dotnet osu.ElasticIndexer.dll queue`



# Typical use-cases



## Queuing items for processing from another client



Push items into the Redis queue "`osu-queue:score-index-${schema}`"

e.g.



```csharp

ListLeftPush("osu-queue:score-index-1", "{ \"ScoreId\": 1 }");

```



or from redis-cli:

```

LPUSH "osu-queue:score-index-1" "{\"ScoreId\":1}"

```



### Indexing a score by `id`

```json

{ "ScoreId": 1 }

```



### Queuing a whole score



```json

{

    "Score": {Solo.Score}

}

```