{"id":19777163,"url":"https://github.com/ppy/osu-elastic-indexer","last_synced_at":"2026-03-05T01:03:19.673Z","repository":{"id":37752306,"uuid":"136581569","full_name":"ppy/osu-elastic-indexer","owner":"ppy","description":"MySQL -\u003e Elasticsearch score pump","archived":false,"fork":false,"pushed_at":"2025-04-16T10:21:39.000Z","size":588,"stargazers_count":11,"open_issues_count":3,"forks_count":6,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-30T19:41:50.449Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ppy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-06-08T07:16:15.000Z","updated_at":"2025-04-16T10:21:44.000Z","dependencies_parsed_at":"2024-02-28T12:28:33.257Z","dependency_job_id":"48b481a5-7cf7-4fd6-83b2-e8b94456fb95","html_url":"https://github.com/ppy/osu-elastic-indexer","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"purl":"pkg:github/ppy/osu-elastic-indexer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppy%2Fosu-elastic-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppy%2Fosu-elastic-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppy%2Fosu-elastic-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppy%2Fosu-elastic-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ppy","download_url":"https://codeload.github.com/ppy/osu-elastic-indexer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppy%2Fosu-elastic-indexer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30104218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T00:38:46.881Z","status":"ssl_error","status_checked_at":"2026-03-05T00:38:45.829Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T05:23:26.786Z","updated_at":"2026-03-05T01:03:14.640Z","avatar_url":"https://github.com/ppy.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ElasticIndexer\n\nComponent for loading [osu!](https://osu.ppy.sh) scores into Elasticsearch.\n\n## Requirements\n\n- .NET 6\n- Elasticsearch 7\n- Redis 6\n\n## Elasticsearch 8 compatiblity\n\nIf using Elasticsearch 8, a minimum version of Elasticsearch 8.2 is required.\n\nThe following env needs to be set on the indexer:\n\n```bash\nELASTIC_CLIENT_APIVERSIONING=true\n```\n\nand the following must be set in elasticsearch server configuration\n`elasticsearch.yml`\n\n```yml\nxpack.security.enabled: false\n```\n\nor docker environment, e.g. in docker compose:\n```yml\nenvironment:\n  xpack.security.enabled: false\n```\n\nThis will enable http connections to elasticsearch and disable the https and authentication requirement, as well as, returning a compatible response to the client.\n\n\n# Usage\n\n## Schema\n\nA string value is used to indicate the current schema version to be used.\n\nWhen the queue processor is running, it will store the version it is processing in a set in Redis at `osu-queue:score-index:${prefix}active-schemas`.\n\nIf a queue processor is stops automatically due to a schema version change,\nit will remove the version it is processing from the set of versions; it _will not_ be removed if the processor if stopped manually or from processor failures; this is to allow other services to continue pushing to those queues.\n\n## Adding items to be indexed\n\nScores with `preserve`=`true` belonging to a user with `user_warnings`=`0` will be added to the index,\nscores where any of the previous conditions are false will be removed from the index.\n\nPush items to `osu-queue:score-index-${schema}`\n\n## Switching to a new schema\n\nRun `dotnet run schema set ${schema}` or set `osu-queue:score-index:${prefix}schema` directly in Redis\n\n### Automatic index switching\n\nIf there is an already running indexer watching the queue for the new schema version,\nit will automatically update the alias to point to the new index.\nWhen the alias is updated, any index previously used by the alias will be closed.\n\nThe alias will not be updated if:\n- the schema value does not change\n- the indexer processing the queue for that version was not running before the change.\n\nWhen the schema version changes, all indexers processing the queues for any other version will automatically stop.\n\n# Configuration\n\nConfiguration is loaded from environment variables. No environment files are automatically loaded.\n\nTo read environment variables from an env file, you can prefix the command to run with `env $(cat {envfile})` replacing `{envfile}` with your env file, e.g.\n\u003e Note that this method of passing envvars does not support values with spaces.\n\n    env $(cat .env) dotnet run\n\nAdditional envs can be set:\n\n    env $(cat .env) SCHEMA=1 dotnet run\n\n## Environment Variables\n\n### BATCH_SIZE\n\nMaximum number of items to handle/dequeue per batch. This affects the size of the `_bulk` request sent to Elasticsearch.\n\nDefaults to `10000`.\n\n### BUFFER_SIZE\n\nMaximum number of `BATCH_SIZE * BUFFER_SIZE` items allowed inflight during queue processing.\n\nDefaults to `5` (default of `50000` items).\n\n### DB_HOST\n\nHost for MySQL.\n\nDefaults to `localhost`.\n\n### DB_NAME\n\nDatabase name.\n\nDefaults to `osu`.\n\n### DB_USER\n\nDatabase username.\n\nDefaults to `root`.\n\n### DB_PASS\n\nDatabase password.\n\n### DD_AGENT_HOST\n\nHost to submit DataDog/StatsD metrics to.\n\nDefaults to `localhost`.\n\n### DD_ENTITY_ID\n\nEnables DataDog origin detection when running in a container. See [DataDog documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=kubernetes\u0026code-lang=dotnet#origin-detection-over-udp).\n\n### ES_INDEX_PREFIX\n\nOptional prefix for the index names in elasticsearch.\n\n### ES_HOST\n\nUrl to the Elasticsearch host.\n\nDefaults to `http://localhost:9200`\n\n### REDIS_HOST\n\nRedis connection string; see [here](https://stackexchange.github.io/StackExchange.Redis/Configuration.html#configuration-options) for configuration options.\n\nDefaults to `localhost`\n\n### SCHEMA\n\nSchema version for the queue; see [Schema](#schema).\n\n# Commands\n\nThis documentation assumes `dotnet run` can be used;\nin cases where `dotnet run` is not available, the assembly should be used, e.g. `dotnet osu.ElasticIndexer.dll`\n\n## Watching a queue for new scores\n\nRunning `queue` will automatically create an index if an open index matching the requested `schema` does not exist.\nIf a matching open index exists, it will be reused.\n\n    SCHEMA=${schema} dotnet run queue watch\n\ne.g.\n\n    SCHEMA=1 dotnet run queue watch\n\n## Getting the current schema version\n\n    dotnet run schema get\n\n## Setting the schema version\n\n    dotnet run schema set ${schema}\n\n## Unsetting the schema version\n\nThis is used to unset the schema version for testing purposes.\n\n    dotnet run schema clear\n\n## Changing the alias to a new index\n\nThe index the alias points to can be changed manually:\n\n    dotnet run schema alias ${schema}\n\nwill update the index alias to the latest index with schema ${schema} tag.\n\n## List indices\n\nTo list all indices and their corresponding states (schema, aliased, open or closed)\n\n    dotnet run index list\n\n## Closing unused indices\n\nThis will close all score indices except the active one, unloading them from Elasticsearch's memory pool.\n\n    dotnet run index close\n\nA specific index can be closed by passing in index's name as an argument; e.g. the following will close `index_1`:\n\n    dotnet run index close index_1\n\n## Cleaning up closed indices\n\nThis will delete all closed indices and free up the storage space used by those indices.\nThe command will only delete an index if it is in the `closed` state.\n\n    dotnet run index delete\n\nPassing arguments to the command will delete the matching index:\n\n    dotnet run index delete index_1\n\n## Adding fake items to the queue\n\nFor testing purposes, we can add fake items to the queue:\n\n    SCHEMA=1 dotnet run queue pump-fake\n\nIt should be noted that these items will not exist or match the ones in the database.\n\n## Queuing a specific score for indexing\n\n    SCHEMA=${schema} dotnet run queue pump-score ${id}\n\nwill queue the score with `${id}` for indexing; the score will be added or deleted as necessary, according to the value of `SoloScore.ShouldIndex`.\n\nSee [Queuing items for processing from another client](#queuing-items-for-processing-from-another-client)\n\n## Adding existing database records to the queue\n\n    SCHEMA=1 dotnet run queue pump-all\n\nwill read existing `solo_scores` in chunks and add them to the queue for indexing. Only scores with a corresponding `phpbb_users` entry will be queued.\n\nExtra options:\n\n`--from {id}`: `solo_scores.id` to start reading from\n\n`--switch`: Sets the schema version after the last item is queued; it does not wait for the item to be indexed; this option is provided as a conveninence for testing.\n\n## Listing known versions currently being processed\n\n    dotnet run active-schemas list\n\nwill list the versions known to have queue processors listening on the queue.\n\n\n## Manually add or remove known versions\n\nFor debugging purposes or to perform and manual maintenance or cleanups, the list of versions can be updated manually:\n\n    dotnet run active-schemas add ${schema}\n    dotnet run active-schemas remove ${schema}\n\n# (Re)Populating an index\n\nPopulating an index is done by pushing score items to a queue.\n\n# Docker\n\n    docker build -t ${tagname} -f osu.ElasticIndexer/Dockerfile osu.ElasticIndexer\n\n    docker run -e SCHEMA=1 -e \"ES_HOST=http://host.docker.internal:9200\" -e \"ES_INDEX_PREFIX=docker.\" -e \"REDIS_HOST=host.docker.internal\" -e \"DB_CONNECTION_STRING=Server=host.docker.internal;Database=osu;Uid=osuweb;SslMode=None;\" ${tagname} ${cmd}\n\nwhere `${cmd}` is the command to run, e.g. `dotnet osu.ElasticIndexer.dll queue`\n\n# Typical use-cases\n\n## Queuing items for processing from another client\n\nPush items into the Redis queue \"`osu-queue:score-index-${schema}`\"\ne.g.\n\n```csharp\nListLeftPush(\"osu-queue:score-index-1\", \"{ \\\"ScoreId\\\": 1 }\");\n```\n\nor from redis-cli:\n```\nLPUSH \"osu-queue:score-index-1\" \"{\\\"ScoreId\\\":1}\"\n```\n\n### Indexing a score by `id`\n```json\n{ \"ScoreId\": 1 }\n```\n\n### Queuing a whole score\n\n```json\n{\n    \"Score\": {Solo.Score}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fppy%2Fosu-elastic-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fppy%2Fosu-elastic-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fppy%2Fosu-elastic-indexer/lists"}