{"id":13481601,"url":"https://github.com/ing-bank/scruid","last_synced_at":"2025-04-14T14:31:16.208Z","repository":{"id":48945761,"uuid":"96191705","full_name":"ing-bank/scruid","owner":"ing-bank","description":"Scala + Druid: Scruid. A library that allows you to compose queries in Scala, and parse the result back into typesafe classes.","archived":false,"fork":false,"pushed_at":"2021-07-04T10:22:44.000Z","size":583,"stargazers_count":115,"open_issues_count":8,"forks_count":27,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-03-28T03:24:57.648Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ing-bank.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-04T07:59:24.000Z","updated_at":"2023-08-22T10:31:42.000Z","dependencies_parsed_at":"2022-09-11T19:21:13.335Z","dependency_job_id":null,"html_url":"https://github.com/ing-bank/scruid","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fscruid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fscruid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fscruid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fscruid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ing-bank","download_url":"https://codeload.github.com/ing-bank/scruid/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248897102,"owners_count":21179539,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:00:53.241Z","updated_at":"2025-04-14T14:31:15.802Z","avatar_url":"https://github.com/ing-bank.png","language":"Scala","funding_links":[],"categories":["Table of Contents","Database"],"sub_categories":["Database"],"readme":"[![Build](https://github.com/ing-bank/scruid/workflows/build/badge.svg?branch=master)](https://github.com/ing-bank/scruid/actions)\n[![Codacy Badge](https://api.codacy.com/project/badge/Coverage/9b7c4adf8ad447efa9c7ea8a9ffda6b2)](https://www.codacy.com/app/fokko/scruid?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=ing-bank/scruid\u0026utm_campaign=Badge_Coverage)\n\n![Scruid](logo/logo-with-tagline.svg)\n\nScruid (Scala+Druid) is an open source library that allows you to compose Druid queries easily in Scala. The library will take care of the translation of the query into json, parse the result in the case class that you define.\n\nCurrently, the API is under heavy development, so changes might occur.\n\n## Release Notes\n\nPlease view the [Releases](https://github.com/ing-bank/scruid/releases) page on GitHub.\n\n## Installation\n\nThe binaries are hosted on Maven Central. We publish builds for Scala 2.11, 2.12 and 2.13.\n\n```sbt\nlibraryDependencies += \"com.ing.wbaa.druid\" %% \"scruid\" % \"2.5.0\"\n```\n\n## Example queries:\n\nScruid provides query constructors for `TopNQuery`, `GroupByQuery`, `TimeSeriesQuery`, `ScanQuery` and `SearchQuery` (see below for details). You can call the `execute` method on a query to send the query to Druid. This will return a `Future[DruidResponse]`. This response contains the [Circe](http://circe.io) JSON data without having it parsed to a specific case class yet. To interpret this JSON data you can run two methods on a `DruidResponse`:\n\n- `.list[T](implicit decoder: Decoder[T]): List[T]` : This decodes the JSON to a list with items of type `T`.\n- `.series[T](implicit decoder: Decoder[T]): Map[ZonedDateTime, T]` : This decodes the JSON to a timeseries map with the timestamp as key and `T` as value.\n\nBelow the example queries supported by Scruid. For more information about how to query Druid, and what query to pick, please refer to the [Druid documentation](http://druid.io/docs/latest/querying/querying.html)\n\n### TopN query\n```scala\ncase class TopCountry(count: Int, countryName: String = null)\n\nval response = TopNQuery(\n  dimension = Dimension(\n    dimension = \"countryName\"\n  ),\n  threshold = 5,\n  metric = \"count\",\n  aggregations = List(\n    CountAggregation(name = \"count\")\n  ),\n  intervals = List(\"2011-06-01/2017-06-01\")\n).execute\n\nval result: Future[Map[ZonedDateTime, List[TopCountry]]] = response.map(_.series[List[TopCountry]])\n```\n\n\n### GroupBy query\n\n```scala\ncase class GroupByIsAnonymous(isAnonymous: Boolean, count: Int)\n\nval response = GroupByQuery(\n  aggregations = List(\n    CountAggregation(name = \"count\")\n  ),\n  dimensions = List(\"isAnonymous\"),\n  intervals = List(\"2011-06-01/2017-06-01\")\n).execute()\n\nval result: Future[List[GroupByIsAnonymous]] = response.map(_.list[GroupByIsAnonymous])\n```\n\nThe returned `Future[DruidResponse]` will contain json data where `isAnonymouse` is either `true or false`. Please keep in mind that Druid is only able to handle strings, and recently also numerics. So Druid will be returning a string, and the conversion from a string to a boolean is done by the json parser.\n\n### TimeSeries query\n\n```scala\ncase class TimeseriesCount(count: Int)\n\nval response = TimeSeriesQuery(\n  aggregations = List(\n    CountAggregation(name = \"count\")\n  ),\n  granularity = GranularityType.Hour,\n  intervals = List(\"2011-06-01/2017-06-01\")\n).execute\n\nval series: Future[Map[ZonedDateTime, TimeseriesCount]] = response.map(_.series[TimeseriesCount])\n```\n\n### Scan query\n\n```scala\ncase class ScanResult(channel: Option[String], cityName: Option[String], countryIsoCode: Option[String], user: Option[String])\n\nval response = ScanQuery(\n    granularity = GranularityType.Hour\n    intervals = List(\"2011-06-01/2017-06-01\")\n    dimensions = List(\"channel\", \"cityName\", \"countryIsoCode\", \"user\"),\n    limit = 100\n).execute() \n\nval result: Future[List[ScanResult]] = response.map(_.list[ScanResult])\n```\n\n### Search query\n\nSearch query is a bit different, since it does not take type parameters as its results are of type `com.ing.wbaa.druid.DruidSearchResult`\n\n```scala\nval response = SearchQuery(\n    granularity = GranularityType.Hour,\n    intervals = List(\"2011-06-01/2017-06-01\"),\n    query = ContainsInsensitive(\"GR\"),\n    searchDimensions = List(\"countryIsoCode\")\n).execute()\n\nval result = Future[List[DruidSearchResult]] = response.map(_.list)\n```\n\n## Query context\n\nQueries can be configured using Druid [query context](https://druid.apache.org/docs/latest/querying/query-context.html),\nsuch as `timeout`, `queryId` and `groupByStrategy`. All types of query contain the argument `context` which\nassociates query parameter with their corresponding values. The parameter names can also be accessed\nby `com.ing.wbaa.druid.definitions.QueryContext` object. Consider, for example, a timeseries query with custom `query id`\nand `priority`:\n\n```scala\nTimeSeriesQuery(\n  aggregations = List(\n    CountAggregation(name = \"count\")\n  ),\n  granularity = GranularityType.Hour,\n  intervals = List(\"2011-06-01/2017-06-01\"),\n  context = Map(\n    QueryContext.QueryId -\u003e \"some_custom_id\",\n    QueryContext.Priority -\u003e 1\n  )\n)\n```\n\n## Druid query language (DQL)\n\nScruid also provides a rich Scala API for building queries using the fluent pattern.\n\n```scala\ncase class GroupByIsAnonymous(isAnonymous: String, country: String, count: Int)\n\nval query: GroupByQuery = DQL\n    .granularity(GranularityType.Day)\n    .interval(\"2011-06-01/2017-06-01\")\n    .agg(count as \"count\")\n    .where(d\"countryName\".isNotNull)\n    .groupBy(d\"isAnonymous\", d\"countryName\".extract(UpperExtractionFn()) as \"country\")\n    .having(d\"count\" \u003e 100 and d\"count\" \u003c 200)\n    .limit(10, d\"count\".desc(DimensionOrderType.Numeric))\n    .build()\n\nval response: Future[List[GroupByIsAnonymous]] = query.execute().map(_.list[GroupByIsAnonymous])\n```\n\nFor details and examples see the [DQL documentation](docs/dql.md).\n\n## Print native Druid JSON representation \n\nFor all types of queries you can call the function `toDebugString`, in order to get the corresponding native Druid JSON \nquery representation.\n\nFor example the following:\n\n```scala\nimport com.ing.wbaa.druid.dql.DSL._\n\nval query: TopNQuery = DQL\n    .from(\"wikipedia\")\n    .agg(count as \"count\")\n    .interval(\"2011-06-01/2017-06-01\")\n    .topN(dimension = d\"countryName\", metric = \"count\", threshold = 5)\n    .build()\n\nprintln(query.toDebugString)\n```\n\nwill print to the standard output:\n\n```json\n{\n  \"dimension\" : {\n    \"dimension\" : \"countryName\",\n    \"outputName\" : \"countryName\",\n    \"outputType\" : null,\n    \"type\" : \"default\"\n  },\n  \"threshold\" : 5,\n  \"metric\" : \"count\",\n  \"aggregations\" : [\n    {\n      \"name\" : \"count\",\n      \"type\" : \"count\"\n    }\n  ],\n  \"intervals\" : [\n    \"2011-06-01/2017-06-01\"\n  ],\n  \"granularity\" : \"all\",\n  \"filter\" : null,\n  \"postAggregations\" : [\n  ],\n  \"context\" : {\n\n  },\n  \"queryType\" : \"topN\",\n  \"dataSource\" : \"wikipedia\"\n}\n```\n\n## Handling large payloads with Akka Streams\n\nFor queries with large payload of results (e.g., half a million of records), Scruid can transform the corresponding response into an [Akka Stream](https://doc.akka.io/docs/akka/2.5/stream/) Source.\nThe results can be processed, filtered and transformed using [Flows](https://doc.akka.io/docs/akka/2.5/stream/stream-flows-and-basics.html) and/or output to Sinks, as a continuous stream, without collecting the entire payload first.\nTo process the results with Akka Stream, you can call one of the following methods:\n\n  - `.stream`: gives a Source of `DruidResult`.\n  - `.streamAs[T](implicit decoder: Decoder[T])`: gives a Source where each JSON record is being decoded to the type of `T`.\n  - `.streamSeriesAs[T](implicit decoder: Decoder[T])`: gives a Source where each JSON record is being decoded to the type of `T` and it is accompanied by its corresponding timestamp.\n\nAll the methods above can be applied to any timeseries, group-by or top-N query created either directly by using query constructors or by DQL.\n\n## Druid SQL support\n\nInstead of using the Druid native API, Scruid also supports Druid queries via [SQL](https://druid.apache.org/docs/latest/querying/sql.html).\n\n```scala\nimport com.ing.wbaa.druid.SQL._\n\nval query = dsql\"\"\"SELECT COUNT(*) as \"count\" FROM wikipedia WHERE \"__time\" \u003e= TIMESTAMP '2015-09-12 00:00:00'\"\"\"\n\nval response = query.execute()\n```\n\nFor details see the [SQL documentation](docs/sql.md).\n\n### Example\n\n```scala\nimplicit val mat = DruidClient.materializer\n\ncase class TimeseriesCount(count: Int)\n\nval query = TimeSeriesQuery(\n  aggregations = List(\n    CountAggregation(name = \"count\")\n  ),\n  granularity = GranularityType.Hour,\n  intervals = List(\"2011-06-01/2017-06-01\")\n)\n\n// Decode each record into the type of `TimeseriesCount` and sum all `count` results\nval result: Future[Int] = query\n        .streamAs[TimeseriesCount]\n        .map(_.count)\n        .runWith(Sink.fold(0)(_ + _))\n```\n\n## Configuration\n\nThe configuration is done by [Typesafe config](https://github.com/typesafehub/config). The configuration can be overridden by using environment variables, e.g. `DRUID_HOSTS` (`DRUID_HOST` and `DRUID_PORT` are still supported for backward compatibility) and `DRUID_DATASOURCE`. Or by placing an application.conf in your own project and this will override the reference.conf of the scruid library.\n\n```\ndruid = {\n  host = \"localhost\"\n  host = ${?DRUID_HOST}\n  port = 8082\n  port = ${?DRUID_PORT}\n  hosts = ${druid.host}\":\"${druid.port}\n  hosts = ${?DRUID_HOSTS}\n  secure = false\n  secure = ${?DRUID_USE_SECURE_CONNECTION}\n  url = \"/druid/v2/\"\n  url = ${?DRUID_URL}\n  health-endpoint = \"/status/health\"\n  health-endpoint = ${?DRUID_HEALTH_ENDPOINT}\n  client-backend = \"com.ing.wbaa.druid.client.DruidHttpClient\"\n  client-backend = ${?DRUID_CLIENT_BACKEND}\n\n  scan-query-legacy-mode = false\n  scan-query-legacy-mode = ${?DRUID_SCAN_QUERY_LEGACY_MODE}\n\n  datasource = \"wikipedia\"\n  datasource = ${?DRUID_DATASOURCE}\n\n  response-parsing-timeout = 5 seconds\n  response-parsing-timeout = ${?DRUID_RESPONSE_PARSING_TIMEOUT}\n\n  zone-id = \"UTC\"\n}\n```\n\nAlternatively it can be programmatically overridden by defining an implicit instance of `com.ing.wbaa.druid.DruidConfig`:\n\n```scala\nimport java.time.ZonedDateTime\nimport com.ing.wbaa.druid._\nimport com.ing.wbaa.druid.definitions._\nimport scala.concurrent.duration._\n\n\nimplicit val druidConf = DruidConfig(\n  hosts = Seq(\"localhost:8082\"),\n  datasource = \"wikipedia\",\n  responseParsingTimeout = 10.seconds\n)\n\ncase class TimeseriesCount(count: Int)\n\nval response = TimeSeriesQuery(\n  aggregations = List(\n    CountAggregation(name = \"count\")\n  ),\n  granularity = GranularityType.Week,\n  intervals = List(\"2011-06-01/2017-06-01\")\n).execute\n\nval series: Map[ZonedDateTime, TimeseriesCount] = response.series[TimeseriesCount]\n```\n\nAll parameters of `DruidConfig` are optional, and in case that some parameter is missing then the default behaviour is to use the value that is defined in the configuration file.\n\n## Druid Clients\n\nScruid provides two client implementations, one for simple requests over a single Druid query host (default) and\nan advanced one with a queue, cached pool connections and, a load balancer when multiple Druid query hosts are provided.\nDepending on your use case, it is also possible to create a custom client. For details regarding clients, their\nconfiguration, as well the creation of a custom one see the [Scruid Clients](docs/scruid_clients.md) documentation.\n\n## Authentication\n\nThe Advanced client can be configured to authenticate with the Druid cluster. See the [Scruid Clients](docs/scruid_clients.md) document for more information.\n\n## Tests\n\nThe test suite relies on a docker-compose with supporting services. The dockerfiles for the images it uses are in the `docker/` subdirectory. Dependency versions of the dockerized resources are defined in `./env`.\n\nTo run the tests, please make sure that you have the Druid instance running:\n\n```\n./services.sh start\n```\n\nThis command will build the local images as needed. You can manually build these using the `./services.sh build_images` command or the Makefile in `./docker`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fing-bank%2Fscruid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fing-bank%2Fscruid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fing-bank%2Fscruid/lists"}