Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/zio/zio-query
Add efficient pipelining, batching, and caching to any data source
https://github.com/zio/zio-query
functional-programming query-optimization scala zio
Last synced: 1 day ago
JSON representation
Add efficient pipelining, batching, and caching to any data source
- Host: GitHub
- URL: https://github.com/zio/zio-query
- Owner: zio
- License: apache-2.0
- Created: 2020-05-10T23:06:58.000Z (over 4 years ago)
- Default Branch: series/2.x
- Last Pushed: 2024-09-29T08:04:07.000Z (3 months ago)
- Last Synced: 2024-12-18T04:03:56.758Z (9 days ago)
- Topics: functional-programming, query-optimization, scala, zio
- Language: Scala
- Homepage: https://zio.dev/zio-query
- Size: 888 KB
- Stars: 151
- Watchers: 16
- Forks: 21
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
[//]: # (This file was autogenerated using `zio-sbt-website` plugin via `sbt generateReadme` command.)
[//]: # (So please do not edit it manually. Instead, change "docs/index.md" file or sbt setting keys)
[//]: # (e.g. "readmeDocumentation" and "readmeSupport".)
# ZIO Query
[ZIO Query](https://github.com/zio/zio-query) is a library for writing optimized queries to data sources in a high-level compositional style. It can add efficient pipelining, batching, and caching to any data source. ZIO Query helps us dramatically reduce load on data sources and improve performance.
[](https://github.com/zio/zio/wiki/Project-Stages)  [](https://oss.sonatype.org/content/repositories/releases/dev/zio/zio-query_2.13/) [](https://oss.sonatype.org/content/repositories/snapshots/dev/zio/zio-query_2.13/) [](https://javadoc.io/doc/dev.zio/zio-query-docs_2.13) [](https://github.com/zio/zio-query)
## Introduction
Some key features of ZIO Query:
- **Batching** — ZIO Query detects parts of composite queries that can be executed in parallel without changing the semantics of the query.
- **Pipelining** — ZIO Query detects parts of composite queries that can be combined together for fewer individual requests to the data source.
- **Caching** — ZIO Query can transparently cache read queries to minimize the cost of fetching the same item repeatedly in the scope of a query.
Compared with Fetch, ZIO Query supports response types that depend on request types, does not require higher-kinded types and implicits, supports ZIO environment and statically typed errors, and has no dependencies except for ZIO.
A `ZQuery[R, E, A]` is a purely functional description of an effectual query that may contain requests from one or more data sources, requires an environment `R`, and may fail with an `E` or succeed with an `A`.
Requests that can be performed in parallel, as expressed by `zipWithPar` and combinators derived from it, will automatically be batched. Requests that must be performed sequentially, as expressed by `zipWith` and combinators derived from it, will automatically be pipelined. This allows for aggressive data source specific optimizations. Requests can also be deduplicated and cached.
This allows for writing queries in a high level, compositional style, with confidence that they will automatically be optimized. For example, consider the following query from a user service.
Assume we have the following database access layer APIs:
```scala
def getAllUserIds: ZIO[Any, Nothing, List[Int]] = {
// Get all user IDs e.g. SELECT id FROM users
ZIO.succeed(???)
}
def getUserNameById(id: Int): ZIO[Any, Nothing, String] = {
// Get user by ID e.g. SELECT name FROM users WHERE id = $id
ZIO.succeed(???)
}
```
We can get their corresponding usernames from the database by the following code snippet:
```scala
val userNames = for {
ids <- getAllUserIds
names <- ZIO.foreachPar(ids)(getUserNameById)
} yield names
```
It works, but this is not performant. It is going to query the underlying database _N + 1_ times, one for `getAllUserIds` and one for each call to `getUserNameById`.
In contrast, `ZQuery` will automatically optimize this to two queries, one for `userIds` and one for `userNames`:
```scala
lazy val getAllUserIds: ZQuery[Any, Nothing, List[Int]] = ???
def getUserNameById(id: Int): ZQuery[Any, Nothing, String] = ???
lazy val userQuery: ZQuery[Any, Nothing, List[String]] = for {
userIds <- getAllUserIds
userNames <- ZQuery.foreachPar(userIds)(getUserNameById)
} yield userNames
```
## Installation
In order to use this library, we need to add the following line in our `build.sbt` file:
```scala
libraryDependencies += "dev.zio" %% "zio-query" % "0.7.6"
```
## Example
Here is an example of using ZIO Query, which optimizes multiple database queries by batching all of them in one query:
```scala
import zio._
import zio.query._
object ZQueryExample extends ZIOAppDefault {
case class GetUserName(id: Int) extends Request[Throwable, String]
lazy val UserDataSource: DataSource.Batched[Any, GetUserName] =
new DataSource.Batched[Any, GetUserName] {
val identifier: String = "UserDataSource"
def run(requests: Chunk[GetUserName])(implicit trace: Trace): ZIO[Any, Nothing, CompletedRequestMap] =
requests.toList match {
case request :: Nil =>
val result: Task[String] = {
// get user by ID e.g. SELECT name FROM users WHERE id = $id
ZIO.succeed(???)
}
result.exit.map(CompletedRequestMap.single(request, _))
case batch: Seq[GetUserName] =>
val result: Task[List[(Int, String)]] = {
// get multiple users at once e.g. SELECT id, name FROM users WHERE id IN ($ids)
ZIO.succeed(???)
}
result.foldCause(
CompletedRequestMap.failCause(requests, _),
CompletedRequestMap.fromIterableWith(_)(kv => GetUserName(kv._1), kv => Exit.succeed(kv._2))
)
}
}
def getUserNameById(id: Int): ZQuery[Any, Throwable, String] =
ZQuery.fromRequest(GetUserName(id))(UserDataSource)
val query: ZQuery[Any, Throwable, List[String]] =
for {
ids <- ZQuery.succeed(1 to 10)
names <- ZQuery.foreachPar(ids)(id => getUserNameById(id)).map(_.toList)
} yield (names)
def run = query.run.tap(usernames => Console.printLine(s"Usernames: $usernames"))
}
```
## Resources
- [Wicked Fast API Calls with ZIO Query](https://www.youtube.com/watch?v=rUUxDXJMzJo) by Adam Fraser (July 2020) (https://www.youtube.com/watch?v=rUUxDXJMzJo)
## Documentation
Learn more on the [ZIO Query homepage](https://zio.dev/zio-query)!
## Contributing
For the general guidelines, see ZIO [contributor's guide](https://zio.dev/contributor-guidelines).
## Code of Conduct
See the [Code of Conduct](https://zio.dev/code-of-conduct)
## Support
Come chat with us on [![Badge-Discord]][Link-Discord].
[Badge-Discord]: https://img.shields.io/discord/629491597070827530?logo=discord "chat on discord"
[Link-Discord]: https://discord.gg/2ccFBr4 "Discord"
## License
[License](LICENSE)