https://github.com/casehubdk/hxl
Pure batching for Scala
https://github.com/casehubdk/hxl
batch-processing functional-programming haxl scala typelevel
Last synced: 2 months ago
JSON representation
Pure batching for Scala
- Host: GitHub
- URL: https://github.com/casehubdk/hxl
- Owner: casehubdk
- License: other
- Created: 2023-05-22T16:50:13.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2025-12-13T00:40:52.000Z (3 months ago)
- Last Synced: 2025-12-14T15:32:55.977Z (3 months ago)
- Topics: batch-processing, functional-programming, haxl, scala, typelevel
- Language: Scala
- Homepage:
- Size: 75.2 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Hxl 
Hxl is a pure applicative batching library for Scala.
Hxl is based on the ideas presented in [Haxl](https://simonmar.github.io/bib/papers/haxl-icfp14.pdf), but diverges in in a few ways.
Notably, Hxl does not perform side effects, but is instead based on a free applicative structure.
Hxl is very small (only a couple hundred lines of code) and only depends on cats.
Hxl is written in tagless final, which allows for molding the library to your needs.
## Installation
Hxl is available on Maven Central for Scala 2.13 and 3.2.
```scala
libraryDependencies += "com.github.casehubdk" %% "hxl" % "0.3.0"
```
## Usage
There are two primitive structures in Hxl: `DataSource[F, K, V]` and `DSKey[K, V]`.
A `DataSource[F, K, V]` abstracts over a function `NonEmptyList[K] => F[Map[K, V]]`.
A `DataSource` is uniquely (as in Scala universal equals) identified by its `DSKey`.
```scala
import cats._
import cats.implicits._
final case class MyDSKey(id: String)
case object MyDSKey extends DSKey[MyDSKey, String]
val database = Map(
"foo" -> "bar",
"baz" -> "qux"
)
def dataSource[F[_]: Applicative] = DataSource.from[F, MyDSKey, String](MyDSKey) { keys =>
keys.toList.flatMap(key => database.get(key.id).toList.tupleLeft(key)).toMap.pure[F]
}
val fa: Hxl[Id, Option[String]] = Hxl(MyDSKey("foo"), dataSource[Id])
val fb: Hxl[Id, Option[String]] = Hxl(MyDSKey("baz"), dataSource[Id])
Hxl.runSequential((fa, fb).mapN(_.mkString + " " + _.mkString)) // "bar qux"
```
Hxl forms an applicative, but sometimes you need a monad.
Hxl is like `Validated` from `cats`, in that it can escape it's applicative nature via a method `andThen`.
However, if you need Hxl to become a monad (like `Either` is to `Validated`), you can use request a monadic view of your effect:
```scala
val fa: Hxl[F, String] = ???
val m: HxlM[F, String] = fa.monadic.flatMap{ x =>
???
}
val back: Hxl[F, String] = m.hxl
```
## Advanced usage
Since Hxl is written in tagless final, you can add various behaviors to your data sources.
For instance, you can add (pure) caching.
```scala
import cats.data._
import cats.implicits._
final case class MyDSKey(id: String)
case object MyDSKey extends DSKey[MyDSKey, String]
val database = Map(
"foo" -> "bar",
"baz" -> "qux"
)
type Cache = Map[String, String]
type Effect[A] = State[Cache, A]
def dataSource: DataSource[Effect, MyDSKey, String] = DataSource.from(MyDSKey) { keys =>
State[Cache, Map[MyDSKey, String]] { cache =>
val (misses, hits) = keys.toList.partitionEither(k => cache.get(k.id).tupleLeft(k).toRight(k))
val fetched = misses.flatMap(key => database.get(key.id).toList.map(key -> _)).toMap
(cache ++ fetched.map{ case (k, v) => k.id -> v }, hits.toMap ++ fetched)
}
}
```
## Extending Hxl
Hxl's interface is public and small, so extension is very possible.
Under the hood, Hxl compiles your structure into `F[A]` via a natural transformation:
```scala
type Target[F[_], G[_], A] = G[Either[Hxl[F, A], A]]
type Compiler[F[_], G[_]] = Hxl[F, *] ~> Target[F, G, *]
```
Hxl repeats your natural transformation until the result becomes `Right`, like `tailRecM`.
```scala
def fa: Hxl[F, A] = ???
fa.foldMap(Hxl.parallelRunner[F]): F[A]
```
`Compiler`s can be composed like ordinary functions such that the core of Hxl is exposed for extension.
As an example, let's add tracing (from `natchez`) to Hxl:
```scala
import _root_.natchez._
import cats.data._
import cats.implicits._
import cats._
import Hxl._
def traceRequests[F[_]: Trace: Applicative, A](req: Requests[F, A]): Requests[F, A] = {
def traceSource[K, V](source: DataSource[F, K, V]): DataSource[F, K, V] =
DataSource.full[F, K, V](source.key) { ks =>
Trace[F].span(s"datasource.${source.key}") {
Trace[F].put("keys" -> ks.size.toString) *> source.batch(ks)
}
}(source.optimization)
req.visit {
new Requests.DataSourceVisitor[F] {
def visit[K, V](source: DataSource[F, K, V], k: K): (DataSource[F, K, V], K) =
(traceSource(source), k)
}
}
}
def composeTracing[F[_]: Trace: Applicative, G[_]: Trace: Applicative](
compiler: Compiler[F, G]
): Compiler[F, StateT[G, Int, *]] = {
type Effect[A] = StateT[G, Int, A]
new Compiler[F, Effect] {
def apply[A](fa: Hxl[F, A]): Hxl.Target[F, Effect, A] =
fa match {
case Hxl.LiftF(unFetch) =>
StateT.liftF {
Trace[G].span("hxl.fetch") {
compiler(Hxl.LiftF(unFetch))
}
}
case bind: Hxl.Bind[F, a, b] =>
StateT { round: Int =>
Trace[G]
.span("hxl.bind") {
Trace[G].put("round" -> round.toString) *> compiler {
Hxl.Bind(traceRequests(bind.requests), bind.f)
}
}
.map(round + 1 -> _)
}
case other => StateT.liftF(compiler(other))
}
}
}
def fa: Hxl[F, String] = ???
val result: F[String] = fa.foldMap(composeTracing[F, F](Hxl.parallelRunner)).runA(0)
```