{"id":13597535,"url":"https://github.com/xebia-functional/fetch","last_synced_at":"2026-01-11T17:02:49.495Z","repository":{"id":37982591,"uuid":"53124080","full_name":"xebia-functional/fetch","owner":"xebia-functional","description":"Simple \u0026 Efficient data access for Scala and Scala.js","archived":false,"fork":false,"pushed_at":"2025-09-15T05:17:41.000Z","size":4537,"stargazers_count":496,"open_issues_count":17,"forks_count":50,"subscribers_count":51,"default_branch":"main","last_synced_at":"2025-12-21T12:15:34.948Z","etag":null,"topics":["cats","concurrency","data","data-fetching","monads","monix","parallelism","scala","scala-js","sequencing"],"latest_commit_sha":null,"homepage":"https://xebia-functional.github.io/fetch/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xebia-functional.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE.md","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2016-03-04T09:32:16.000Z","updated_at":"2025-11-23T14:06:26.000Z","dependencies_parsed_at":"2023-11-22T06:28:21.896Z","dependency_job_id":"e1ad7dbb-9088-428d-b803-f20c4f273b33","html_url":"https://github.com/xebia-functional/fetch","commit_stats":null,"previous_names":["47degrees/fetch","47deg/fetch"],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/xebia-functional/fetch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xebia-functional%2Ffetch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xebia-functional%2Ffetch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xebia-functional%2Ffetch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xebia-functional%2Ffetch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xebia-functional","download_url":"https://codeload.github.com/xebia-functional/fetch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xebia-functional%2Ffetch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28314260,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-11T14:58:17.114Z","status":"ssl_error","status_checked_at":"2026-01-11T14:55:53.580Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cats","concurrency","data","data-fetching","monads","monix","parallelism","scala","scala-js","sequencing"],"created_at":"2024-08-01T17:00:35.511Z","updated_at":"2026-01-11T17:02:49.481Z","avatar_url":"https://github.com/xebia-functional.png","language":"Scala","readme":"# Fetch\n\n[![Join the chat at https://gitter.im/47deg/fetch](https://badges.gitter.im/47deg/fetch.svg)](https://gitter.im/47deg/fetch?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge) [![Maven Central](https://img.shields.io/badge/maven%20central-1.2.1-green.svg)](https://oss.sonatype.org/#nexus-search;gav~com.47deg~fetch*) [![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](https://raw.githubusercontent.com/xebia-functional/fetch/master/LICENSE.md) [![Latest version](https://img.shields.io/badge/fetch-1.2.1-green.svg)](https://index.scala-lang.org/47deg/fetch) [![Scala.js](http://scala-js.org/assets/badges/scalajs-0.6.15.svg)](http://scala-js.org) [![GitHub Issues](https://img.shields.io/github/issues/47deg/fetch.svg)](https://github.com/xebia-functional/fetch/issues)\n\nA library for Simple \u0026 Efficient data access in Scala and Scala.js\n\n- [Documentation](https://xebia-functional.github.io/fetch/docs)\n\n---\n\n- [Installation](#installation)\n- [Remote data](#remote-data)\n- [Define your data sources](#define-your-data-sources)\n- [Creating a runtime](#creating-a-runtime)\n- [Creating and running a fetch](#creating-and-running-a-fetch)\n- [Batching](#batching)\n- [Parallelism](#parallelism)\n- [Deduplication \u0026 Caching](#deduplication--caching)\n\n## Installation\n\nAdd the following dependency to your project's build file.\n\nFor Scala 2.12.x through 3.x:\n\n```scala\n\"com.47deg\" %% \"fetch\" % \"3.1.2\"\n```\n\nOr, if using Scala.js (1.8.x):\n\n```scala\n\"com.47deg\" %%% \"fetch\" % \"3.1.2\"\n```\n\n\n## Remote data\n\nFetch is a library for making access to data both simple and efficient. Fetch is especially useful when querying data that\nhas a latency cost, such as databases or web services.\n\n## Define your data sources\n\nTo tell Fetch how to get the data you want, you must implement the `DataSource` typeclass. Data sources have `fetch` and `batch` methods that define how to fetch such a piece of data.\n\nData Sources take two type parameters:\n\n\u003col\u003e\n\u003cli\u003e\u003ccode\u003eIdentity\u003c/code\u003e is a type that has enough information to fetch the data\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eResult\u003c/code\u003e is the type of data we want to fetch\u003c/li\u003e\n\u003c/ol\u003e\n\n```scala\nimport cats.data.NonEmptyList\nimport cats.effect.Concurrent\n\ntrait DataSource[F[_], Identity, Result]{\n  def data: Data[Identity, Result]\n  def CF: Concurrent[F]\n  def fetch(id: Identity): F[Option[Result]]\n  def batch(ids: NonEmptyList[Identity]): F[Map[Identity, Result]]\n}\n```\n\nReturning `Concurrent` instances from the fetch methods allows us to specify if the fetch must run synchronously or asynchronously, and use all the goodies available in `cats` and `cats-effect`.\n\nWe'll implement a dummy data source that can convert integers to strings. For convenience, we define a `fetchString` function that lifts identities (`Int` in our dummy data source) to a `Fetch`.\n\n```scala\nimport cats._\nimport cats.data.NonEmptyList\nimport cats.effect._\nimport cats.implicits._\n\nimport fetch._\n\ndef latency[F[_] : Sync](milis: Long): F[Unit] =\n  Sync[F].delay(Thread.sleep(milis))\n\nobject ToString extends Data[Int, String] {\n  def name = \"To String\"\n\n  def source[F[_] : Async]: DataSource[F, Int, String] = new DataSource[F, Int, String]{\n    override def data = ToString\n\n    override def CF = Concurrent[F]\n\n    override def fetch(id: Int): F[Option[String]] = for {\n      _ \u003c- CF.delay(println(s\"--\u003e [${Thread.currentThread.getId}] One ToString $id\"))\n      _ \u003c- latency(100)\n      _ \u003c- CF.delay(println(s\"\u003c-- [${Thread.currentThread.getId}] One ToString $id\"))\n    } yield Option(id.toString)\n\n    override def batch(ids: NonEmptyList[Int]): F[Map[Int, String]] = for {\n      _ \u003c- CF.delay(println(s\"--\u003e [${Thread.currentThread.getId}] Batch ToString $ids\"))\n      _ \u003c- latency(100)\n      _ \u003c- CF.delay(println(s\"\u003c-- [${Thread.currentThread.getId}] Batch ToString $ids\"))\n    } yield ids.toList.map(i =\u003e (i, i.toString)).toMap\n  }\n}\n\ndef fetchString[F[_] : Async](n: Int): Fetch[F, String] =\n  Fetch(n, ToString.source)\n```\n\n## Creating a runtime\n\nSince we'll use `IO` from the `cats-effect` library to execute our fetches, we'll need an `IORuntime` for executing our `IO` instances.\n\n```scala\nimport cats.effect.unsafe.implicits.global //Gives us an IORuntime in places it is normally not provided\n```\n\nNormally, in your applications, this is provided by `IOApp`, and you should not need to import this except in limited scenarios such as test environments that do not have Cats Effect integration.\nFor more information, and particularly on why you would usually not want to make one of these yourself, [see this post by Daniel Spiewak](https://github.com/typelevel/cats-effect/discussions/1562#discussioncomment-254838)\n\n## Creating and running a fetch\n\nNow that we can convert `Int` values to `Fetch[F, String]`, let's try creating a fetch.\n\n```scala\ndef fetchOne[F[_] : Async]: Fetch[F, String] =\n  fetchString(1)\n```\n\nLet's run it and wait for the fetch to complete. We'll use `IO#unsafeRunTimed` for testing purposes, which will run an `IO[A]` to `Option[A]` and return `None` if it didn't complete in time:\n\n```scala\nimport scala.concurrent.duration._\n\nFetch.run[IO](fetchOne).unsafeRunTimed(5.seconds)\n// --\u003e [173] One ToString 1\n// \u003c-- [173] One ToString 1\n// res0: Option[String] = Some(value = \"1\")\n```\n\nAs you can see in the previous example, the `ToStringSource` is queried once to get the value of 1.\n\n## Batching\n\nMultiple fetches to the same data source are automatically batched. For illustrating this, we are going to compose three independent fetch results as a tuple.\n\n```scala\ndef fetchThree[F[_] : Async]: Fetch[F, (String, String, String)] =\n  (fetchString(1), fetchString(2), fetchString(3)).tupled\n```\n\nWhen executing the above fetch, note how the three identities get batched, and the data source is only queried once.\n\n```scala\nFetch.run[IO](fetchThree).unsafeRunTimed(5.seconds)\n// --\u003e [172] Batch ToString NonEmptyList(1, 2, 3)\n// \u003c-- [172] Batch ToString NonEmptyList(1, 2, 3)\n// res1: Option[(String, String, String)] = Some(value = (\"1\", \"2\", \"3\"))\n```\n\nNote that the `DataSource#batch` method is not mandatory. It will be implemented in terms of `DataSource#fetch` if you don't provide an implementation.\n\n```scala\nobject UnbatchedToString extends Data[Int, String] {\n  def name = \"Unbatched to string\"\n\n  def source[F[_]: Async] = new DataSource[F, Int, String] {\n    override def data = UnbatchedToString\n\n    override def CF = Concurrent[F]\n\n    override def fetch(id: Int): F[Option[String]] = \n      CF.delay(println(s\"--\u003e [${Thread.currentThread.getId}] One UnbatchedToString $id\")) \u003e\u003e\n      latency(100) \u003e\u003e\n      CF.delay(println(s\"\u003c-- [${Thread.currentThread.getId}] One UnbatchedToString $id\")) \u003e\u003e\n      CF.pure(Option(id.toString))\n  }\n}\n\ndef unbatchedString[F[_]: Async](n: Int): Fetch[F, String] =\n  Fetch(n, UnbatchedToString.source)\n```\n\nLet's create a tuple of unbatched string requests.\n\n```scala\ndef fetchUnbatchedThree[F[_] : Async]: Fetch[F, (String, String, String)] =\n  (unbatchedString(1), unbatchedString(2), unbatchedString(3)).tupled\n```\n\nWhen executing the above fetch, note how the three identities get requested in parallel. You can override `batch` to execute queries sequentially if you need to.\n\n```scala\nFetch.run[IO](fetchUnbatchedThree).unsafeRunTimed(5.seconds)\n// --\u003e [172] One UnbatchedToString 1\n// --\u003e [173] One UnbatchedToString 2\n// \u003c-- [172] One UnbatchedToString 1\n// --\u003e [172] One UnbatchedToString 3\n// \u003c-- [173] One UnbatchedToString 2\n// \u003c-- [172] One UnbatchedToString 3\n// res2: Option[(String, String, String)] = Some(value = (\"1\", \"2\", \"3\"))\n```\n\n## Parallelism\n\nIf we combine two independent fetches from different data sources, the fetches can be run in parallel. First, let's add a data source that fetches a string's size.\n\n```scala\nobject Length extends Data[String, Int] {\n  def name = \"Length\"\n\n  def source[F[_] : Async] = new DataSource[F, String, Int] {\n    override def data = Length\n\n    override def CF = Concurrent[F]\n\n    override def fetch(id: String): F[Option[Int]] = for {\n      _ \u003c- CF.delay(println(s\"--\u003e [${Thread.currentThread.getId}] One Length $id\"))\n      _ \u003c- latency(100)\n      _ \u003c- CF.delay(println(s\"\u003c-- [${Thread.currentThread.getId}] One Length $id\"))\n    } yield Option(id.size)\n\n    override def batch(ids: NonEmptyList[String]): F[Map[String, Int]] = for {\n      _ \u003c- CF.delay(println(s\"--\u003e [${Thread.currentThread.getId}] Batch Length $ids\"))\n      _ \u003c- latency(100)\n      _ \u003c- CF.delay(println(s\"\u003c-- [${Thread.currentThread.getId}] Batch Length $ids\"))\n    } yield ids.toList.map(i =\u003e (i, i.size)).toMap\n  }\n}\n\ndef fetchLength[F[_] : Async](s: String): Fetch[F, Int] =\n  Fetch(s, Length.source)\n```\n\nAnd now we can easily receive data from the two sources in a single fetch.\n\n```scala\ndef fetchMulti[F[_] : Async]: Fetch[F, (String, Int)] =\n  (fetchString(1), fetchLength(\"one\")).tupled\n```\n\nNote how the two independent data fetches run in parallel, minimizing the latency cost of querying the two data sources.\n\n```scala\nFetch.run[IO](fetchMulti).unsafeRunTimed(5.seconds)\n// --\u003e [173] One Length one\n// --\u003e [172] One ToString 1\n// \u003c-- [172] One ToString 1\n// \u003c-- [173] One Length one\n// res3: Option[(String, Int)] = Some(value = (\"1\", 3))\n```\n\n## Deduplication \u0026 Caching\n\nThe Fetch library supports deduplication and optional caching.\nBy default, fetches that are chained together will share the same cache backend, providing some deduplication.\n\nWhen fetching an identity twice within the same `Fetch`, such as a batch of fetches or when you `flatMap` one fetch into another, subsequent fetches for the same identity are cached.\nLet's try creating a fetch that asks for the same identity twice, by using `flatMap` (in a for-comprehension) to chain the requests together:\n\n```scala\ndef fetchTwice[F[_] : Async]: Fetch[F, (String, String)] = for {\n  one \u003c- fetchString(1)\n  two \u003c- fetchString(1)\n} yield (one, two)\n```\n\nWhile running it, notice that the data source is only queried once.\nThe next time the identity is requested, it's served from the internal cache.\n\n```scala\nval runFetchTwice = Fetch.run[IO](fetchTwice)\n```\n```scala\nrunFetchTwice.unsafeRunTimed(5.seconds)\n// --\u003e [173] One ToString 1\n// \u003c-- [173] One ToString 1\n// res4: Option[(String, String)] = Some(value = (\"1\", \"1\"))\n```\n\nThis will still fetch the data again, however, if we call it once more:\n```scala\nrunFetchTwice.unsafeRunTimed(5.seconds)\n// --\u003e [172] One ToString 1\n// \u003c-- [172] One ToString 1\n// res5: Option[(String, String)] = Some(value = (\"1\", \"1\"))\n```\n\nIf we want to cache between multiple individual fetches, you should use `Fetch.runCache` or `Fetch.runAll` to return the cache for reusing later.\nHere is an example where we fetch four separate times, and explicitly share the cache to keep the deduplication functionality:\n\n```scala\n//We get the cache from the first run and pass it to all subsequent fetches\nval runFetchFourTimesSharedCache = for {\n  (cache, one) \u003c- Fetch.runCache[IO](fetchString(1))\n  two \u003c- Fetch.run[IO](fetchString(1), cache)\n  three \u003c- Fetch.run[IO](fetchString(1), cache)\n  four \u003c- Fetch.run[IO](fetchString(1), cache)\n} yield (one, two, three, four)\n```\n```scala\nrunFetchFourTimesSharedCache.unsafeRunTimed(5.seconds)\n// --\u003e [173] One ToString 1\n// \u003c-- [173] One ToString 1\n// res6: Option[(String, String, String, String)] = Some(\n//   value = (\"1\", \"1\", \"1\", \"1\")\n// )\n```\n\nAs you can see above, the cache will now work between calls and can be used to deduplicate requests over a period of time.\nNote that this does not support any kind of automatic cache invalidation, so you will need to keep track of which values you want to re-fetch if you plan on sharing the cache.\n---\n\nFor more in-depth information, take a look at our [documentation](https://xebia-functional.github.io/fetch/docs.html).\n\n# Copyright\n\nFetch is designed and developed by 47 Degrees\n\nCopyright (C) 2016-2023 47 Degrees. \u003chttp://47deg.com\u003e\n","funding_links":[],"categories":["Libraries"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxebia-functional%2Ffetch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxebia-functional%2Ffetch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxebia-functional%2Ffetch/lists"}