{"id":13499447,"url":"https://github.com/engagingspaces/vertx-dataloader","last_synced_at":"2025-10-23T15:30:24.341Z","repository":{"id":73924401,"uuid":"65939518","full_name":"engagingspaces/vertx-dataloader","owner":"engagingspaces","description":" Efficient, asynchronous batching and caching in clustered environments, port of Facebook DataLoader","archived":false,"fork":false,"pushed_at":"2017-08-06T07:12:46.000Z","size":105,"stargazers_count":71,"open_issues_count":1,"forks_count":10,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-01-30T19:07:59.053Z","etag":null,"topics":["batch","dataloader","facebook-dataloader","graphql","java-8","vertx","vertx-dataloader"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/engagingspaces.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-08-17T20:26:24.000Z","updated_at":"2024-03-31T14:18:15.000Z","dependencies_parsed_at":"2024-01-03T02:25:31.152Z","dependency_job_id":"603a694c-dcc6-4a94-b6fd-4806f935c40e","html_url":"https://github.com/engagingspaces/vertx-dataloader","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engagingspaces%2Fvertx-dataloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engagingspaces%2Fvertx-dataloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engagingspaces%2Fvertx-dataloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/engagingspaces%2Fvertx-dataloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/engagingspaces","download_url":"https://codeload.github.com/engagingspaces/vertx-dataloader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237843804,"owners_count":19375206,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","dataloader","facebook-dataloader","graphql","java-8","vertx","vertx-dataloader"],"created_at":"2024-07-31T22:00:33.216Z","updated_at":"2025-10-23T15:30:23.827Z","avatar_url":"https://github.com/engagingspaces.png","language":"Java","readme":"# Vert.x `DataLoader`\n\n[![Build Status](https://travis-ci.org/engagingspaces/vertx-dataloader.svg?branch=master)](https://travis-ci.org/engagingspaces/vertx-dataloader/)\u0026nbsp;\u0026nbsp;\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/ec906aa3a12147e28b69b93e3a9d9bf7)](https://www.codacy.com/app/engagingspaces/vertx-dataloader?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=engagingspaces/vertx-dataloader\u0026amp;utm_campaign=Badge_Grade)\u0026nbsp;\u0026nbsp;\n[![Apache licensed](https://img.shields.io/hexpm/l/plug.svg?maxAge=2592000)](https://github.com/engagingspaces/vertx-dataloader/blob/master/LICENSE)\u0026nbsp;\u0026nbsp;\n[ ![Download](https://api.bintray.com/packages/engagingspaces/maven/vertx-dataloader/images/download.svg) ](https://bintray.com/engagingspaces/maven/vertx-dataloader/_latestVersion)\n\n**Note**: A pure java 8 (non-Vert.x) port of this project is now an official [graphql-java](https://github.com/graphql-java/awesome-graphql-java#batch-loading) project: [java-dataloader](https://github.com/graphql-java/java-dataloader) \n\n![vertx-dataloader-concepts](https://cloud.githubusercontent.com/assets/5111931/17837825/f5748bfc-67bd-11e6-9c7a-d407bb92c3d9.png)\n\nThis small and simple utility library is a port of [Facebook DataLoader](https://github.com/facebook/dataloader)\nto Java 8 for use with [Vert.x](http://vertx.io). It can serve as integral part of your application's data layer to provide a\nconsistent API over various back-ends and reduce message communication overhead through batching and caching.\n\nAn important use case for `DataLoader` is improving the efficiency of GraphQL query execution, but there are\nmany other use cases where you can benefit from using this utility.\n\nMost of the code is ported directly from Facebook's reference implementation, with one IMPORTANT adaptation to make\nit work for Java 8 and Vert.x. ([more on this below](manual-dispatching)).\n\nBut before reading on, be sure to take a short dive into the\n[original documentation](https://github.com/facebook/dataloader/blob/master/README.md) provided by Lee Byron (@leebyron)\nand Nicholas Schrock (@schrockn) from [Facebook](https://www.facebook.com/), the creators of the original data loader.\n\n## Table of contents\n\n- [Features](#features)\n- [Differences to reference implementation](#differences-to-reference-implementation)\n  - [Manual dispatching](#manual-dispatching)\n- [Let's get started!](#lets-get-started)\n  - [Installing](#installing)\n  - [Building](#building)\n- [Project plans](#project-plans)\n  - [Current releases](#current-releases)\n  - [Known issues](#known-issues)\n  - [Future ideas](#future-ideas)\n- [Other information sources](#other-information-sources)\n- [Contributing](#contributing)\n- [Acknowledgements](#acknowledgements)\n- [Licensing](#licensing)\n\n## Features\n\nVert.x `DataLoader` is a feature-complete port of the Facebook reference implementation with [one major difference](#manual-dispatching). These features are:\n\n- Simple, intuitive API, using generics and fluent coding\n- Define batch load function with lambda expression\n- Schedule a load request in queue for batching\n- Add load requests from anywhere in code\n- Request returns [`Future\u003cV\u003e`](http://vertx.io/docs/apidocs/io/vertx/core/Future.html) of requested value\n- Can create multiple requests at once, returns [`CompositeFuture`](http://vertx.io/docs/apidocs/io/vertx/core/CompositeFuture.html)\n- Caches load requests, so data is only fetched once\n- Can clear individual cache keys, so data is fetched on next batch queue dispatch\n- Can prime the cache with key/values, to avoid data being fetched needlessly\n- Can configure cache key function with lambda expression to extract cache key from complex data loader key types\n- Dispatch load request queue after batch is prepared, also returns [`CompositeFuture`](http://vertx.io/docs/apidocs/io/vertx/core/CompositeFuture.html)\n- Individual batch futures complete / resolve as batch is processed\n- `CompositeFuture`s results are ordered according to insertion order of load requests\n- Deals with partial errors when a batch future fails\n- Can disable batching and/or caching in configuration\n- Can supply your own [`CacheMap\u003cK, V\u003e`](https://github.com/engagingspaces/vertx-dataloader/blob/master/src/main/java/io/engagingspaces/vertx/dataloader/CacheMap.java) implementations\n- Has very high test coverage (see [Acknowledgements](#acknowlegdements))\n\n## Differences to reference implementation\n\n### Manual dispatching\n\nThe original data loader was written in Javascript for NodeJS. NodeJS is single-threaded in nature, but simulates\nasynchronous logic by invoking functions on separate threads in an event loop, as explained\n[in this post](http://stackoverflow.com/a/19823583/3455094) on StackOverflow.\n\n[Vert.x](http://vertx.io) on the other hand also uses an event loop ([that you should not block!!](http://vertx.io/docs/vertx-core/java/#golden_rule)), but comes\nwith actor-like [`Verticle`](http://vertx.io/docs/vertx-core/java/#_verticles)s and a\ndistributed [`EventBus`](http://vertx.io/docs/vertx-core/java/#event_bus) that make it inherently asynchronous, and non-blocking.\n\nNow in NodeJS generates so-call 'ticks' in which queued functions are dispatched for execution, and Facebook `DataLoader` uses\nthe `nextTick()` function in NodeJS to _automatically_ dequeue load requests and send them to the batch execution function for processing.\n\nAnd here there is an **IMPORTANT DIFFERENCE** compared to how _this_ data loader operates!!\n\nIn NodeJS the batch preparation will not affect the asynchronous processing behaviour in any way. It will just prepare\nbatches in 'spare time' as it were.\n\nThis is different in Vert.x as you will actually _delay_ the execution of your load requests, until the moment where you make a call\nto `dataLoader.dispatch()` in comparison to when you would just handle futures directly.\n\nDoes this make Java `DataLoader` any less useful than the reference implementation? I would argue this is not the case,\nand there are also gains to this different mode of operation:\n\n- In contrast to the NodeJS implementation _you_ as developer are in full control of when batches are dispatched\n- You can attach any logic that determines when a dispatch takes place\n- You still retain all other features, full caching support and batching (e.g. to optimize message bus traffic, GraphQL query execution time, etc.)\n\nHowever, with batch execution control comes responsibility! If you forget to make the call to `dispatch()` then the futures\nin the load request queue will never be batched, and thus _will never complete_! So be careful when crafting your loader designs.\n\n## Let's get started!\n\n### Installing\n\nGradle users configure the `vertx-dataloader` dependency in `build.gradle`:\n\n```\nrepositories {\n    maven {\n        jcenter()\n    }\n}\n\ndependencies {\n    compile 'io.engagingspaces:vertx-dataloader:1.0.0'\n}\n```\n\n### Building\n\nTo build from source use the Gradle wrapper:\n\n```\n./gradlew clean build\n```\n\nOr when using Maven add the following repository to your `pom.xml`:\n\n```\n\u003crepositories\u003e\n    \u003crepository\u003e\n        \u003csnapshots\u003e\n            \u003cenabled\u003efalse\u003c/enabled\u003e\n        \u003c/snapshots\u003e\n        \u003cid\u003ecentral\u003c/id\u003e\n        \u003cname\u003ebintray\u003c/name\u003e\n        \u003curl\u003ehttp://jcenter.bintray.com\u003c/url\u003e\n    \u003c/repository\u003e\n\u003c/repositories\u003e\n```\n\nAnd add the dependency to `vertx-dataloader`:\n\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003eio.engagingspaces\u003c/groupId\u003e\n    \u003cartifactId\u003evertx-dataloader\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0\u003c/version\u003e\n    \u003ctype\u003epom\u003c/type\u003e\n\u003c/dependency\u003e\n```\n\n### Using\n\nPlease take a look at the example project [vertx-graphql-example](https://github.com/bmsantos/vertx-graphql-example)\ncreated by [Bruno Santos](https://github.com/bmsantos).\n\n## Project plans\n\n### Current releases\n\n- `1.0.0` Initial release\n\n### Known issues\n\n- Tests on job queue ordering need refactoring to Futures, one test currently omitted\n\n### Future ideas\n\n- `CompletableFuture` implementation\n\n## Other information sources\n\n- [Facebook DataLoader Github repo](https://github.com/facebook/dataloader)\n- [Facebook DataLoader code walkthrough on YouTube](https://youtu.be/OQTnXNCDywA)\n- [Using DataLoader and GraphQL to batch requests](https://github.com/gajus/gajus.com-blog/blob/master/posts/using-dataloader-to-batch-requests/index.md)\n\n## Contributing\n\nAll your feedback and help to improve this project is very welcome. Please create issues for your bugs, ideas and\nenhancement requests, or better yet, contribute directly by creating a PR.\n\nWhen reporting an issue, please add a detailed instruction, and if possible a code snippet or test that can be used\nas a reproducer of your problem.\n\nWhen creating a pull request, please adhere to the Vert.x coding style where possible, and create tests with your\ncode so it keeps providing an excellent test coverage level. PR's without tests may not be accepted unless they only\ndeal with minor changes.\n\n## Acknowledgements\n\nThis library is entirely inspired by the great works of [Lee Byron](https://github.com/leebyron) and\n[Nicholas Schrock](https://github.com/schrockn) from [Facebook](https://www.facebook.com/) whom I like to thank, and\nespecially @leebyron for taking the time and effort to provide 100% coverage on the codebase. A set of tests which\nI also ported.\n\n## Licensing\n\nThis project [vertx-dataloader](https://github.com/engagingspaces/vertx-dataloader) is licensed under the\n[Apache Commons v2.0](https://github.com/engagingspaces/vertx-dataloader/LICENSE) license.\n\nCopyright \u0026copy; 2016 Arnold Schrijver and other\n[contributors](https://github.com/engagingspaces/vertx-dataloader/graphs/contributors)\n","funding_links":[],"categories":["Libraries","Utilities","Implementations"],"sub_categories":["Java Libraries","Java"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fengagingspaces%2Fvertx-dataloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fengagingspaces%2Fvertx-dataloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fengagingspaces%2Fvertx-dataloader/lists"}