https://github.com/gemfire/gemfire-distributed-types
Distributed data type examples for GemFire
https://github.com/gemfire/gemfire-distributed-types
Last synced: 6 months ago
JSON representation
Distributed data type examples for GemFire
- Host: GitHub
- URL: https://github.com/gemfire/gemfire-distributed-types
- Owner: gemfire
- License: other
- Created: 2024-05-15T22:08:53.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2025-10-17T22:16:32.000Z (8 months ago)
- Last Synced: 2025-10-18T22:11:22.087Z (8 months ago)
- Language: Java
- Homepage:
- Size: 344 KB
- Stars: 2
- Watchers: 18
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# GemFire Distributed Types
[](https://central.sonatype.com/artifact/dev.gemfire/gemfire-distributed-types/overview)
[](https://javadoc.io/doc/dev.gemfire/gemfire-distributed-types)
### What is this?
This is a collection of various Java data types that are backed by GemFire, thus making them
naturally distributed and highly available.
These types are intended to be used from GemFire clients to provide concurrent and distributed
access.
### Collection types
- DSet
- DList
- DBlockingQueue
- DCircularQueue
### Concurrency types
- DAtomicLong
- DAtomicReference
- DSemaphore
- DCountDownLatch
- DCounter
### Other
- DSnowflake
### Examples
Almost all the types are created using a `DTypeFactory`:
```java
ClientCache client = new ClientCacheFactory()
.addPoolLocator("localhost", locatorPort)
.create();
DTypeFactory factory = new DTypeFactory(client);
DList list = factory.createDList("myList");
DSet accounts = factory.createDSet("accounts");
DSemaphore semaphore = factory.createDSemaphore("semaphore", 1);
DBlockingQueue queue = factory.createDQueue("queue", 5);
DCircularQueue circular = factory.createDCircularQueue("circular", 100);
```
(`DSnowflake`s are simply created with regular Java instantiation)
### Details
`DSet`s and `DList`s implement the standard Java `Set` and `List` interfaces respectively.
`DBlockingQueue` implements Java's `BlockingDeque` interface and provides a double-ended queue which
provides both head and tail semantics. Note that methods of this class that support timeouts, and
are thus interruptible, can only be interrupted locally. The interrupt 'signal' is not passed on to
the cluster member that is actually performing the operation. This may lead to unexpected results if
not taken into account. For example, if we have a thread trying to take an entry:
```java
Object obj = queue.poll(10, TimeUnit.SECONDS);
```
If this thread is interrupted, the thread executing on the server will continue. If, before the
timeout expires, another thread places an entry into the queue, the `poll`ing thread will retrieve
it which may leave the queue in an unexpected state as far as the client is concerned.
`DCircularQueue` implements the `Queue` interface and provides a first-in first-out queue with a
fixed size that replaces its oldest element if full.
`DSemaphore` provides an implementation with similar semantics to Java's `Semaphore` class.
DSemaphores provide a concurrency type that can be used to co-ordinate between clients running on
different JVMs. Since DSemaphores are highly available, they maintain their state even when
servers crash and clients re-connect to different servers.
`DAtomicLong` provides a counter implementation similar to Java's `AtomicLong` class.
`DAtomicReference` provides an implementation similar to Java's `AtomicReference` class. One
difference to note is that the `compareAndSet` method does not compare on object identity, but
rather on object equality. This is because the identity on the client will obviously be different to
the identity on the server, where the operation is actually being performed. Hence the need to use
equality as a way to compare objects.
`DCountDownLatch` provides an implementation similar to Java's `CountDownLatch`.
`DCounter` provides a counter that is a bit simpler than `DAtomicLong` but with better performance.
The current implementation utilizes GemFire's Delta interface in order to distribute updates to the
value instead of the actual value. This approach avoids the need to ensure write ordering (always a
problem when updates are being generated across a distributed system).
> Note that any methods which can wait (and block) will automatically be retried if the server they
> are connected to crashes or stops. If the particular method semantics also provide a timeout, the
> timeout will be restarted.
### Developing and Deploying
The package can easily be used from either Maven or Gradle:
#### Maven:
```xml
dev.gemfire
gemfire-distributed-types
0.1.4
```
#### Gradle:
```groovy
dependencies {
implementation 'dev.gemfire:gemfire-distributed-types:0.1.4'
}
```
If necessary, instructions to access the dependent GemFire artifacts can be found here -
[gemfire.dev](https://gemfire.dev/quickstart/java/)
In order to deploy the package to a GemFire cluster it should be added as an extension. The
extension bundle can be found under [Releases](https://github.com/gemfire/gemfire-distributed-types/releases).
The `.gfm` file should be added to the `extensions` directory within the GemFire distribution.
Look for a line in the logs that state `Initialized service for GemFire Distributed Types` to
verify that the extension has been found and initialized.
### Building
Building artifacts locally is done using gradle. You will need to set the environment variables
`COMMERCIAL_MAVEN_USERNAME` and `COMMERCIAL_MAVEN_PASSWORD` to the username and access token
required by the Broadcom Maven repository:
```shell
export COMMERCIAL_MAVEN_USERNAME=
export COMMERCIAL_MAVEN_PASSWORD=
./gradlew publishToMavenLocal extensionDistZip
```
This will also create the required `.gfm` file in `build/distributions/`.
### Implementation details
These types are primarily implemented using a partitioned region and function calls. Operations are
captured as lambdas and then routed to the server hosting the primary bucket for the given instance,
where the operation is applied. Each collection implements GemFire's Delta interface which allows
the operation to be sent as a delta change to the secondary server.
The backing region is called `DTYPES`. It is a Partitioned Region with a redundancy of 1 (i.e.
an additional copy of each structure is stored on a different server). Currently, this region is not
persisted and is not user-configurable.
DSemaphores are created with an implicit lease time which is periodically refreshed by a client
pinger thread. If the client crashes, or otherwise disconnects without releasing its permits, the
permits, previously held by the client, will eventually be released. The default lease time for a
semaphore is 30 seconds. Clients ping servers every 10 seconds to indicate their aliveness. These
values can be adjusted by setting the system properties `dtype.dsemaphore.pendingReleaseTime` and
`dtype.client.pingIterval` respectively. Units are in milliseconds.
### Looking for Support
This project is maintained by the VMware Tanzu GemFire R&D team with help from the community.
If you need assistance, support is available with a licensed GemFire deployment. Visit
https://www.broadcom.com/support for additional details.