Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/spotify/simple-bigtable


https://github.com/spotify/simple-bigtable

bigtable client java wrapper

Last synced: about 2 months ago
JSON representation

Awesome Lists containing this project

README 

        
# Simple Bigtable



## Overview



[Cloud Bigtable](https://cloud.google.com/bigtable/docs/) is a datastore supported by Google for storing huge amounts

of data and maintaining very low read latency. The main drawback to using Bigtable is that Google does

not currently have an official asynchronous client. Within Spotify we have been using the RPC client which is

a pain to use. This library aims to fix that by making the most common interactions with Bigtable clean and easy

to use while not preventing you from doing anything you could do with the RPC client.



To import with maven, add this to your pom:



```xml



    com.spotify

    simple-bigtable

    LATEST_RELEASE



```



## Raw RPC Client vs Bigtable Client Comparison



### Using The RPC Client



To give an example of using the base RPC client (which gives the `BigtableSession` object), this is how you would

request a single cell from Bigtable.

```java

String projectId;

String zone;

String cluster;

BigtableSession session;



String fullTableName = String.format("projects/%s/zones/%s/clusters/%s/tables/%s",

        projectId,

        zone,

        cluster,

        "table");



// Could also use a filter chain, but you can't actually set all the filters within the same RowFilter object

// without a merge or chain of some sort

final RowFilter.Builder filter = RowFilter.newBuilder().setFamilyNameRegexFilter("column-family");

filter.mergeFrom(RowFilter.newBuilder().setColumnQualifierRegexFilter(ByteString.copyFromUtf8("column-1")).build());

filter.mergeFrom(RowFilter.newBuilder().setCellsPerColumnLimitFilter(1).build()); // By default it is 1



final ReadRowsRequest readRowsRequest = ReadRowsRequest.newBuilder()

        .setTableName(fullTableName)

        .setRowKey(ByteString.copyFromUtf8("row"))

        .setNumRowsLimit(1)

        .setFilter(filter.build())

        .build();



final ListenableFuture> future = session.getDataClient().readRowsAsync(readRowsRequest);

final ListenableFuture cell = FuturesExtra.syncTransform(future, rows -> {

  // This doesnt actually check if the row, column family, and qualifier exist

  // IndexOutOfBoundsException might be thrown

  return rows.get(0).getFamilies(0).getColumns(0).getCells(0);

});

```



### Bigtable Client



The goal of this client is to let you query what you want with minimal overhead (there should

be no need to create all these filter objects) as well as give you the object you want without

needing to constantly convert a list of rows down to a single cell.  Note that these examples

use a String as a row key.  Bigtable keys are really byte arrays. Strings in this api is just

a convenience.  Under the cover the string "row" is converted to a ByteString.  In reality

you should use byte arrays as keys as that will

be more efficient.

 

Here is the same query as above using this client wrapper.

```java

String projectId;

String zone;

String cluster;

BigtableSession session;



Bigtable bigtable = new Bigtable(session, projectId, zone, cluster);

final ListenableFuture> cell = bigtable.read("table")

    .row("row")

    .column("family:qualifier") // specify both column family and column qualifier separated by colon

    .latestCell()

    .executeAsync();

```



## Performing Reads



The goal of this client is to make the most tedious and common interactions with Bigtable as painless as possible.

 Therefore reading data is an extremely large focus. Here are some examples of reading data.



Get full column family within row

```java

final ListenableFuture> family = bigtable.read("table")

    .row("row")

    .family("family")

    .executeAsync();

```



Get multiple columns within a row (Currently all need to be in the same column family but hopefully that gets fixed)

```java

// Get the entire column

final ListenableFuture> family = bigtable.read("table")

    .row("row")

    .family("family")

    .columnQualifiers(Lists.newArrayList("qualifier-1", "qualifier-2"))

    .executeAsync();



// Get the latest cell in each column

final ListenableFuture> family = bigtable.read("table")

    .row("row")

    .family("family")

    .columnQualifiers(Lists.newArrayList("qualifier1", "qualifier2"))

    .latestCell()

    .executeAsync();

```



Get columns within a single family and within column qualifier range

```java

final ListenableFuture> columns = bigtable.read("table")

    .row("row")

    .family("family")

    .columns()

    .startQualifierInclusive(startBytestring)

    .endQualifierExclusive(endBytestring)

    .executeAsync();

```



Get cells between certain timestamps within a column

```java

final ListenableFuture> cells = bigtable.read("table")

    .row("row")

    .column("family:qualifier")

    .cells()

    .startTimestampMicros(someTimestamp)

    .endTimestampMicros(someLatertimestamp)

    .executeAsync();

```



Get the latest cell of a certain value within a column

```java

final ListenableFuture> cells = bigtable.read("table")

    .row("row")

    .column("family:qualifier")

    .cells()

    .startValueInclusive(myValueByteString)

    .endValueInclusive(myValueByteString)

    .latest()

    .executeAsync();

```



Get the latest cell of a between 2 timestamps within a column for multiple rows

```java

final ListenableFuture> cells = bigtable.read("table")

    .rows(ImmutableSet.of("row1", "row2"))

    .column("family:qualifier")

    .cells()

    .startTimestampMicros(someTimestamp)

    .endTimestampMicros(someLatertimestamp)

    .latest()

    .executeAsync();

```



Get the multiple column families and column qualifiers (will match all combinations)

```java

final ListenableFuture> cells = bigtable.read("table")

    .row("row")

    .families(ImmutableSet.of("family1, family2"))

    .columnQualifiers(ImmutableSet.of("qualifier1", "qualifier2")

    .cells()

    .startTimestampMicros(someTimestamp)

    .endTimestampMicros(someLatertimestamp)

    .latest()

    .executeAsync();

```



Get all rows between different ranges or with certain specific keys

(these functions add rows to the row set, instead of filtering)

```java

final ListenableFuture> rows = bigtable.read("table")

    .rows()

    .addRowRangeOpen(myStartKeyOpen, myEndKeyOpen) // add an exclusive range

    .addRowRangeClosed(myStartKeyClosed, myEndKeyClosed) // add an inclusive range

    .addKeys(extraKeys) // add some keys you always want

    .executeAsync();

```

Note that currently there is no half open, half closed range.



## Other Operations



The client supports other Bigtable operations as well, with hopefully the rest of all possible operations coming

soon.



### Mutations (Writes, Deletions)



Mutations are performed on the row level with many mutations possible within a single call. Mutations include

writing new values as well as deleting a column, column family, or an entire row and all data help in each.



Write a new cell within a column

```java

final ListenableFuture mutation = bigtable.mutateRow("table", "row")

    .write("family:qualifier", ByteString.copyFromUtf8("value"))

    .executeAync()

```



Perform multiple writes in different columns setting an explicit timestamp on some

```java

final ListenableFuture mutation = bigtable.mutateRow("table", "row")

    .write("family:qualifier", ByteString.copyFromUtf8("value-1"), timestampMicros)

    .write("family", "qualifier", ByteString.copyFromUtf8("value-2"))

    .executeAync()

```



Delete a column and then write to the same column

```java

final Empty mutation = bigtable.mutateRow("table", "row")

    .deleteColumn("family:qualifier")

    .write("family:qualifier", ByteString.copyFromUtf8("brand-new-value"))

    .execute()

```



### ReadModifyWrite (Atomically Update or Append To A Column)



ReadModifyWrite is useful for either incrementing the latest cell within a column by a long or appending bytes

to the value. If the column is empty, is will write a new value. Once again this operation is on the row level

with multiple ReadModifyWrites possible in a single request.



Increment a couple counter columns and append a value to another

```java

bigtable.readModifyWrite("table", "row")

    .read("request-numbers:number-1")

    .thenIncrementAmount(1L)

    .read("request-numbers:number-2")

    .thenIncrementAmount(5L)

    .read("family:values")

    .thenAppendValue(ByteString.copyFromUtf8("new-value"))

    .executeAsync();

```



### SampleRowKeys



Sample some row keys from a table.

```java

final List sampleRowKeys = bigtable.sampleRowKeys("table").execute();

```



### CheckAndMutateRow - NOT YET IMPLEMENTED



Perform a read and a set of mutations depending on whether the read returns data. This is not yet implemented but

here are some ideas on how this operation might be implemented in the future.



Have the check specified like a read, then allow mutations to be added.

```java

bigtable.checkAndMutateRow("table", "row")

    .column("family:qualifier")

    .cells()

    .endTimestampMicros(timestamp)

    .ifExists()

    .deleteColumn("family:qualifier")

    .write("family:qualifier", "had-data")

    .ifDoesNotExist()

    .write("family:qualifier", "did-not-have-data")

    .executeAsync();

```



Pass in Bigtable protobuf objects, kind of against the purpose of the library but keeps things simple.

```java

bigtable.checkAndMutateRow("table", "row")

    .rowFilter(someRowFilter)

    .ifExists(someMutation)

    .ifExists(someOtherMutation)

    .ifDoesNotExist(someOtherMutation)

    .executeAsync();

```

Pull requests with other ideas are encouraged.



### Table and Cluster Admin Operations - NOT YET IMPLEMENTED



It is unclear whether there is a need this wrapper to provide the admin operations, though it would be pretty easy

to include. 



## How to Release

- A local build and deploy is the easiest way to make a release at this point.  For setup follow instructions in:

   [scio instructions](https://github.com/spotify/scio/wiki/How-to-Release#prerequisites) modified to work with maven.

   For credentials to push to [sonatype](https://oss.sonatype.org/content/repositories/snapshots/com/spotify/simple-bigtable/), create an access token with your sonatype login and place the access token

   in your maven settings.xml as in:



```xml

    

      

      ossrh

      access-token-name 

      access-token-password

    

```



-   `mvn clean javadoc:jar source:jar deploy` should do the rest.



-   Release deployments (from mvn deploy) will be put into a staging area at sonatype.  Read the following to release the staged deployment [how to release](http://central.sonatype.org/pages/releasing-the-deployment.html) and [maven](http://central.sonatype.org/pages/apache-maven.html).



-    With effort we could get automatic deployments via travis.  The travis build console is here:

     [travis.org](https://travis-ci.org/spotify/simple-bigtable).  You'll need access to travis.org (not .com) to access

     the builds.

   

## Open Problems and Questions



- One problem is that currently it is not really possible to do queries for nested lists. For example there really is

 no way to do a ColumnRange within a RowRange or request multiple columns within different column families. 

 Something like `BigtableColumnsWithinFamilies` could be added where it keeps track of needing different column

 families but that is confusing. Another option is adding the filtering methods to every Read object which would also

 be super confusing.



## Code of conduct

This project adheres to the [Open Code of Conduct][code-of-conduct]. By participating, you are expected to honor this code.



[code-of-conduct]: https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md