https://github.com/gaiandb/gaiandb

A light-weight data-federation technology built on Apache Derby. Gaian accesses and transforms data securely "at the edge" and can discover other instances of itself to form a scalable data network.
https://github.com/gaiandb/gaiandb

Last synced: 30 days ago
JSON representation

A light-weight data-federation technology built on Apache Derby. Gaian accesses and transforms data securely "at the edge" and can discover other instances of itself to form a scalable data network.

Host: GitHub
URL: https://github.com/gaiandb/gaiandb
Owner: gaiandb
Created: 2015-07-16T14:56:07.000Z (almost 11 years ago)
Default Branch: master
Last Pushed: 2023-01-12T08:11:56.000Z (over 3 years ago)
Last Synced: 2025-08-09T12:22:42.581Z (11 months ago)
Language: Java
Homepage:
Size: 39.1 MB
Stars: 73
Watchers: 17
Forks: 35
Open Issues: 21
Metadata Files:
- Readme: README.md
- Contributing: Contributing.md
- License: LICENSE/LICENSE

Awesome Lists containing this project

awesome-java - GaianDB

README

          # GaianDB

A light-weight data-federation technology built on Apache Derby. Gaian accesses and transforms data securely "at the edge" and can discover other instances of itself to form a scalable data network.

Readme.html taken from build/GAIANDB_V2.1.8_20160523.zip





Contents



What is the Gaian Database ?




Key benefits - what does Gaian give you ?




What's new in version 2






Installation and First Steps

	

	Upgrading Gaian

	Installation Prerequisites

	Supported Operating Systems

	First Steps

	Verify the installation

	

Quick table federation examples (matching a Logical Table to a Physical One)

		

		Federate an RDBMS table

		Federate an existing CSV File

		Federate an existing Excel spreadsheet

		

	

	Further worked examples

	








GaianDB Usage

	

	GaianDB node usage

	Launching multiple GaianDB nodes

	Launching a GaianDB node from a separate directory

	How to specify a location for configuration and log files and the physical database folder

	How to manage multiple GaianDB node executions with multiple working directories

	How to initialise a GaianDB node with User defined Derby objects like Tables, views, procedures,functions

	How to integrate GaianDB into a wrapping class or framework: startup, start detection, kill

	








Client Usage

	

	Command line processor options with 'queryDerby'

	Issuing Distributed SQL Queries

	Distributed Sub-queries or Pushing query processing to each node

	Listing server warnings

	Dashboard Graphical User Interface

	








Configuration

	

	

Using the Configuration File

		

		Configuration file: Properties for Logical Tables, their Data Sources and associated Connections

		Configuration file: Global System properties and identifiers

		

	

	Using the management API to apply configuration updates

	Pluralized data sources

	In-Memory tables and indexes on their columns

	Constant Column definitions

	Using the LISTQUERIES and CANCELQUERY procedures as well as the GDB_TIMEOUT and GDB_WID features

	Connection maintenance configuration in intermittent/high latency/low bandwidth networks

	Hard-wiring connections between GaianDB nodes (incl setting up gateways)

	Text file federation configuration options (e.g: how to specify different record and field separators)

	








Security

	

	

Communication encryption using SSL (Secure Sockets Layer)

		

		Server Configuration for SSL

		Client Setup for SSL

		

	

	User authentication

	User access restriction to chosen databases

	Password scrambling

	








Advanced Features

	

	Special SQL Query Options (with_provenance, explain, maxDepth, ...)

	Provenance Columns: Special columns identifying origin of data rows

	Explain Queries: Getting and showing the route of a query

	Specifying advanced table and column mappings

	SQL Query as a Logical Table Data Source

	Message Storer: Message Broker Integration (e.g. use with sensors)

	

Custom VTIs

		

		IBM Content Analytics Restful Interface (ICAREST)

		Spatial Query

		

	

	






Supported RDBMS Data sources




Supported SQL Types for Logical Tables Definitions




Supported Number of Logical Tables




Error Messages




FAQ & Troubleshooting




Features




Authors/Contacts




What is the Gaian Database ?


The Gaian Database - or GaianDB for short - is a lightweight (<4MB) dynamically distributed federated database (DDFD) engine based on Apache Derby 10.x. It provides a single centralized view over multiple, heterogeneous back-end data sources (e.g. RDBMS dbs, files, text indexes, spreadsheets, etc.) using an extensible logical table abstraction layer. However, its principal feature is its ability to automatically discover and federate other GaianDB instances (federating nodes) such that a whole network of these can be automatically formed.




   

     

          

	  

     

          

	  

   

     Fig. 1 - GaianDB high level architecture

     Fig. 2 - GaianDB connectivity model

    

GaianDB advocates a flexible 'store locally, query anywhere' (SLQA) paradigm, which means the data remains where it is, but you have a centralized view over which you can query. Each data source should have its data managed autonomously and applications can choose which sections of it to make available to the GaianDB network. For this reason distributed insert/update/delete operations are not currently supported.

Queries can be injected at any node and are routed effectively around the network using a 'biologically inspired' connectivity model that strives to minimise query time and maximise efficiency (by minimising network diameter and maximising connections to the fittest nodes).

Architecturally, GaianDB sits underneath Derby, and is invoked by Derby to process any GaianDB-specific SQL. To query GaianDB you typically identify the 'GaianTable' class (which references a logical table, similar to a view definition), or the 'GaianQuery' class (which references an embedded subquery), either of which will retrieve federated data from every GaianDB node in a connected network. Other GaianDB SQL usage may be invocations of one of the stored procedures or functions that make up GaianDB's system management API. These can be used to define Logical Tables and their federated Data Sources, relational database connections, user-defined Gaian Connections (for nodes that couldn't be discovered), or any of GaianDB's general system properties individually.




The distributed, federated and store-local nature of GaianDB makes it extremely scalable and suitable for a wide variety of deployments and scenarios. Also, GaianDB is 100% Java and so it can run in a multitude of places; anywhere from small devices, like mobile phones, to large enterprise systems; easily integrating between all of these.

GaianDB has already been used in the context of complex text analytics applications, performing distributed semantic join queries, and drawn the attention of significant customers in the military space.

As a DDFD technology, its positioning is shown below:





   

     

     

   

     Fig. 3 - GaianDB positioning

     




GaianDB also offers a wide range of other features such as: Dynamic re-configuration; an 'Explain' network-route query option; In-Memory tables and indexing; Message Broker integration; contextual meta-data.

For more information on GaianDB, take a look at this recently (2008) published paper:

A Dynamic Distributed Federated Database





Key benefits - what does Gaian give you ?




Gaian federation technology provides a "data virtualisation" layer - with many beneficial aspects:





Reduced Maintenance



	Using simplified abstract data model as opposed to full physical data model for each back-end

	Views do not need to be created in back-end databases

	Data sets from separate providers can be joined with SQL + configuration only (no need for code)

	Many types of data-sources are supported out-of-the-box (including RDBMS tables, structured text files or XL spreadsheets)

	Other data-sources can be exposed via a Gaian Java plugin interface wrapper.

	Simplified model for Migrating/swapping underlying data-sources - only requires configuration updates.



In-line SQL transformation of data records



	Ad-hoc simple data cleansing

	Data matching



Real-time data



	Gaian provides access to current endpoint data - as opposed to "old" data exported by data owner.



Data Ownership



	Gaian nodes can be attributed to each data owner.

	Data owners provide/deny access to their data as they wish - Graphical/intuitive tools for this are in development.

	Intermediary data stores for published data are no longer required.



Distributed Access + Scalability



	A plurality of interconnected Gaian nodes provides the ability to connect and query through any node.

	Nodes can be configured differently to allow access to more or less data structures.

	Gaian networks can provide scalability by attributing distinct data set partitions to different Gaian instances.



Distributed Processing



	Distributed queries can result in parallel processing, e.g. invoking procedure calls (can also be used to implement map/reduce).

	Processing functions can be deployed dynamically (in a JAR file). Gaian does not need restarting.



Provenance



	Gaian provides provenance information for every data record.

	This facilitates development of Governance aspects which can also be applied at the Gaian layer.



Policy



	Access rules can be implemented to cover access to all data providers (incl. files etc) in one place.

	Policy functions can be deployed dynamically (in a JAR file). Gaian does not need to be restarted.



Caching



	Gaian can act as a cache location for pre-processed results, faster access, etc

	Gaian can also host surrogate key tables for linking different datasets together (e.g. same locations expressed in different languages)








What's new in version 2



	New features

	

    Updated Derby pre-req to version 10.8.3.1.1505077

    Query timeout and cancellation capabilities

    Support for batched record filtering for access to high-latency authentication systems.

    Extended ICAREST + SpatialQuery VTIs such that they call out to the policy plugin to enforce required filtering.

    New TRANSIENT option ** Deprecated in favour of PLURALIZED ** allows in-memory data sources definitions, thus speeding up their registration and avoiding having to remove them on start-up.

    Improved capability to integrate Gaian in a wrapping JVM process (e.g. OSGi framework) - with start detection, termination handling and workspace configuration.

    Allowed sub-queries to include stored procedure calls and CRUD operations.

    New capability to target sub-queries at specific nodes.

    Added expiry capability to INMEMORY caching option.

    Improved core capability to distinguish ordinary from delimited identifiers; and eliminated need for column naming in sub-queries.

    Added capability to connect and authenticate using a separate user database or schema whilst retaining access to all GaianDB logical tables and procedures/functions.

    New query path visualisation in dashboard - ability to discover loaded logical tables and paths to them.

    Pluralized data sources for accessing multiple data source end-points through a single Gaian data source wrapper - e.g. a set of files in a folder.

    Policy framework improvements: Now passes cluster ID and session user ID to allow more control over user and query access.

    New API batch update capability with procedure: setconfigproperties().

    Capability to restrict user access to designated derby databases, see shipped derby.properties.

    Auto-resoltuion of schemas and column-mappings for custom VTIs

    Generic pooling of all data source wrappers - now includes custom VTIs

	

	Bug fixes

	

    Fixed and improved reliability of Excel data source wrapper

    Improved persisted caching behaviour when handling expensive join queries

    Simplified logical table reload when nodes connect or disconnect

    Improved policy plugin framework to allow GaianDB to call out to the plugin at any stage in the query life cycle in future.

    Reduced GaianNode startup time and improved reliability of refresh of logical table views and stored procedures.

    Fixed propagation of query meta-data information (such as user credentials) so it is only sent with GaianDB queries and can now reach endpoint data sources via sub-queries.

    Improved robustness of discovery capability and fixed local discovery for MAC OS X.

    Made policy framework generically extensible without impacting on backwards compatibility in future.

    Fixed/improved resilience to concurrent configuration updates, reducing risk of corruption to the configuration file.

    Improved reliability and cross-platform capability of start-up and kill scripts.

    Fixed intermittent hang and deadlock issues.

    Increased size of all VARCHAR input parameters of APIs to maximum size 32672 chars.

    Improved resolution of file paths on Windows for structured text file data sources.

    Fixed OutOfMemoryError related to long running queries - by ignoring hanging suspects which are not inter-node connections (as these cannot be polled anyway), clearing executing data sources after a query. Also added logging for memory reporting.

    Fixed intermittent GaianDB inter-node connection hiccups during discovery

    Fixed run status when the main GaianNode thread completes such that it can subsequently be re-started

    Fixed inconsistent state after a REMOVELT() causing issues with subsequent SETDSXXX().

    Fixed GaianNode shutdown() to complete despite issues, removing use of Thread.isAlive(), and only when Derby has shut down - thus allowing a clean restart.

    Removed use of System.gc() calls when releasing cached rows for JOINs.

    Fixed logical table table creation issues with full synchronization of SETXXX() API calls.

    Performance improvement with logical table API updates by avoiding checks on unaffected tables.

    Resolved issues with IS [NOT] NULL construct against Gaian VTIs and Table Functions.

    Fixed issue whereby a data source pointing to an RDBMS table expression would not load after being changed (throwing a StringIndexOutOfBoundsException).

    Improved resilience to issues with logical table view reloads such that subsequent view reloads are not impacted.

    Fixed condition whereby GaianDB at rest having Oracle data source consumes its cursors until failure.

    Cleaner error reporting when GAIANDB.jar cannot be found on startup.

    Changed condition for DROP statements which raised silent derby.log ERRORs, e.g. at startup.

    Fixed issue with CACHE tables used for JOIN processing when an logical table definition contains a LONGVARCHAR, by normalising types to Derby syntax.

    Avoid Gaian hang when connection to a peer node is abruptly lost, e.g. termination of a peer node's VM

    Allow special characters in column names    

	

	Prototyped - coming soon:

	

    UDP Driver for inter-node connectivity over UDP

    'Lite' Driver to run independantly from Derby (with limited SQL capability)

    Client node discovery capability to allow load balancing, e.g. from WAS

    New deployment capability to IBM Bluemix cloud

    Generic access to web services

    MongoDB VTI

    Accumulo VTI

	






Version 2.1.4



	New features

	

    Dynamic class-loading of newly deployed JAR libraries containing JDBC drivers, policy code, and data source connectors (VTIs).

    Policy framework improvements: Now receives full-sub-query for pass-through queries, and ResultSetMetaData is supplemented with table names of queried columns.

    New capability to enable Derby's SSL encryption for client-server and inter-node connections.

    New procedure RUNSQL() to run SQL directly against any specified connection ID or JDBC properties specified directly.

    New capability to specify <GAIAN_WORKSPACE> tag in _ARGS property of a Text file DS.

    New procedures LISTTHREADS() to list Gaian's JVM threads; and LISTENV() to list Gaian's system environment properties.

	

	Bug fixes

	

	Fixed condition leading to a Derby NullPointerException occurring with 4-way (or higher complexity) self-joins.

    Fixed queryDerby behaviour (e.g. with delimiters, comments) to be consistent across all modes: file, query-list, interactive

    Fixed lookup of control files for structured text file data sources (e.g. CSV)

    RDBMS types normalisation for Oracle data types - particularly INTERVAL type and TIMESTAMP types involving a TIME ZONE.

    Resolved OutOfMemoryError occurring from accumulated stale objects after repeated runs of a query.	

	

	Prototyping in progress - coming soon:

	

	Windows Service for Gaian - with documentation for registry changes required

    GaianServlet for deploying Gaian as a webapp

    Gaian in Bluemix

    GenericWS connector hardening for access to a web service as a data source

	






Installation and First Steps


Upgrade Gaian

If you are already using GaianDB, and want to upgrade please refer to the upgrade instructions.


Installation Prerequisites



	Java JRE version: IBM JRE 1.7.0 or above. (Oracle JRE 1.7.0 should work; GNU versions of Java have caused problems). Make sure that the 'java' executable is in your PATH.

	


	Any RDBMS JDBC drivers required to connect to and federate your Relational Database tables. These will often

	be included in the installation package for the RDBMS.

	


	To make a new JDBC driver available to GaianDB, add the JAR file that contains the driver class to the Java CLASSPATH

	variable before starting GaianDB (for convenience, the Java CLASSPATH is defined in 'launchGaianServer.bat(/.sh)' batch and script files).

	


	GaianDB only supports the official JDBC drivers provided by the Relational Database providers listed below:

	

			

			

				Name

				Jar file(s)

				JDBC driver class

				URL connection template

			

			

		

			

				DB2				

				db2jcc.jar, db2jcc_licence_cu.jar

				com.ibm.db2.jcc.DB2Driver

				jdbc:db2://<server>:<port50000>/<database>

			

			

				SQLServer

				sqljdbc4.jar

				com.microsoft.sqlserver.jdbc.SQLServerDriver

				jdbc:sqlserver://<server>:<port1433>;DatabaseName=<database>

			

			

				MySQL

				mysql-connector-java-5.1.7-bin.jar

				com.mysql.jdbc.Driver

				jdbc:mysql://<server>:<port3306>/<database>

			

			

				Oracle

				ojdbc14.jar or ojdbc6.jar

				oracle.jdbc.OracleDriver

				jdbc:oracle:thin:@<server>:<port3306>/<database>

			

		

	

	

	Knowledge of SQL. All SQL conventions that apply to Apache Derby apply to the IBM Gaian Database. See Apache Derby's Reference Manual for more information.






Supported Operating Systems

GaianDB has primarily been tested on RHEL, Ubuntu, Windows and MAC OS X. The documentation uses the term 'Unix' to refer to Linux and MAC OS X. 

   GaianDB should work on other Unix systems (AIX, HP-UX, Solaris, etc ..) and was demonstrated running on many virtualization engines (VMWare,KVM,Xen) and Cloud software (IBM SmartCloud Provisioning (SCP), Amazon EC2, OpenStack), but no testing has been done.

Windows and Unix scripts are provided in the installation for starting a GaianDB server ('launchGaianServer.bat' & 'launchGaianServer.sh') and running the Command Line Processor ('queryDerby.bat' & 'queryDerby.sh').


Note: All paths, whether on Windows or Unix, should use forward slashes (/) and not backslashes (\).

When launching a GaianDB node, the system must be able to resolve its own hostname to an IP address. You can check this by:



	You can find out what the server's hostname is by running the

	'hostname' command in a command prompt.

	Test whether the system can resolve its own IP address using

	the 'ping <hostname>' command (valid for Unix and Windows).



If it cannot:



Check whether your firewall is blocking the requests. For more information on configuring firewalls for use with GaianDB, please refer to the FAQ & Troubleshooting section. 

Check your network settings:



	For Windows systems:

		

			Check your

	network connection's IP configuration properties (Control Panel->Network Connections->Select a connection->

	right click, Properties->Internet protocol). 

		

	

    For Unix systems:

		

			Check that the file /etc/hosts has a line (otherwise add one)

			in the format: <ipaddress> < hostname>

			<hostname.domain>

		

	

	  For virtualization software and systems:

		

			Check that the network layer exposes the proper IP addresses and allows the IP and UDP traffic through as described above. 

			    You may be required to use NAT instead of BRIDGE for instance. 

		

	







First Steps

Install Gaian

To 'install' GaianDB, simply unzip the zip install file to a directory of your choice. (For instance 'C:\GaianDB' on Windows or '/home/myuser/GaianDB' on Unix)

Launch a GaianDB Node

You are now ready to launch a GaianDB node. To do so,  run 'launchGaianServer.bat' on Windows or './launchGaianServer.sh' on Unix.

By default, the GaianDB node will start a Derby server listening on port 6414 and use the properties file 'gaiandb_config.properties' in the local directory.

Examine the default configuration





To see the default startup settings for a GaianDB node, take a look at the USAGE statement. You can see this by running the launch scripts and passing in any unrecognised argument, e.g: '-?' , '-help' or '-usage'. For example, from a command prompt, by running 'launchGaianServer.bat -help' on Windows or './launchGaianServer.sh -help' on Unix.  





Have a look at the configuration file 'gaiandb_config.properties'. This defines logical tables (LT0, LT1, ..), and their federated physical data sources (LT0_DS0, LT0_DS1, ..). You can find out more about these configuration settings here.





Verify the installation

To quickly test that GaianDB is up and running:

Run 'testGaianDB.bat' on Windows or './testGaianDB.sh' on Unix. This launches a sample query against the sample LT0 logical table, which should retrieve & display the records from the CSV flat file found at: '<GaianDB Install Directory>/csvtestfiles/datafile.dat'.


You can also run other queries against GaianDB using the Command Line Processor or Dashboard Graphical User Interface client utilities.

For example, you can use to the Command Line Processor to run 'select' commands or run some of the stored procedures the GaianDB provides to verify the installation is correct, as follows:



	

		Run 'queryDerby.bat' on Windows or './queryDerby.sh' on Unix.

		
Once the the Command Line Processor is ready to receive commands you will see: sql>

	




	

		To query a GaianDB logical table, use the syntax: 'select xxx from LT0 where ...' where LT0 is the name of a logical table defined in the 'gaiandb_config.properties'.

		
For example you can type:select * from LT0

	




	To execute the stored procedure API calls use the following:

		

		

call listspfs()
- lists all defined stored procedures and functions.

		

call listlts()
- lists all defined Logical Tables.

		

call listds() 
- lists all Data Sources federated by all logical tables.

		

call listrdbc()
- lists all active (i.e. that have an attached data source) relational database connections

		

	






Quick table federation examples (matching a Logical Table to a Physical One)




	Federate an RDBMS table

	Federate an existing CSV File

	Federate an existing Excel spreadsheet



For each example, first start a GaianDB node (see First Steps).

Then, when the server has started, run 'queryDerby.bat' on Windows or './queryDerby.sh' on Unix to start a Command Line Processor connected to the GaianDB server.

Note: You can also use the Dashboard Graphical User Interface client utility to perform these examples.

Federate an RDBMS table



In this quick example we will show you how to federate an existing RDBMS table using GaianDB. For this we will use the DB2 sample table 'employee'; please refer to the DB2 documentation for how to install the sample tables.





List all the active Relational Database Connections to see the current state of the GaianDB node:


sql>

sql> call listrdbc()

call listrdbc()

============================================================================================================================================

GDB_NODE        |CID             |CDRIVER                                       |CURL                                 |CUSR                |

============================================================================================================================================

L3R3844         |LOCALDERBY      |org.apache.derby.jdbc.EmbeddedDriver          |jdbc:derby:gaiandb;create=true       |gaiandb             |

L3R3844         |mysql5          |com.mysql.jdbc.Driver                         |jdbc:mysql://localhost:3306/test     |root                |

============================================================================================================================================

Fetched 2 rows. Total Time: 47ms (Execution Time: 47ms)

sql>






Create a new Relational Database Connection.


To do this we use the GaianDB stored procedure setrdbc(...) with a connection id of 'db2conn' and the driver & url as shown below (unless your DB2 instance is running on a different machine or port). Enter your own user name and password for DB2 in place of the last 2 arguments.


sql>

sql> call setrdbc('db2conn', 'com.ibm.db2.jcc.DB2Driver', 'jdbc:db2://localhost:50000/sample', 'DavidVyvyan', '???????')

call setrdbc('db2conn', 'com.ibm.db2.jcc.DB2Driver', 'jdbc:db2://localhost:50000/sample', 'DavidVyvyan', '???????')

sql>






Setup a Logical Table that includes all columns of the 'employee' table and federates it.


To do this we use the GaianDB stored procedure setltforrdbtable(<New Logical Table Name>, <Database Connection>, <Selection Predicate>). For example:


sql> call setltforrdbtable('db2employee', 'db2conn', 'employee')

call setltforrdbtable('db2employee', 'db2conn', 'employee')

sql>



Where:



'db2employee' is the new logical table name.

'db2conn' designates the RDBMS connection we just created.

'employee' matches the entire 'employee' table.



NOTE: The DB2 driver classes must be on the CLASSPATH in launchGaianServer.bat.



 
Query 
To do 
 
sql> select * from db2employee 
select * from db2employee 
============= 
EMPNO |FIRSTNME 
============= 
000010|CHRISTINE   |I 
000020|MICHAEL     |L 
000030|SALLY 
000050|JOHN 
000060|IRVING 
000070|EVA 
000090|EILEEN 
000100|THEODORE    |Q 
000110|VINCENZO    |G 
000120|SEAN 
000130|DELORES     |M 
000140|HEATHER     |A 
000150|BRUCE 
000160|ELIZABETH   |R 
000170|MASATOSHI   |J 
000180|MARILYN     |S 
000190|JAMES 
000200|DAVID 
000210|WILLIAM     |T 
000220|JENNIFER    |K 
000230|JAMES 
000240|SALVATORE   |M 
000250|DANIEL 
000260|SYBIL 
000270|MARIA 
000280|ETHEL 
000290|JOHN 
000300|PHILIP 
000310|MAUDE 
000320|RAMLAL 
000330|WING 
000340|JASON 
200010|DIAN 
200120|GREG 
200140|KIM 
200170|KIYOSHI     | 
200220|REBA 
200240|ROBERT 
200280|EILEEN 
200310|MICHELLE    |F 
200330|HELENA      | 
200340|ROY 
============= 
Fetched

the new Logical Table.
 this you can simply issue a standard SQL SELECT query on the table 'db2employee'. For example:
 =========================================================================================================================================== |MIDINIT|LASTNAME       |WORKDEPT|PHONENO|HIREDATE  |JOB     |EDLEVEL|SEX|BIRTHDATE |SALARY     |BONUS      |COMM       |NODEINDEX  | =========================================================================================================================================== |HAAS           |A00     |3978   |1995-01-01|PRES    |     18|F  |1963-08-24|  152750.00|    1000.00|    4220.00|          1| |THOMPSON       |B01     |3476   |2003-10-10|MANAGER |     18|M  |1978-02-02|   94250.00|     800.00|    3300.00|          1| |A      |KWAN           |C01     |4738   |2005-04-05|MANAGER |     20|F  |1971-05-11|   98250.00|     800.00|    3060.00|          1| |B      |GEYER          |E01     |6789   |1979-08-17|MANAGER |     16|M  |1955-09-15|   80175.00|     800.00|    3214.00|          1| |F      |STERN          |D11     |6423   |2003-09-14|MANAGER |     16|M  |1975-07-07|   72250.00|     500.00|    2580.00|          1| |D      |PULASKI        |D21     |7831   |2005-09-30|MANAGER |     16|F  |2003-05-26|   96170.00|     700.00|    2893.00|          1| |W      |HENDERSON      |E11     |5498   |2000-08-15|MANAGER |     16|F  |1971-05-15|   89750.00|     600.00|    2380.00|          1| |SPENSER        |E21     |0972   |2000-06-19|MANAGER |     14|M  |1980-12-18|   86150.00|     500.00|    2092.00|          1| |LUCCHESSI      |A00     |3490   |1988-05-16|SALESREP|     19|M  |1959-11-05|   66500.00|     900.00|    3720.00|          1| |       |O'CONNELL      |A00     |2167   |1993-12-05|CLERK   |     14|M  |1972-10-18|   49250.00|     600.00|    2340.00|          1| |QUINTANA       |C01     |4578   |2001-07-28|ANALYST |     16|F  |1955-09-15|   73800.00|     500.00|    1904.00|          1| |NICHOLLS       |C01     |1793   |2006-12-15|ANALYST |     18|F  |1976-01-19|   68420.00|     600.00|    2274.00|          1| |       |ADAMSON        |D11     |4510   |2002-02-12|DESIGNER|     16|M  |1977-05-17|   55280.00|     500.00|    2022.00|          1| |PIANKA         |D11     |3782   |2006-10-11|DESIGNER|     17|F  |1980-04-12|   62250.00|     400.00|    1780.00|          1| |YOSHIMURA      |D11     |2890   |1999-09-15|DESIGNER|     16|M  |1981-01-05|   44680.00|     500.00|    1974.00|          1| |SCOUTTEN       |D11     |1682   |2003-07-07|DESIGNER|     17|F  |1979-02-21|   51340.00|     500.00|    1707.00|          1| |H      |WALKER         |D11     |2986   |2004-07-26|DESIGNER|     16|M  |1982-06-25|   50450.00|     400.00|    1636.00|          1| |       |BROWN          |D11     |4501   |2002-03-03|DESIGNER|     16|M  |1971-05-29|   57740.00|     600.00|    2217.00|          1| |JONES          |D11     |0942   |1998-04-11|DESIGNER|     17|M  |2003-02-23|   68270.00|     400.00|    1462.00|          1| |LUTZ           |D11     |0672   |1998-08-29|DESIGNER|     18|F  |1978-03-19|   49840.00|     600.00|    2387.00|          1| |J      |JEFFERSON      |D21     |2094   |1996-11-21|CLERK   |     14|M  |1980-05-30|   42180.00|     400.00|    1774.00|          1| |MARINO         |D21     |3780   |2004-12-05|CLERK   |     17|M  |2002-03-31|   48760.00|     600.00|    2301.00|          1| |S      |SMITH          |D21     |0961   |1999-10-30|CLERK   |     15|M  |1969-11-12|   49180.00|     400.00|    1534.00|          1| |P      |JOHNSON        |D21     |8953   |2005-09-11|CLERK   |     16|F  |1976-10-05|   47250.00|     300.00|    1380.00|          1| |L      |PEREZ          |D21     |9001   |2006-09-30|CLERK   |     15|F  |2003-05-26|   37380.00|     500.00|    2190.00|          1| |R      |SCHNEIDER      |E11     |8997   |1997-03-24|OPERATOR|     17|F  |1976-03-28|   36250.00|     500.00|    2100.00|          1| |R      |PARKER         |E11     |4502   |2006-05-30|OPERATOR|     12|M  |1985-07-09|   35340.00|     300.00|    1227.00|          1| |X      |SMITH          |E11     |2095   |2002-06-19|OPERATOR|     14|M  |1976-10-27|   37750.00|     400.00|    1420.00|          1| |F      |SETRIGHT       |E11     |3332   |1994-09-12|OPERATOR|     12|F  |1961-04-21|   35900.00|     300.00|    1272.00|          1| |V      |MEHTA          |E21     |9990   |1995-07-07|FIELDREP|     16|M  |1962-08-11|   39950.00|     400.00|    1596.00|          1| |       |LEE            |E21     |2103   |2006-02-23|FIELDREP|     14|M  |1971-07-18|   45370.00|     500.00|    2030.00|          1| |R      |GOUNOT         |E21     |5698   |1977-05-05|FIELDREP|     16|M  |1956-05-17|   43840.00|     500.00|    1907.00|          1| |J      |HEMMINGER      |A00     |3978   |1995-01-01|SALESREP|     18|F  |1973-08-14|   46500.00|    1000.00|    4220.00|          1| |       |ORLANDO        |A00     |2167   |2002-05-05|CLERK   |     14|M  |1972-10-18|   39250.00|     600.00|    2340.00|          1| |N      |NATZ           |C01     |1793   |2006-12-15|ANALYST |     18|F  |1976-01-19|   68420.00|     600.00|    2274.00|          1| |YAMAMOTO       |D11     |2890   |2005-09-15|DESIGNER|     16|M  |1981-01-05|   64680.00|     500.00|    1974.00|          1| |K      |JOHN           |D11     |0672   |2005-08-29|DESIGNER|     18|F  |1978-03-19|   69840.00|     600.00|    2387.00|          1| |M      |MONTEVERDE     |D21     |3780   |2004-12-05|CLERK   |     17|M  |1984-03-31|   37760.00|     600.00|    2301.00|          1| |R      |SCHWARTZ       |E11     |8997   |1997-03-24|OPERATOR|     17|F  |1966-03-28|   46250.00|     500.00|    2100.00|          1| |SPRINGER       |E11     |3332   |1994-09-12|OPERATOR|     12|F  |1961-04-21|   35900.00|     300.00|    1272.00|          1| |WONG           |E21     |2103   |2006-02-23|FIELDREP|     14|F  |1971-07-18|   35370.00|     500.00|    2030.00|          1| |R      |ALONZO         |E21     |5698   |1997-07-05|FIELDREP|     16|M  |1956-05-17|   31840.00|     500.00|    1907.00|          1| =========================================================================================================================================== 42 rows. Total Time: 219ms (Execution Time: 172ms)

sql>

This contains all columns from the physical table 'employee' and also an example constant column 'NODEINDEX' to identify which GaianDB node the information is from. For more info on constant columns, refer to this section.

List the active RDBMS connections and data sources to verify updates.

To do this we use two GaianDB stored procedures:

listrdbc() - this shows the active RDBMS connections of every connected GaianDB node (the only node in the example below is L3R3844).


sql>

sql> call listrdbc()

call listrdbc()

============================================================================================================================================

GDB_NODE        |CID             |CDRIVER                                       |CURL                                 |CUSR                |

============================================================================================================================================

L3R3844         |LOCALDERBY      |org.apache.derby.jdbc.EmbeddedDriver          |jdbc:derby:gaiandb;create=true       |gaiandb             |

L3R3844         |mysql5          |com.mysql.jdbc.Driver                         |jdbc:mysql://localhost:3306/test     |root                |

L3R3844         |DB2CONN         |com.ibm.db2.jcc.DB2Driver                     |jdbc:db2://localhost:50000/sample    |DavidVyvyan         |

============================================================================================================================================

Fetched 3 rows. Total Time: 47ms (Execution Time: 16ms)

sql>

listds() - this shows all the active data sources of every connected GaianDB node.


sql> call listds()

call listds()

=====================================================================================================================================================================================

GDB_NODE        |DSID                |DSTYPE|DSWRAPPER           |DSHANDLE                                                                               |DSOPTIONS                 |

=====================================================================================================================================================================================

L3R3844         |DB2SALES_DS0        |R     |tk:1                |jdbc:db2://localhost:50000/sample::sales                                               |-                         |

L3R3844         |LT1_DS0             |R     |EmbedStatement:1    |jdbc:derby:gaiandb;create=true::TABLE1                                                 |-                         |

L3R3844         |LT0_DS0             |V     |InMemoryRows:1      |./csvtestfiles/datafile.dat                                                            |INMEMORY                  |

L3R3844         |DB2EMPLOYEE_DS0     |R     |InMemoryRows:1      |jdbc:db2://localhost:50000/sample::employee where firstnme > 'T'                       |INMEMORY INDEX ON FIRSTNME|

L3R3844         |DF2_DSDATAFILE2     |V     |FileImport:1        |./csvtestfiles/datafile2.dat                                                           |-                         |

L3R3844         |MYSQLLT_DSCARS      |R     |InMemoryRows:1      |jdbc:mysql://localhost:3306/test::cars                                                 |INMEMORY                  |

L3R3844         |DERBY_TABLES_DSLOC  |R     |EmbedStatement:1    |jdbc:derby:gaiandb;create=true::sys.systables,sys.syscolumns where tableid=referenceid |-                         |

=====================================================================================================================================================================================

Fetched 7 rows. Total Time: 62ms (Execution Time: 46ms)

sql>

The data source id (DSID) is a composite name which has as prefix of the logical table name to which it is attached.
Other columns denote:

Column NameDescription

DSTYPEData source type (e.g. R for RDBMS, V for Virtual Table)
DSWRAPPER
Wrapping class and the number of pool instances in memory.
DSHANDLE
Physical source handle descriptor (e.g data source filename or db url+table name)
DSOPTIONS
Data source options list. This currently only describes whether the data source is held in memory and, if so, any index that might be defined on one of its columns.

Federate only a portion of the 'employee' table by using a selection predicate on it.

Here we will pass a more complex selection predicate to the setltforrdbtable(...) stored procedure so that only matching records from the 'employee' table are federated.


sql>

sql> call setltforrdbtable('db2employee', 'db2conn', 'employee where firstnme > ''T''')

call setltforrdbtable('db2employee', 'db2conn', 'employee where firstnme > ''T''')

sql>

sql> select * from db2employee

select * from db2employee

========================================================================================================================================================

EMPNO |FIRSTNME    |MIDINIT|LASTNAME       |WORKDEPT|PHONENO|HIREDATE  |JOB     |EDLEVEL|SEX|BIRTHDATE |SALARY     |BONUS      |COMM       |NODEINDEX  |

========================================================================================================================================================

000100|THEODORE    |Q      |SPENSER        |E21     |0972   |2000-06-19|MANAGER |     14|M  |1980-12-18|   86150.00|     500.00|    2092.00|          1|

000110|VINCENZO    |G      |LUCCHESSI      |A00     |3490   |1988-05-16|SALESREP|     19|M  |1959-11-05|   66500.00|     900.00|    3720.00|          1|

000210|WILLIAM     |T      |JONES          |D11     |0942   |1998-04-11|DESIGNER|     17|M  |2003-02-23|   68270.00|     400.00|    1462.00|          1|

000330|WING        |       |LEE            |E21     |2103   |2006-02-23|FIELDREP|     14|M  |1971-07-18|   45370.00|     500.00|    2030.00|          1|

========================================================================================================================================================

Fetched 4 rows. Total Time: 31ms (Execution Time: 15ms)

sql>

sql>

These selection predicates can be very complex. For example, you could use one to federate the result of a join between two tables.

Federate an existing CSV File

In this quick example we will show you how to federate an existing CSV flat file using GaianDB. For this we will use the file found at '<GaianDB Install Directory>/csvtestfiles/datafile3.dat'.

Note that GaianDB's FileImport VTI can read many different file formats, not just CSV.
The record and field separators and other 'control parameters' such as these may be altered using a 'control file'. Refer to
Text file federation configuration options (e.g: how to specify different record and field separators) for details.

Note: All paths, whether on Windows or Unix, should use forward slashes (/) and not backslashes (\).

Federate the CSV flat file 'datafile3.dat' as an RDBMS table called 'zzz'.

To do this, we call the stored GaianDB procedure setltforfile(<logical table name>, <file location>).


sql>

sql> call setltforfile('zzz', './csvtestfiles/datafile3.dat')

call setltforfile('zzz', './csvtestfiles/datafile3.dat')

sql>

List data sources to verify that the update was applied.

To do this, we call the stored GaianDB procedure listds().
Refer to the
previous explanation on the columns returned by listrdbc() and listds() for details on the columns used here.


sql>

sql> call listds()

call listds()

=========================================================================================================================================================================================

GDB_NODE            |DSID                |DSTYPE|DSWRAPPER           |DSHANDLE                                                                               |DSOPTIONS                 |

=========================================================================================================================================================================================

L3R3844             |DB2SALES_DS0        |R     |tk:1                |jdbc:db2://localhost:50000/sample::sales                                               |-                         |

L3R3844             |LT1_DS0             |R     |EmbedStatement:1    |jdbc:derby:gaiandb;create=true::TABLE1                                                 |-                         |

L3R3844             |LT0_DS0             |V     |InMemoryRows:1      |./csvtestfiles/datafile.dat                                                            |INMEMORY                  |

L3R3844             |DB2EMPLOYEE_DS0     |R     |InMemoryRows:1      |jdbc:db2://localhost:50000/sample::employee where firstnme > 'T'                       |INMEMORY INDEX ON FIRSTNME|

L3R3844             |ZZZ_DS0             |V     |FileImport:1        |./csvtestfiles/datafile3.dat                                                           |-                         |

L3R3844             |MYSQLLT_DSCARS      |R     |InMemoryRows:1      |jdbc:mysql://localhost:3306/test::cars                                                 |INMEMORY                  |

L3R3844             |DERBY_TABLES_DSLOC  |R     |EmbedStatement:1    |jdbc:derby:gaiandb;create=true::sys.systables,sys.syscolumns where tableid=referenceid |-                         |

=========================================================================================================================================================================================

Fetched 7 rows. Total Time: 31ms (Execution Time: 0ms)

sql>

If it was successful, you should see an entry with a DSID of 'ZZZ_DS0' and a DSHANDLE of './csvtestfiles/datafile3.dat'.

Query the new Logical Table.

To do this you can simply issue a standard SQL SELECT query on the table 'zzz'. For example:


sql>

sql> select * from zzz

select * from zzz

====================================================================================================================================================================================

COLUMN1           |COLUMN2               |COLUMN3             |COLUMN4             |COLUMN5             |COLUMN6             |COLUMN7             |COLUMN8             |NODEINDEX  |

====================================================================================================================================================================================

2                 |YYYYYYYYYYY2222222222 |62                  |35                  |17                  |10                  |122                 |65.6                |          1|

91                |BLAHBLAH              |3                   |2                   |44                  |35                  |0                   |2.1                 |          1|

9                 |SHG*&^22datafile3     |62                  |35                  |17                  |10                  |122                 |65.6                |          1|

9                 |FFFFFFFFFFFK2222222222|62                  |35                  |17                  |10                  |122                 |65.6                |          1|

====================================================================================================================================================================================

Fetched 4 rows. Total Time: 16ms (Execution Time: 16ms)

sql>

sql>

This contains all columns from the CSV file and also an example constant column 'NODEINDEX' to identify which GaianDB node the information is from. For more info on constant columns, refer to this section.

Federate an existing Excel spreadsheet

In this quick example we will show you how to federate an existing Excel spreadsheet using GaianDB. For this we will use the spreadsheet found at '<GaianDB Install Directory>/exceltestfiles/address.xls'.

Note: All paths, whether on Windows or Unix, should use forward slashes (/) and not backslashes (\).

Prerequisites

If you wish to perform this example, you will require the POI libraries (version 3.6 or above) for manipulating Microsoft Documents from the Apache POI website. If you are happy to accept the license terms, you can choose to download the binary distributions. The necessary jar files are:
- poi-XX.jar
- poi-ooxml-XX.jar
- poi-ooxml-schemas-XX.jar
- dom4j-XX.jar
- geronimo-stax-api_XX.jar (or stax-api-xx.jar in some versions of POI)
- xmlbeans-XX.jar
Where XX is the version number.
These jars would need to be placed into the GaianDB lib directory and then added to the GaianDB classpath as follows:
- On Windows, edit the 'launchGaianServer.bat' file (right click -> edit).
- Add the following lines :
```
rem Apache - POI jars for spreadsheet federation

SET CLASSPATH=%CLASSPATH%;%GDBL%\geronimo-stax-api_XX.jar;%GDBL%\poi-ooxml-schemas-XX.jar;%GDBL%\dom4j-XX.jar;%GDBL%\poi-XX.jar;%GDBL%\poi-ooxml-XX.jar;%GDBL%\xmlbeans-XX.jar

			 
```
- On Unix, edit the 'launchGaianServer.sh' file.
- Add the following lines :
```
# Apache - POI jars for spreadsheet federation

export CLASSPATH="$CLASSPATH:$GDBL/geronimo-stax-api_XX.jar:$GDBL/poi-ooxml-schemas-XX.jar:$GDBL/dom4j-XX.jar:$GDBL/poi-XX.jar:$GDBL/poi-ooxml-XX.jar:$GDBL/xmlbeans-XX.jar"

                          
```
If you had already started the GaianDB by using the launchGaianServer script, you will need to stop the node and restart it for the new changes to be taken into account.
To stop GaianDB, you can either close the command window on Windows, or use Ctrl-C on Unix or use the killGaianServer script.
When starting multiple nodes, you will need to use the killGaianServers script to stop them all.

Federate the Excel spreadsheet 'Sheet1' from the file 'address.xls' as an RDBMS table.
To do this, we use the GaianDB stored procedure setltforexcel(...):


sql>

sql> call setltforexcel('lsheet1', 'exceltestfiles/address.xls,Sheet1')

call setltforexcel('lsheet1', 'exceltestfiles/address.xls,Sheet1')

sql>

You can also federate a part of the spreadsheet by defining a range :


sql>

sql> call setltforexcel('lsheet2', 'exceltestfiles/address.xls,Sheet1,A1,D7')

call setltforexcel('lsheet2', 'exceltestfiles/address.xls,Sheet1,A1,D7')

sql>

By default, the values of the first row are used as the column names for the federated table. However, if the first row does not contain the column names, you can add 'false' to the parameters to have the column names automatically generated; in the form COLUMN1, COLUMN2, COLUMN<n>:


sql>

sql> call setltforexcel('lsheet3', 'exceltestfiles/address.xls,Sheet1,A5,D7,false')

call setltforexcel('lsheet3', 'exceltestfiles/address.xls,Sheet1,A5,D7,false')

sql>

List data sources to verify that the update was applied.
To do this, we call the stored GaianDB procedure listds(). Refer to the previous explanation on the columns returned by listrdbc() and listds() for details on the columns used here.


sql>

sql> call listds()

call listds()

===============================================================================================================================================================================

GDB_NODE    |DSID                |DSTYPE|DSWRAPPER         |DSHANDLE                                                                               |DSOPTIONS                 |

===============================================================================================================================================================================

L3R3844     |DB2SALES_DS0        |R     |tk:1              |jdbc:db2://localhost:50000/sample::sales                                               |-                         |

L3R3844	    |LSHEET1_DS0         |V     |-                 |address.xls,Sheet1                                                                     |-                         |

L3R3844     |LSHEET2_DS0         |V     |-                 |address.xls,Sheet1,A:1,D:7                                                             |-                         |

L3R3844     |LT1_DS0             |R     |EmbedStatement:1  |jdbc:derby:gaiandb;create=true::TABLE1                                                 |-                         |

L3R3844     |LT0_DS0             |V     |InMemoryRows:1    |./csvtestfiles/datafile.dat                                                            |INMEMORY                  |

L3R3844     |DB2EMPLOYEE_DS0     |R     |InMemoryRows:1    |jdbc:db2://localhost:50000/sample::employee where firstnme > 'T'                       |INMEMORY INDEX ON FIRSTNME|

L3R3844     |ZZZ_DS0             |V     |FileImport:1      |./csvtestfiles/datafile3.dat                                                           |-                         |

L3R3844     |MYSQLLT_DSCARS      |R     |InMemoryRows:1    |jdbc:mysql://localhost:3306/test::cars                                                 |INMEMORY                  |

L3R3844     |DERBY_TABLES_DSLOC  |R     |EmbedStatement:1  |jdbc:derby:gaiandb;create=true::sys.systables,sys.syscolumns where tableid=referenceid |-                         |

===============================================================================================================================================================================

Fetched 7 rows. Total Time: 31ms (Execution Time: 0ms)

Query the new Logical Tables.
To do this you can simply issue a standard SQL SELECT query against the 'LSheet1', 'LSheet2' and 'LSheet3' tables. For example:


sql> select * from LSheet1

select * from LSheet1

================================================================================================

LAST        |FIRST          |ADDRESS                    |CITY            |STATE    |ZIP        |

================================================================================================

Buffet      |Jimmy          |Somewhere on the Beach     |KeyWest         |FL       |      33040|

Bush        |George         |1600 Pennsylvania Ave      |Washington      |DC       |      20500|

Cartman     |Eric           |84 Bigboned Way            |South Park      |CO       |      84214|

Crockett    |Davey          |The Alamo                  |San Antonio     |TX       |      78210|

Doe         |Jane           |821 Zimbabwe Ave           |Washington      |DC       |      20021|

Gates       |Bill           |1 Microsoft Way            |Redmond         |WA       |      98052|

Jefferson   |George         |194 Deelux Apartments      |In the Sky      |NY       |      10041|

Kong        |King           |Empire State Building      |New York        |NY       |      10118|

Munster     |Herman         |1313 Mockingbird Lane      |Fargo           |ND       |      58102|

Rockne      |Knute          |146 Keenan Hall            |Notre Dame      |IN       |      46556|

Simpson     |Homer          |742 Evergreen Terrace      |Springfield     |US       |      12345|

Smith       |Bob            |12 Main Street             |Anytown         |IN       |      46001|

================================================================================================

Fetched 12 rows. Total Time: 62ms (Execution Time: 31ms)


sql> select * from LSheet2

select * from LSheet2

==========================================================================

LAST        |FIRST          |ADDRESS                    |CITY            |

==========================================================================

Buffet      |Jimmy          |Somewhere on the Beach     |KeyWest         |

Bush        |George         |1600 Pennsylvania Ave      |Washington      |

Cartman     |Eric           |84 Bigboned Way            |South Park      |

Crockett    |Davey          |The Alamo                  |San Antonio     |

Doe         |Jane           |821 Zimbabwe Ave           |Washington      |

Gates       |Bill           |1 Microsoft Way            |Redmond         |

==========================================================================

Fetched 6 rows. Total Time: 46ms (Execution Time: 15ms)


sql> select * from LSheet3

select * from LSheet3

==========================================================================

COLUMN1     |COLUMN2        |COLUMN3                    |COLUMN4         |

==========================================================================

Crockett    |Davey          |The Alamo                  |San Antonio     |

Doe         |Jane           |821 Zimbabwe Ave           |Washington      |

Gates       |Bill           |1 Microsoft Way            |Redmond         |

==========================================================================

Fetched 3 rows. Total Time: 63ms (Execution Time: 47ms)

Note


sql> select * from lsheet1 where "LAST" > 'C'

select * from lsheet1 where "LAST" > 'C'

==========================================================================================================

LAST        |FIRST          |ADDRESS                    |CITY             |STATE    |ZIP        |COLUMN7 |

==========================================================================================================

Cartman     |Eric           |84 Bigboned Way            |South Park       |CO       |      84214|-       |

Crockett    |Davey          |The Alamo                  |San Antonio      |TX       |      78210|-       |

Doe         |Jane           |821 Zimbabwe Ave           |Washington       |DC       |      20021|-       |

Gates       |Bill           |1 Microsoft Way            |Redmond          |WA       |      98052|-       |

Jefferson   |George         |194 Deelux Apartments      |In the Sky       |NY       |      10041|-       |

Kong        |King           |Empire State Building      |New York         |NY       |      10118|-       |

Munster     |Herman         |1313 Mockingbird Lane      |Fargo            |ND       |      58102|-       |

Rockne      |Knute          |146 Keenan Hall            |Notre Dame       |IN       |      46556|-       |

Simpson     |Homer          |742 Evergreen Terrace      |Springfield      |US       |      12345|-       |

Smith       |Bob            |12 Main Street             |Anytown          |IN       |      46001|-       |

==========================================================================================================

Fetched 10 rows. Total Time: 59ms (Execution Time: 55ms)

If you wish, you can also add another Excel file as a Data Source to one of the logical tables created above. Here we select row 2 and 3 to be added to LSheet3.
So LSheet3 will now have a view of rows: 2,3 and 5,6,7.


sql> call setdsexcel('lsheet3', '1', 'exceltestfiles/address.xls,Sheet1,A2,D3,false','MAP_COLUMNS_BY_POSITION', '')

call setdsexcel('lsheet3', '1', 'exceltestfiles/address.xls,Sheet1,A2,D3,false','MAP_COLUMNS_BY_POSITION', '')


sql> call listds()

call listds()

=================================================================================================================================================================================

GDB_NODE    |DSID                |DSTYPE|DSWRAPPER           |DSHANDLE                                                                               |DSOPTIONS                 |

=================================================================================================================================================================================

L3R3844     |DB2SALES_DS0        |R     |tk:1                |jdbc:db2://localhost:50000/sample::sales                                               |-                         |

L3R3844	    |LSHEET1_DS0         |V     |-                   |address.xls,Sheet1                                                                     |-                         |

L3R3844     |LSHEET2_DS0         |V     |-                   |address.xls,Sheet1,A:1,D:7                                                             |-                         |

L3R3844     |LSHEET3_DS0         |V     |-                   |address.xls,Sheet1,A5,D7,false                                                         |-                         |

L3R3844     |LSHEET3_DS1         |V     |-                   |address.xls,Sheet1,A2,D3,false                                                         |MAP_COLUMNS_BY_POSITION   |

L3R3844     |LT1_DS0             |R     |EmbedStatement:1    |jdbc:derby:gaiandb;create=true::TABLE1                                                 |-                         |

L3R3844     |LT0_DS0             |V     |InMemoryRows:1      |./csvtestfiles/datafile.dat                                                            |INMEMORY                  |

L3R3844     |DB2EMPLOYEE_DS0     |R     |InMemoryRows:1      |jdbc:db2://localhost:50000/sample::employee where firstnme > 'T'                       |INMEMORY INDEX ON FIRSTNME|

L3R3844     |ZZZ_DS0             |V     |FileImport:1        |./csvtestfiles/datafile3.dat                                                           |-                         |

L3R3844     |MYSQLLT_DSCARS      |R     |InMemoryRows:1      |jdbc:mysql://localhost:3306/test::cars                                                 |INMEMORY                  |

L3R3844     |DERBY_TABLES_DSLOC  |R     |EmbedStatement:1    |jdbc:derby:gaiandb;create=true::sys.systables,sys.syscolumns where tableid=referenceid |-                         |

=================================================================================================================================================================================

Fetched 9 rows. Total Time: 31ms (Execution Time: 0ms)


sql> select * from lsheet3 order by column1

select * from lsheet3 order by column1

==========================================================================

COLUMN1     |COLUMN2        |COLUMN3                    |COLUMN4         |

==========================================================================

Buffet      |Jimmy          |Somewhere on the Beach     |KeyWest         |

Bush        |George         |1600 Pennsylvania Ave      |Washington      |

Crockett    |Davey          |The Alamo                  |San Antonio     |

Doe         |Jane           |821 Zimbabwe Ave           |Washington      |

Gates       |Bill           |1 Microsoft Way            |Redmond         |

==========================================================================

Fetched 5 rows. Total Time: 296ms (Execution Time: 294ms)

Further Examples

For more examples see the Worked Examples page.

GaianDB Usage

GaianDB node usage

To launch a GaianDB node, simply run 'launchGaianServer.bat' on Windows or './launchGaianServer.sh' on Unix; which can be found in the GaianDB install directory.

This will start a GaianDB node using the default options.

To specify different options at startup, from the command line, simply run the appropriate startup script for your operating system as follows:

On Windows:

launchGaianServer.bat [-n <nodeID>] [-p <gaian node port>] [-c <configuration file name>] [-mt <message storer topic>] [-log <log level>] [-console] [-g <gateways>] [-initscript <sql init file>]

On Unix:

./launchGaianServer.sh [-n <nodeID>] [-p <gaian node port>] [-c <configuration file name>] [-mt <message storer topic>] [-log <log level>] [-console] [-g <gateways>] [-initscript <sql init file>]

The parameters are defined as follows:

Parameter
Default
Description

-n <nodeID>
<hostname value>
The node ID you wish to use. A suffix of ':<port number>' will be appended for all nodes not running on the default port (6414)

-p <gaian node port>
6414
The TCP port for the GaianDB node to use.

-c <configuration file name>
gaiandb_config.properties
The name of the configuration file for the GaianDB node to use (located in the current working directory).

The file must have the '.properties' extension and will not be created automatically if it does not exist.

For more information, please refer to the Configuration section.

-mt <message storer topic>
None
The topic for the Message Storer to use.
For more information please refer to the Message Storer: Message Broker Integration (e.g. use with sensors) section.

-log <log level>
None
The level of logging to use. This can only be one of: [NONE, LESS, MORE, ALL].

If NONE, then no logging is performed.

Otherwise logging is performed to a file in the current working directory. This file will be named 'gaiandb<port>.log' if a port was specified (using -p) or simply 'gaiandb.log' if no port was specified.

The logging level can otherwise be dynamically updated using the configuration file.

-console
N/A
If specified, then the GaianDB node will log to System.out.

If used in combination with the -log parameter, then logging will be performed to both System.out and a file.

-g <gateways>
None
List of Discovery Gateways. A discovery gateway is a node outside of your subnet that acts as a relay allowing you to join nodes in its network.

-initscript <sql init file>
None
File location of optional custom SQL initialisation script. For example this may setup logical or physical tables; or stored procedures/functions.

Note: these are all optional and when not provided the default values will be used. Also, ordering is not important.

To stop GaianDB, you can either close the command window it was started from, or run the killGaianServers script

Launching multiple GaianDB nodes

To launch multiple GaianDB nodes on the same machine, simply run 'launchMultipleNodes.bat' on Windows or './launchMultipleNodes.sh' on Unix; which can be found in the GaianDB install directory.

By default this will kick off 3 GaianDB nodes that respectively use the pre-canned configuration files: 'gaiandb_config.properties', 'gaiandb_config2.properties' and 'gaiandb_config3.properties' in the current working directory.

You can optionally pass in the number of nodes to launch and you may also use the '-sameconfig' flag to request that each node use the default configuration file: 'gaiandb_config.properties', as follows:

On Windows:

launchMultipleNodes.bat [<numnodes> [-sameconfig]]

On Unix:

./launchMultipleNodes.sh [<numnodes> [-sameconfig]]

If the '-sameconfig' flag is not specified, each new launched node will expect to find a new 'gaiandb_configN.properties' file where N is the index of the launched node.

To stop GaianDB, you will need to run the killGaianServers script.

Launching a GaianDB node from a separate directory

A GaianDB node will use the working directory to lookup its system files, i.e. gaiandb_config.properties, derby.properties, gaiandb.log, derby.log and the local physical derby database 'gaiandb'.

However this directory does not neccessarily have to be the GaianDB install folder... This means that you can store separate databases/configurations/logs etc for GaianDB in separate folders,
e.g. one may have tables accessing a DB2 and another may just be a test instance. These would effectively be 'workspace' folders where you can run GaianDB from.
This can be very useful in networked file system scenarios where you may have multiple machines all having their own workspace folder, and loading the GaianDB code from a remote
location (which you may also not have write-access to).

To run GaianDB from a separate workspace directory, you just need to set the environment variable GDBH to the GaianDB install folder before launching the 'launchGaianServer' script.
For example:

On Windows:


  set GDBH=<install path>

  %GDBH%\launchGaianServer.bat

On Unix:


  export GDBH=<install path>

  $GDBH/launchGaianServer.sh

To stop GaianDB, you will need to run the killGaianServers script.

How to specify a location for configuration and log files and the physical database folder

Continuing on the topic above, in some scenarios you may wish to explicitly set the default location of GaianDB's workspace files:
i.e. gaiandb_config.properties, derby.properties, gaiandb.log, derby.log and the local physical derby database 'gaiandb'

This can be useful in particular if you have wrapped GaianDB in an OSGi framework or some other 'owning process' which has no concept of a working directory for GaianDB.
To set the location of these GaianDB workspace files, set the Java system property: 'derby.system.home' to point to it before starting the Gaian Database. Note that the gaiandb
config file location can also be overriden beyond this using the -c option passed to launchGaianServer (or GaianNode class directly).

Here's an example where you want to run GaianDB from an independant "User directory", referencing a separate "GaianDB workspace folder" and "GaianDB install path":


  # Inside <User directory>/launchGaianServer.sh...:

  export GDBH=<GaianDB install path>

  java -Xmx128m -Dderby.system.home="<GaianDB workspace folder>" -cp "$CLASSPATH" com.ibm.gaiandb.GaianNode $args

How to manage multiple GaianDB node executions with multiple working directories

While not a typical use case, it may sometimes be desirable to run multiple GaianDB nodes on the same computer. For instance, if you are performing testing or for the purposes of resiliency or load balancing.

If you run multiple GaianDB nodes from the same working directory, they will by default share the same configuration file. This may be desirable if every node shares the same logical table definitions and configuration settings. Otherwise, you would need to specify a different configuration file for each node (using the -c parameter, as explained in GaianDB Server usage).

A cleaner approach is to have each GaianDB node use it's own working directory; and consequently it's own configuration file. The Launching GaianDB from a separate directory section details how to do this.

One issue to consider when running multiple GaianDB nodes on the same machine is whether there will be any physical resource contention. As an example, a single Derby database will only allow one process to access it directly at any one time and so multiple nodes trying to access it simultaneously will suffer from resource contention and may behave unexpectedly. However, you are unlikely to encounter this kind of issue in typical usage scenarios.

How to initialise a GaianDB node with User defined Derby objects like Tables, views, procedures,functions

It may be sometimes desirable to initialise the derby database of a GaianDB node with tables, views, index, records, procedures and functions before the Logical tables and datasources are created by GaianDB.

An easy implementation is to prime the Derby database before launching the GaianDB as usual.
The priming can be linked to the presence of the GaianDB directory.If the directory is not present, that means the GaianDB has not been launched previously and therefore the derby database need to be created and primed.
For this, one can create a copy of the Queryderby.bat to run a SQL script in standalone derby mode with eventually creation of the Database, with the following lines:

The new batch file launchGaianServerWithInit.bat can be modified like the following (on Windows):

@echo off
TITLE Initiliasing the Derby database

if not defined GDBH set GDBH=.
set GDBL=%GDBH%\lib

SET CLASSPATH="%GDBL%\;%GDBL%\GAIANDB.jar;%GDBL%\db2jcutdown.jar;%GDBL%\derby.jar;%GDBL%\derbyclient.jar"

SET ARGS=%*

REM Run a SQL script only if gaianDB directory doesn't exist.
IF EXIST GAIANDBX GOTO DB_ALREADY_INITIALIZED
echo Creating the necessary SQL objects for my application....
java -cp "%CLASSPATH%" com.ibm.gaiandb.tools.SQLDerbyRunner -td@ -standalone -createdb gaiandb_init2.sql
:DB_ALREADY_INITIALIZED

Call launchGaianServer.bat

The file gaiandb_init2.sql looks like this below:


  -- a set of SQL statements delimited by @ (signs for multiline support)

  -- and prefixed with ! (for ignoring exceptions:see CLP paragraph in GaianDB documentation)

  !drop table Mytable@

  !create table Mytable (a varchar(20), 

                             bb varchar(10),

                             c int)@

  !create index i1 on  Mytable(a)@                            

  !insert into Mytable values ( 'yoya', 'drvy', 23 )@

  --....more DDL and SQL statements here....

The output of launchGaianServerWithInit.bat will look something like this the first time:

Creating the necessary SQL objects for my application....
Processing args: [-td@, -standalone, -createdb, gaiandb_init2.sql]

Connecting to derby database: jdbc:derby:gaiandb;create=true, usr: gaiandb...

drop table Mytable
Suppressed Exception: 'DROP TABLE' cannot be performed on 'MYTABLE' because it does not exist.
create table Mytable (a varchar(20), bb varchar(10), c int)
Update count: 0 (Execution Time: 23ms)

insert into Mytable values ( 'yoya', 'drvy', 23 )
Update count: 1 (Execution Time: 16ms)

Launching Server...
PROCESS ID: 101832
NODE ID: MYGDBNODE
WORKING DIRECTORY: C:\GAIANDB_Demo
LOG FILE: gaiandb.log
PHYSICAL DATABASE: gaiandb
CONFIG FILE: gaiandb_config.properties
VERSION INFO: V2.x - JAR sizes: [640461, 372333], timestamps: [2012/03/14-17:02:12, 2012/03/14-
17:02:14]

GaianNode started for Derby network server on port: 6414 at Wed Mar 14 17:03:48 GMT 2012

Wed Mar 14 17:03:48 GMT 2012: Connections: Maintained to [MYGDBNODE] (seeking 1), Accepted from []

How to integrate GaianDB into a wrapping class or framework: startup, start detection, kill

To integrate GaianDB as an OSGi bundle or as a child 'task' of an owning parent class, you will need to have a means of starting a node, detecting when it has started and stopping it.

This can be done using public methods 'start()', 'isStarted()' and 'killNode()' on the GaianNode class in GAIANDB.jar. Here's some sample startup code:


String gdbHome = System.getProperty("derby.system.home");

if ( null == gdbHome )

	System.setProperty( "derby.system.home", gdbHome = com.ibm.gaiandb.Util.getInstallPath() );

			

final String[] gaianTaskStartupOptions = new String[] { "-c", gdbHome + "/gaiandb_config.properties" }; // e.g new String[] { "-p", "6414", "-n", "MyNodeID" }

final GaianNode gdbNode = new GaianNode();

Thread gdbParentThread = new Thread( new Runnable() {

	public void run() {

		try {

			gdbNode.start( gaianTaskStartupOptions );

			System.out.println("GaianNode parent thread exited cleanly (no Exception)");

		}

		catch (Exception e) {

			System.out.println("GaianNode parent thread Exception: " + e);

		}

	}

});

gdbParentThread.start();

boolean gdbStartupGood = true;

while ( gdbNode.isStarted() == false ) {

	if ( ! gdbParentThread.isAlive() ) {

		System.out.println("GaianNode parent thread has died");

		gdbStartupGood = false;

		break;

	}

	Thread.sleep( 100 );

}

Client Usage

Command line processor (CLP) options with 'queryDerby'

A command line processor (CLP) client utility is provided with the GaianDB installation. This can be used to run queries on GaianDB and also to issue the GaianDB stored procedures. You can find the CLP in the installation directory and it can be launched by running 'queryDerby.bat' on Windows and 'queryDerby.sh' on Unix.

By default the CLP tries to connect to a local Derby Network server, running on the default GaianDB port: 6414, which is started automatically when a GaianDB node is started.

You can specify different options at startup by launching 'queryDerby' from the command line as follows:

On Windows:


queryDerby.bat [-h <host>] [-p <port>] [-usr <usr>] [-pwd <pwd>] [-standalone] [-d <database>] [-createdb] [-nocreatedb] [-repeat <count>] [-tab] [-csv] [-raw] [-csvraw] [-quiet] [-batchprefix <prefix sql>] <sql queries | queries files>*

On Unix:


./queryDerby.sh [-h <host>] [-p <port>] [-usr <usr>] [-pwd <pwd>] [-standalone] [-d <database>] [-createdb] [-nocreatedb] [-repeat <count>] [-tab] [-csv] [-raw] [-csvraw] [-quiet] [-batchprefix <prefix sql>] <sql queries | queries files>*

The parameters are defined as follows:

Parameter
Default
Description

-h <host>

localhost

The host of the GaianDB node to connect to.

-p <port>

6414

The port of the GaianDB node to connect to.

-usr <usr>

gaiandb

The username to use to connect.

-pwd <pwd>

passw0rd

The password to use to connect.

-standalone

N/A

Connect to Derby directly using the Embedded driver.

This option is incompatible with any host or port setting.

Subsequently setting the -h or -p parameter will invalidate the setting of this parameter.

-d <database>

gaiandb
(or gaian<port>db, if -p is specified)

The database to connect to.

-createdb

N/A

On connect, create the database if it doesn't already exist (see -d for which database this will be).

Note: If a database is created, the default schema will not have been created yet and the shorthand GaianDB views & stored procedures will not be defined.

-nocreatedb

N/A

On connect, do NOT create the database if it doesn't already exist (see -d for which database this will be).

-repeat <count>

N/A

Specifies the number of times any query will be immediately re-issued. i.e. if you set '-repeat 3', then any query you issue will be run 3 times in a row.

-tab

N/A

Output query results in the default table format which is with table headings and vertical line separators.

This is the default.

-csv

N/A

Output query results in a CSV (comma-separated-values) format.

-raw

N/A

Output query results as raw data (space-separated) format, with no information or headers.

-csvraw

N/A

Output query results as raw data in CSV (comma-separated-values) format, with no information or headers.

-quiet

N/A

No output to stdout.

-batchprefix <prefix sql>

N/A

Specifies an SQL fragment to insert as the prefix to every SQL query batch file passed in on the command line.

See here for more details.

<sql queries | queries files>*

N/A

A space-separated list of SQL queries, or files containing SQL queries, to run immediately on connection.

See here for more details.

In addition to specifying these parameters when launching 'queryDerby', all flag switches (-h, -p, -usr, etc.) may be entered in the CLP at any time and will take effect immediately.

As an example, this makes it possible to manage multiple GaianDB nodes from a single window (simply by switching between them using -h and -p). As a special case, note that specifying the -p flag will automatically change the database name to 'gaiandb' (as this is the default database name for a GaianDB node running on a different port). To override this, you can use the '-d' flag to explicitly specify which database name to use.

Whenever query execution fails (for example, because the Derby connection was lost after we changed the 'port' connection value with the '-p' option), the CLP attempts to reconnect automatically:


sql>

sql> -p 6415

sql>

sql> select * from LT0_P

Attempting to re-connect...

Connecting to GAIAN server using url: jdbc:derby://localhost:6415/gaiandb6415;create=false, usr: gaiandb...

Connection attempt succeeded, Re-run query [n/other] ?

This is also the case when any Exception occurs during query execution. Although in this case, after recycling the connection, the CLP also automatically runs the stored procedure: call listwarnings() to retrieve and display any recently raised server warnings.

When launching 'queryDerby', you can pass in a list of SQL queries and/or files containing multiple SQL statements to execute immediately. For example:

On Windows:


queryDerby.bat "select * from LT0;select * from LT0" FileWithSQLQueriesToTest.sql

On Unix:


./queryDerby.sh "select * from LT0;select * from LT0" FileWithSQLQueriesToTest.sql

Where the file 'FileWithSQLQueriesToTest.sql' contains:


select * from LT0;

select * from LT0;

Note: When creating a batch file of SQL statements, each new statement should be on it's own line and suffixed with ';'.

This example executes the query 'select * from LT0' four times.

As well as this, a 'batchprefix' SQL fragment value can be passed in using the '-batchprefix' flag. This fragment of SQL is prefixed to all SQL statements to be executed from batch files passed in when invoking 'queryDerby'. Note: this only applies to the queries passed in in batch files.

Issuing Distributed SQL Queries

A GaianDB distributed query is a query against a logical table (or a distributed sub-query) defined on multiple nodes in the network.

Each of the GaianDB nodes may have a different underlying data source for the logical table (e.g. an RDBMS, an excel file, etc.), however they all share the same logical table definition and so they can be queried consistently.

When the query is executed, it is 'distributed' to each GaianDB node in the network by forwarding the predicates against the queried columns. Each node then maps these locally to the underlying physical types of the data sources attached to the logical table, and performs the query.

For a GaianDB node to be included in the distributed query, it's local logical table definition must have matching data types for all the columns involved in the distributed query (based on the logical table definition of the original querying node). However, there is some flexibility, as columns don't have to be present (in which case a null value will be returned) and extra columns will be ignored.

Results from each of these distributed queries are then union'ed together as they are fed back to the querying node in the network.

Here is the syntax for referring to a logical table that can be used in place of any table identifier in SQL:

new com.ibm.db2j.GaianTable('<Logical Table Name>'[, '<Logical Table Arguments>']) <Alias Identifier>

For example:

select * from new com.ibm.db2j.GaianTable('LT0') LT0

Or, to query against a logical table 'ACCOUNTS', requesting provenance columns:

select * from new com.ibm.db2j.GaianTable('ACCOUNTS', 'with_provenance') T

Note: The Alias Identifier should be included so that the nested query can be uniquely referenced. This is standard SQL best practice and is useful in cases where you wish to join multiple tables/queries and need to refer to each of the tables'/sub-queries' columns uniquely. For example, if two queries, aliased as Q1 & Q2 respectively, both returned the column FIRSTNAME, you could reference each queries' column uniquely by using Q1.FIRSTNAME and Q2.FIRSTNAME.

A full list of Logical Table Argumentsis described later in the document.

For convenience, GaianDB automatically maintains some system managed views to query a GaianDB logical table. For a logical table LT0, the list of all managed views is:

View
Description

LT0
Propagated logical table - represents the logical table over all GaianDB nodes

LT0_0
Just local (not propagated) - represents the logical table on the local GaianDB node only

LT0_P
Propagated, with provenance columns

LT0_X
Explain query - see Explain Queries: Getting and showing the route of a query for more information.

LT0_XF
Explain query to file graph.dot - see Explain Queries: Getting and showing the route of a query for more information.

For example, to retrieve all rows from logical table LT0 for data sources federated by the local GaianDB node only, use the SQL:

select * from LT0_0

Note that these views are updated automatically when a change to the GaianDB configuration for the logical table is made.

Distributed Sub-queries or Pushing query processing to each node

Another, much more powerful, approach to distributed querying with GaianDB is to use distributed sub-queries. In this case, you specify a sub-query (nested query) that is 'distributed' to each GaianDB node, run locally and the results are then union'ed together as they are fed back to the original querying node in the network. The results are then queried by the querying node to get the final result set.

As an example, the following statement defines a query to select everything from ('select * from') the results of the
distributed sub-query 'select * from example':

select * from new com.ibm.db2j.GaianQuery('select * from example') Q

In this case, the sub-query will be distributed to each GaianDB node in the network and will be executed
against the local derby RDBMS that each GaianDB node rests on; querying a physical table named 'example',
under the schema 'gaiandb' of Derby database 'gaiandb' (as this is the default behaviour).

Note: The 'example' table is not defined in a default GaianDB installation. If you wish to create it,
please refer to the Worked Examples.

Here is the syntax for referring to a distributed sub-query that can be used in place of any table identifier in SQL:

new com.ibm.db2j.GaianQuery('<Nested SQL Query>'[, '<Logical Table Arguments>'[, '<Nested Query Arguments>']]) <Alias Identifier>

Note: any single quotes (') appearing inside Nested SQL Queries must be escaped with another single quote ('').

A full list of Logical Table Arguments and Nested Query Arguments is described later in the document.

By default, the nested query will be executed against the local Derby database that an GaianDB node runs on (as in the example above).
However, the nested query can also target a particular SOURCELIST; which is a way of targeting an abstracted list of
exposed data sources individually on every node. In this case, the query is 'passed on' to each exposed data source for it to execute.
See the distributed query options section for more information. As an example, the following query targets a SOURCELIST
called TOYS:

select * from new com.ibm.db2j.GaianQuery('select * from toysales', 'with_provenance', 'SOURCELIST=TOYS') Q

Note: the resources referenced in this example are not setup in a default GaianDB configuration and so the example will not work
out of the box.

The nested query can also reference a logical table, as follows:

select * from new com.ibm.db2j.GaianQuery('select * from new com.ibm.db2j.GaianTable(''LT0'', ''maxDepth=2'') T') Q

In this case, the query will be distributed out from the local Derby GaianDB node to every node in the network and each of them will query the logical table 'LT0' out to a depth of 2.

Note: it would make no sense to specify a SOURCELIST in this case as the sub-query contains a GaianDB construct (GaianTable()) which will not be meaningful to exposed data sources other than the GaianDB nodes themselves.

Here are some useful tips on propagating queries using GaianQuery():

To push query processing to each node, i.e. to make all nodes query their local federated data sources and return their own timestamps and provenance information, you can reference the managed view with suffix '_0'. For example:
```
select * from new com.ibm.db2j.GaianQuery('select CURRENT_TIMESTAMP TS, LOCATION, MISC from LT0_0', 'with_provenance') Q
```
This makes every node query its own local federated sources for LT0 columns LOCATION and MISC and compute its own local timestamp.
The provenance columns are added to the results on every node.

One way of propagating a function without needing a logical table is using a dummy system physical table.
A dummy table tends to have a single record and is used for selecting when you're not actually interested in the data, but instead want the results of some system function in a select statement.
The default dummy table on Derby is 'sysibm.sysdummy1' and, as an example, it can be used to return the timestamp from each node in the network as follows:
```
select * from new com.ibm.db2j.GaianQuery('select current_timestamp ts from sysibm.sysdummy1', 'with_provenance') Q
```

If a nested query contains a distributed table or query, it may only reference the provenance, explain or constant columns if these are renamed within the nested query.
This is because otherwise they would be seen as potential duplicates with the columns defined by the outer GaianQuery().

e.g. this is invalid:
```
select * from new com.ibm.db2j.GaianQuery('select CURRENT_TIMESTAMP TS, GDB_NODE from LT0_P') Q
```
but this is OK:
```
select * from new com.ibm.db2j.GaianQuery('select CURRENT_TIMESTAMP TS, GDB_NODE NESTED_PROVENANCE from LT0_P') Q
```
Note: here we use the managed view 'LT0_P' which uses suffix '_P' to designate logical table 'LT0' with provenance columns.

Using distributed sub-queries is a powerful mechanism that provides the opportunity to perform many complex operations.
As a final example of this, note here how it allows non-aggregate functions on partitioned data to be pushed down to all nodes in the network.
The GaianQuery below targets local data sources of logical table LT at each node, using LT's view having the "_0" suffix (i.e. restricted to depth 0).
This trick allows us to independently compare at each node the msg_received time values in LT at that node with it's own individual local system time
and only retrieve data that was generated in the last 10 minutes.

select * from new com.ibm.db

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/gaiandb/gaiandb

Awesome Lists containing this project

README

Readme.html taken from build/GAIANDB_V2.1.8_20160523.zip

Contents

What is the Gaian Database ?

Key benefits - what does Gaian give you ?

What's new in version 2

Version 2.1.4

Installation and First Steps

Upgrade Gaian

Installation Prerequisites

Supported Operating Systems

First Steps

Verify the installation

Quick table federation examples (matching a Logical Table to a Physical One)

Federate an RDBMS table

Federate an existing CSV File

Federate an existing Excel spreadsheet

Further Examples

GaianDB Usage

GaianDB node usage

Launching multiple GaianDB nodes

Launching a GaianDB node from a separate directory

How to specify a location for configuration and log files and the physical database folder

How to manage multiple GaianDB node executions with multiple working directories

How to initialise a GaianDB node with User defined Derby objects like Tables, views, procedures,functions

How to integrate GaianDB into a wrapping class or framework: startup, start detection, kill

Client Usage

Command line processor (CLP) options with 'queryDerby'

Issuing Distributed SQL Queries

Distributed Sub-queries or Pushing query processing to each node