Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kxsystems/kxkdb
Kdb+ interface for the Rust programming language
https://github.com/kxsystems/kxkdb
kdb q rust
Last synced: about 2 months ago
JSON representation
Kdb+ interface for the Rust programming language
- Host: GitHub
- URL: https://github.com/kxsystems/kxkdb
- Owner: KxSystems
- Created: 2023-01-21T19:04:52.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2023-06-05T14:28:43.000Z (over 1 year ago)
- Last Synced: 2024-10-29T00:51:34.466Z (2 months ago)
- Topics: kdb, q, rust
- Language: Rust
- Homepage: https://code.kx.com/q/interfaces
- Size: 286 KB
- Stars: 3
- Watchers: 2
- Forks: 3
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# kxkdb
Kdb+ interface for the Rust programming language.
The interface comprises two features:
- IPC: Connecting Rust and kdb+ processes via IPC
- API: Embedding Rust code inside kdb+ processes
## Documentation
Documentation for this interface can be found at https://docs.rs/kxkdb/.
## Kdbplus
The kxkdb interface is forked from the excellent [kdbplus](https://crates.io/crates/kdbplus) interface, developed by [diamondrod](https://github.com/diamondrod).
## IPC
The IPC feature enables [qipc](https://code.kx.com/q/basics/ipc/) communication between Rust and kdb+.
Connectivity is via TCP or Unix Domain Sockets, with support for both [compression](https://code.kx.com/q/basics/ipc/#compression) and [TLS encryption](https://code.kx.com/q/kb/ssl/) of messages.
Connection and listener methods are provided, enabling development of both
- Rust IPC clients of kdb+ server processes
- Rust IPC servers of kdb+ client processes
### Installation
Add `kxkdb` as a dependency, with feature `ipc`.
You may also want to add an asynchronous runtime such as [Tokio](https://tokio.rs).
e.g.
```toml
[dependencies]
kxkdb = { version = "0.0", features = ["ipc"] }
tokio = { version = "1.24", features = ["full"] }
```
### Examples
#### Client
```rust
use kxkdb::ipc::*;
use kxkdb::qattribute;
#[tokio::main]
async fn main() -> Result<()> {
let mut socket; // socket connection to kdb+ process
let mut result; // result of sync query to kdb+ process
let mut message; // compound list containing message
// connect via UDS to local kdb+ process listening on port 4321
socket = QStream::connect(ConnectionMethod::UDS, "", 4321_u16, "").await?;
// confirm connection type
println!("Connection type: {}", socket.get_connection_type());
// synchronously query kdb+ process using string
result = socket.send_sync_message(&"sum 1+til 100").await?;
println!("result1: {}", result);
// asynchronously define function in kdb+ process
socket.send_async_message(&"add_one:{x+1}").await?;
// synchronously call function (correctly)
result = socket.send_sync_message(&"add_one 41").await?;
println!("result2: {}", result);
// synchronously call function (incorrectly)
result = socket.send_sync_message(&"add_one`41").await?;
println!("result3: {}", result);
// synchronously query kdb+ process using compound list
message = K::new_compound_list(vec![K::new_symbol(String::from("add_one")), K::new_long(100)]);
result = socket.send_sync_message(&message).await?;
println!("result4: {}", result);
// asynchronously call show function in kdb+ process
message = K::new_compound_list(vec![K::new_string(String::from("show"), qattribute::NONE), K::new_symbol(String::from("hello from rust"))]);
socket.send_async_message(&message).await?;
// close socket
socket.shutdown().await?;
Ok(())
}
```
#### Server
Setup a credentials file containing usernames and (SHA-1 encrypted) passwords.
e.g.
```
$ cat userpass.txt
fred:e962cde7053eed120f928cd18e58ebd31be77543
homer:df43ad44d44e898f8f4e6ed91e6952bfce573e12
```
Note: Hashed passwords can be generated in q using `.Q.sha1`.
Store the path of this file in environment variable KDBPLUS_ACCOUNT_FILE.
e.g.
```
$ export KDBPLUS_ACCOUNT_FILE=`pwd`/userpass.txt
```
The following code will establish a Rust server process, listening on port 4321.
```rust
use kxkdb::ipc::*;
#[tokio::main]
async fn main() -> Result<()> {
let mut socket; // socket connection to kdb+ process
// listen for incoming TCP connections on port 4321
socket = QStream::accept(ConnectionMethod::TCP, "127.0.0.1", 4321).await?;
// when a connection is established, synchronously send a message to the client
let response = socket.send_sync_message(&"0N!string `Hello").await?;
println!("result: {}", response);
// close socket
socket.shutdown().await?;
Ok(())
}
```
A kdb+ client can then connect using the correct credentials.
e.g.
```q
q)hopen`:127.0.0.1:4321:fred:flintstone;
"Hello"
```
### Type Mapping
The following table displays the input types used to construct different q types (implemented as `K` objects).
| q | Rust |
|------------------|-------------------------------------------|
| `boolean` | `bool` |
| `guid` | `[u8; 16]` |
| `byte` | `u8` |
| `short` | `i16` |
| `int` | `i32` |
| `long` | `i64` |
| `real` | `f32` |
| `float` | `f64` |
| `char` | `char` |
| `symbol` | `String` |
| `timestamp` | `chrono::DateTime` |
| `month` | `chrono::NaiveDate` |
| `date` | `chrono::NaiveDate` |
| `datetime` | `chrono::DateTime` |
| `timespan` | `chrono::Duration` |
| `minute` | `chrono::Duration` |
| `second` | `chrono::Duration` |
| `time` | `chrono::Duration` |
| `list` | `Vec` (`T` a corresponding type above) |
| `compound list` | `Vec` |
| `table` | `Vec` |
| `dictionary` | `Vec` |
| `generic null` | `()` |
Note: The input type can differ from the inner type. For example, timestamp has an input type of `chrono::DateTime` but the inner type is `i64`, denoting an elapsed time in nanoseconds since `2000.01.01D00:00:00`.
### Environment Variables
#### KDBPLUS_ACCOUNT_FILE
Path to a credential file, used by a Rust server to manage access from kdb+ clients.
Contains a user name and SHA-1 hashed password on each line, delimited by `':'`.
#### KDBPLUS_TLS_KEY_FILE
The path to a pkcs12 file used for TLS connections.
#### KDBPLUS_TLS_KEY_FILE_SECRET
The password for the above pkcs12 file.
#### QUDSPATH
Defines the (real or abstract) path used for [Unix Domain Sockets](https://code.kx.com/q/basics/listening-port/#unix-domain-socket) to `$QUDSPATH/kx.[PORT]`.
n.b. If not defined, this will default to `/tmp/kx.[PORT]`
## API
The API feature enables the development of shared object libraries in Rust, which can be [dynamically loaded](https://code.kx.com/q/ref/dynamic-load/) into kdb+.
In order to avoid large `unsafe` blocks, most native C API functions are provided with a wrapper funtion and with intuitive implementation as a trait method. The exceptions are variadic functions `knk` and `k`, which are provided under `native` namespace with the other C API functions.
### Installation
Add `kxkdb` as a dependency, with feature `api`.
```toml
[dependencies]
kxkdb={version="0.0", features=["api"]}
```
### Examples
#### C API Style
```rust
use kxkdb::qtype;
use kxkdb::api::*;
use kxkdb::api::native::*;
#[no_mangle]
pub extern "C" fn create_symbol_list(_: K) -> K {
unsafe{
let mut list=ktn(qtype::SYMBOL_LIST as i32, 0);
js(&mut list, ss(str_to_S!("Abraham")));
js(&mut list, ss(str_to_S!("Isaac")));
js(&mut list, ss(str_to_S!("Jacob")));
js(&mut list, sn(str_to_S!("Josephine"), 6));
list
}
}
#[no_mangle]
pub extern "C" fn catchy(func: K, args: K) -> K {
unsafe{
let result=ee(dot(func, args));
if (*result).qtype == qtype::ERROR{
println!("error: {}", S_to_str((*result).value.symbol));
// Decrement reference count of the error object
r0(result);
KNULL
} else {
result
}
}
}
#[no_mangle]
pub extern "C" fn dictionary_list_to_table() -> K {
unsafe{
let dicts = knk(3);
let dicts_slice = dicts.as_mut_slice::();
for i in 0..3 {
let keys = ktn(qtype::SYMBOL_LIST as i32, 2);
let keys_slice = keys.as_mut_slice::();
keys_slice[0] = ss(str_to_S!("a"));
keys_slice[1] = ss(str_to_S!("b"));
let values = ktn(qtype::INT_LIST as i32, 2);
values.as_mut_slice::()[0..2].copy_from_slice(&[i*10, i*100]);
dicts_slice[i as usize] = xD(keys, values);
}
// Format list of dictionary as a table.
// ([] a: 0 10 20i; b: 0 100 200i)
k(0, str_to_S!("{[dicts] -1 _ dicts, (::)}"), dicts, KNULL)
}
}
```
A kdb+ process can then dynamically load and call these functions as follows:
```q
q)summon:`libc_api_examples 2: (`create_symbol_list; 1)
q)summon[]
`Abraham`Isaac`Jacob`Joseph
q)`Abraham`Isaac`Jacob`Joseph ~ summon[]
q)catchy: `libc_api_examples 2: (`catchy; 2);
q)catchy[$; ("J"; "42")]
42
q)catchy[+; (1; `a)]
error: type
q)behold: `libc_api_examples 2: (`dictionary_list_to_table; 1);
q)behold[]
a b
------
0 0
10 100
20 200
```
#### Rust Style
The examples below are written without `unsafe` code.
```rust
use kxkdb::qtype;
use kxkdb::api::*;
use kxkdb::api::native::*;
#[no_mangle]
pub extern "C" fn create_symbol_list2(_: K) -> K {
let mut list = new_list(qtype::SYMBOL_LIST, 0);
list.push_symbol("Abraham").unwrap();
list.push_symbol("Isaac").unwrap();
list.push_symbol("Jacob").unwrap();
list.push_symbol_n("Josephine", 6).unwrap();
list
}
#[no_mangle]
fn no_panick(func: K, args: K) -> K {
let result = error_to_string(apply(func, args));
if let Ok(error) = result.get_error_string() {
println!("FYI: {}", error);
// Decrement reference count of the error object which is no longer used.
decrement_reference_count(result);
KNULL
}
else{
println!("success!");
result
}
}
#[no_mangle]
pub extern "C" fn create_table2(_: K) -> K {
// Build keys
let keys = new_list(qtype::SYMBOL_LIST, 2);
let keys_slice = keys.as_mut_slice::();
keys_slice[0] = enumerate(str_to_S!("time"));
keys_slice[1] = enumerate_n(str_to_S!("temperature_and_humidity"), 11);
// Build values
let values = new_list(qtype::COMPOUND_LIST, 2);
let time = new_list(qtype::TIMESTAMP_LIST, 3);
// 2003.10.10D02:24:19.167018272 2006.05.24D06:16:49.419710368 2008.08.12D23:12:24.018691392
time.as_mut_slice::().copy_from_slice(&[119067859167018272_i64, 201766609419710368, 271897944018691392]);
let temperature = new_list(qtype::FLOAT_LIST, 3);
temperature.as_mut_slice::().copy_from_slice(&[22.1_f64, 24.7, 30.5]);
values.as_mut_slice::().copy_from_slice(&[time, temperature]);
flip(new_dictionary(keys, values))
}
```
And q code is here:
```q
q)summon:`libc_api_examples 2: (`create_symbol_list2; 1)
q)summon[]
`Abraham`Isaac`Jacob`Joseph
q)chill: `libc_api_examples 2: (`no_panick; 2);
q)chill[$; ("J"; "42")]
success!
42
q)chill[+; (1; `a)]
FYI: type
q)climate_change: libc_api_examples 2: (`create_table2; 1);
q)climate_change[]
time temperature
-----------------------------------------
2003.10.10D02:24:19.167018272 22.1
2006.05.24D06:16:49.419710368 24.7
2008.08.12D23:12:24.018691392 30.5
```
## Test
Testing is conducted in two ways:
1. Using cargo
2. Running a q test script
### 1. Using Cargo
Before starting the test, start a kdb+ process listening on port 5000.
```bash
$ q -p 5000
q)
```
Then run the test:
```bash
kxkdb]$ cargo test
```
**Note:** Currently 20 tests fails for `api` examples in document. This is because the examples do not have `main` function by nature of `api` but still use `#[macro_use]`.
### 2. Running a q Test Script
Tests are conducted with `tests/test.q` by loading the example functions built in `api_examples`.
```bash
kxkdb]$ cargo build --release
kxkdb]$ cp target/release/libapi_examples.so tests/
kxkdb]$ cd tests
tests]$ q test.q
Initialized something, probably it is your mindset.
bool: true
bool: false
byte: 0xc4
GUID: 8c6b-8b-64-68-156084
short: 10
int: 42
int: 122
int: 7336
int: 723
int: 14240
int: 2056636
long: -109210
long: 43200123456789
long: -325389000000021
long: 0
real: 193810.31
float: -37017.09330000
float: 742.41927468
char: "k"
symbol: `locust
string: "gnat"
string: "grasshopper"
error: type
What do you see, son of man?: a basket of summer fruit
What do you see, son of man?: boiling pot, facing away from the north
symbol: `rust
success!
FYI: type
this is KNULL
Planet { name: "earth", population: 7500000000, water: true }
Planet { name: "earth", population: 7500000000, water: true }
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
"Collect the clutter of apples!"
test result: ok. 147 passed; 0 failed
q)What are the three largest elements?: `belief`love`hope
```