https://github.com/chifisource/chidb.jl
toolips + algebra streams = database server
https://github.com/chifisource/chidb.jl
Last synced: 4 months ago
JSON representation
toolips + algebra streams = database server
- Host: GitHub
- URL: https://github.com/chifisource/chidb.jl
- Owner: ChifiSource
- License: mit
- Created: 2024-01-02T16:29:33.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-07-26T05:35:41.000Z (11 months ago)
- Last Synced: 2025-07-26T11:21:16.651Z (11 months ago)
- Language: Julia
- Homepage:
- Size: 311 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
###### a toolips-based data-base server
`ChiDB` is a unique data-base server designed around the `AlgebraFrames` concept and the `.ff` file format. Schema is laid using directories and filenames and data is live-read into memory. This is currently in a state of relative infancy, but is primarily being developed for my own use cases and to demonstrate the various server-building capabilities of `Toolips`.
- [get started](#get-started)
- [adding chidb](#adding)
- [documentation](#documentation)
- [chidb setup](#setup)
- [schema](#schema)
- [feature files](#feature-files)
- [readable data-types](#readable-data-types)
- [editing schema](#editing-schema)
- [querying](#querying)
- [clients](#clients)
- [command list](#commands)
- [query examples](#example-queries)
- [creating chidb clients](#creating-clients)
- [existing clients](#existing-clients)
- [chidb headers](#headers)
- [header RFC](#opcodes)
## get started
#### adding
#### documentation
## setup
In order to use `ChiDB`, we first need [julia](https://julialang.org). With Julia installed, the package may be added with `Pkg`:
```julia
using Pkg; Pkg.add(url = "https://github.com/ChifiSource/ChiDB.jl")
using ChiDB
```
To setup a `ChiDB` server directory, run `ChiDB.start` and provide an **empty** directory as `path`. This will create a new folder called `db`, which will contain the data-bases core information.
```julia
start(path::String, ip::IP4 = "127.0.0.1":8005)
```
Our `admin` login will also be printed here; by querying with this new `admin` login we may create new users.
#### schema
`ChiDB` schema is created using two different techniques:
- *querying*
- or by creating a filesystem of `.ff` and `.ref` files.
`ChiDB`'s internal data is primarily stored in the `.ff` (*feature file*) format. There are no sub-tables, only reference columns. Both references and `.ff` feature files are represented by files, and the tables they reside in are represented by directories. In order to create schema, we would simply add new folders with new `.ff` files for each column to our new data-base directory. Consider the following sample directory structure:
- project directory
- /db
- /table1
- /col1.ff
- /col2.ff
- /table2_col1.ref
- /table2
- /col1.ff
##### feature files
Each `.ff` file's first line will be a readable data-type. For example, `col1.ff` from above could be
```ff
Integer
```
###### readable data-types
### querying
##### clients
###### commands
Commands are issued to the server using
- `()` indicates an optional argument.
- `(table)/column` indicates the ability to provide the column if a table is selected, *or* provide a column and table in the `table/column` format. For example:
```julia
write!(sock, "$(curr_header)vnewt/name|!|1|!|frank\n")
```
character
name
description
arguments
l
list
lists the columns within a table, and their types, or lists all tables when provided with no argument
(table)
s
select
Selects a table.
table
t
create
creates a new table
tablename
get-store commands
g
get
gets values using vertical indexing
(table)/column (range)
r
getrow
Gets a full row of data
(table)/column rown
i
index
Gets the index where a certain value occurs in a given table.
(table)/column value
a
store
Stores values, separated by `!;`, into a given table. Will return an argument error if the incorrect shape is provided.
(table) value!;value2
v
set
Sets a singular value in a table.
(table)/column row value
w
setrow
Sets the values of an entire row on a table
(table) row value1!;value2
column management
j
join
Adds a new column to a frame, creates a reference column when used with a column path from another table.
(table) (reftable)/colname (Type)
k
type
Attempts to cast a given type to a provided column.
(table)/colname Type
e
rename
Renames a given column or table
(table) table_or_colname
deleters
d
deleteat
Deletes a row from a given table.
(table) row
z
delete
Deletes a table
table
built-in operations
p
compare
Checks if the provided value is the same as the stored data.
(table)/column rown value
n
in
Checks if the provided value is within the column.
(table)/column value
server
U
users
Lists current users.
C
newuser
Creates a new user. Will return the new name, password, and dbkey.
user (pwd)
K
setuser
Sets any users login -- must be admin to perform.
user, name (pwd)
D
rmuser
Removes a user by name. Must be admin to perform.
username
L
logout
Disconnects from the server.
#### query examples
---
## creating clients
#### existing clients
#### headers
You will likely want an *API* of some sort to query a `ChiDB` servers. Every query, including your initial query, will be sent with a two-byte header. This header includes three fields: the `opcode` (4 bits), the `transaction id` (4 bits) (composing the first byte) and the second byte (8 bits) is dedicated to the *command character* -- a single-character reference that requests a query command.
- The `opcode` returns a success code dependent on the status of the last query. See [opcodes](#opcodes) for a full list of opcodes.
- The `transaction id` will be the ID of the transaction that is just issued. This needs to be sent **back** to the server on each query, and will need to be wrapped into each header we send to the server.
```text
Request header
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16
| opcode | transac ID | command byte |
Response header
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| opcode | transac ID |
```
So, from the API's perspective both the opcode and transaction ID are sent back to the server, meaning we can just send back the first character we get as our response as the header. The second char would then be our selected query command, and from there our arguments are provided directly and separated by `|!|`. For the initial connection the first character would be nothing, and for `S` we provide spaces as separators.
```julia
using Toolips
sock = Toolips.connect("127.0.0.1":2025)
write!(sock, "nS dbkey admin adminpwd\n")
resp = String(readavailable(sock))
# (opcode)
eightbit_header = UInt8(resp[1])
header = bitstring(eightbit_header)
if header[1:4] == "0001"
println("query accepted!")
else
throw("not verified, the query will not work")
end
# list tables:
write!(sock, string(eightbit_header) * "l\n")
resp = String(readavailable(sock))
println(resp)
```
###### opcodes
code
status
name
has output
0001
OK
query accept
false
0011
OK
user created
false
0101
OK
password set
false
1000
ERROR
bad packet
false
1100
ERROR
login denied (connection closed)
false
1001
ERROR
bad dbkey (connection closed)
false
1110
ERROR
command error
true
1010
ERROR
argument error
true
1111
ERROR
bad transaction (connection closed)
false