https://github.com/sixthnormal/clj-3df
Clojure(Script) client for Declarative Dataflow.
https://github.com/sixthnormal/clj-3df
client clojure datalog differential-dataflows relational
Last synced: 3 months ago
JSON representation
Clojure(Script) client for Declarative Dataflow.
- Host: GitHub
- URL: https://github.com/sixthnormal/clj-3df
- Owner: sixthnormal
- License: epl-2.0
- Created: 2018-06-06T10:49:13.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2019-11-08T16:36:09.000Z (over 6 years ago)
- Last Synced: 2025-12-13T05:01:31.225Z (6 months ago)
- Topics: client, clojure, datalog, differential-dataflows, relational
- Language: Clojure
- Homepage:
- Size: 229 KB
- Stars: 332
- Watchers: 24
- Forks: 13
- Open Issues: 16
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# clj-3df
[](https://clojars.org/com.sixthnormal/clj-3df)
**This is alpha-quality software, not feature-complete, and not yet
ready for use in production services.**
3DF is best thought of as a pub/sub system in which subscriptions can
be arbitrary Datalog expressions. Subscribers register queries with
the broker, and data sources (such as Kafka, Datomic, or any other
source-of-truth) publish new data to it. All subscriber queries
affected by incoming data will be notified with a diff, describing how
their results have changed. The Datalog implementation is modeled
after [Datomic's query
language](https://docs.datomic.com/on-prem/query.html) and aims to
support the same set of features.
3DF does this efficiently, thanks to being built on top of
[differential
dataflows](https://github.com/frankmcsherry/differential-dataflow). In
particular, Differential Dataflow will only compute changes, rather
than execute a computation from scratch.
This repository contains the Clojure client for 3DF. The broker is
written in Rust and can be found in the [Declarative Differential
Dataflows](https://github.com/comnik/declarative-dataflow) repository.
## How it works
The 3DF client compiles Datalog expressions into an intermediate
representation that can be synthesised into a differential dataflow on
the server. This dataflow is then registered and executed across any
number of workers. Whenever query results change due to new data
entering the system, the server will push the neccessary changes via a
WebSocket connection.
For example, consider a subscriber created the following subscription:
``` clojure
(exec! conn
(query db "user inbox"
'[:find ?msg ?content
:where
[?msg :msg/recipient "me@nikolasgoebel.com"]
[?msg :msg/content ?content]]))
```
and a new message arrives in the system.
``` clojure
[{:msg/receipient "me@nikolasgoebel.com"
:msg/content "Hello!"}]
```
Then the server will push the following results to the subscriber:
``` clojure
[[[ "Hello!"] +1]]
```
If at some later point in time, this message was retracted
``` clojure
[[:db/retractEntity ]]
```
the server would again notify the subscriber, this time indicating the
retraction:
``` clojure
[[[ "Hello!"] -1]]
```
This guarantees, that subscribers maintaining any form of functionally
derived information will always have a consistent view of the data.
## Query Language Features
- [x] Implicit joins and unions, `and` / `or` operators
- [x] Stratified negation
- [x] Parameterized queries
- [x] Rules, self-referential / mutually recursive rules
- [x] Aggregation (min, max, count, etc...)
- [x] Grouping via `:with`
- [x] Basic predicates (<=, <, >, >=, =, not=)
- [ ] As-of queries
- [ ] More find specifications (e.g. collection, scalar)
- [ ] Pull queries
- [x] Queries across many heterogeneous data sources
Please also have a look at the open issues to get a sense for what
we're working on.
## Non-Features
3DF is neither concerned with durability nor with consistency in the
ACID sense. It is intended to be used in combination with a
consistent, durable source-of-truth such as
[Datomic](https://www.datomic.com/) or
[Kafka](https://kafka.apache.org/).
Consequently, 3DF will accept whatever tuples it is supplied with. For
example, whereas in Datomic two subsequent transactions on an empty
database
``` clojure
(d/transact conn [[:db/add 123 :user/balance 1000]])
...
(d/transact conn [[:db/add 123 :user/balance 1500]])
```
would result in the following sets of datoms being added into the
database:
``` clojure
[[123 :user/balance 1000 true]]
...
[[123 :user/balance 1000 false]
[123 :user/balance 1500 true]]
```
3DF will by itself not take any previous information into account on
transactions. Again, 3DF is intended to be fed data from a system like
Datomic, which would ensure that transactions produce consistent
tuples.
## License
Copyright © 2018 Nikolas Göbel
Licensed under Eclipse Public License (see [LICENSE](LICENSE)).