Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/lambdaisland/metabase-datomic

Datomic driver for Metabase
https://github.com/lambdaisland/metabase-datomic

business-intelligence data-visualization metabase

Last synced: about 1 month ago
JSON representation

Datomic driver for Metabase

Awesome Lists containing this project

README 

        
# metabase-datomic



A Metabase driver for Datomic.



Commercial support is provided by [Gaiwan](http://gaiwan.co).



## Try it!



```

docker run -p 3000:3000 lambdaisland/metabase-datomic

```



## Design decisions



See the [Architecture Decision Log](doc/architecture_decision_log.org)



## Developing



To get a REPL based workflow, do a git clone of both `metabase` and

`metabase-datomic`, such that they are in sibling directories



``` shell

$ git clone [email protected]:metabase/metabase.git

$ git clone [email protected]:plexus/metabase-datomic.git



$ tree -L 1

.

│

├── metabase

└── metabase-datomic

```



Before you can use Metabase you need to build the frontend. This step you only

need to do once.



``` shell

cd metabase

yarn build

```



And install metabase locally



``` shell

lein install

```



Now `cd` into the `metabase-datomic` directory, and run `bin/start_metabase` to

lauch the process including nREPL running on port 4444.



``` shell

cd metabase-datomic

bin/start_metabase

```



Now you can connect from Emacs/CIDER to port 4444, or use the

`bin/cider_connect` script to automatically connect, and to associate the REPL

session with both projects, so you can easily evaluate code in either, and

navigate back and forth.



Once you have a REPL you can start the web app. This also opens a browser at

`localhost:3000`



``` clojure

user=> (go)

```



The first time it will ask you to create a user and do some other initial setup.

To skip this step, invoke `setup!`. This will create a user with username

`[email protected]` and password `dev`. It will also create a Datomic database

with URL `datomic:free://localhost:4334/mbrainz`. You are encouraged to run a

datomic-free transactor, and

[import the MusicBrainz](https://github.com/Datomic/mbrainz-sample)

database for testing.



``` clojure

user=> (setup!)

```



## Installing



The general process is to build an uberjar, and copy the result into

your Metabase `plugins/` directory. You can build a jar based on

datomic-free, or datomic-pro (assuming you have a license). Metabase

must be available as a local JAR.



``` shell

cd metabase

lein install

mkdir plugins

cd ../metabase-datomic

lein with-profiles +datomic-free uberjar

# lein with-profiles +datomic-pro uberjar

cp target/uberjar/datomic.metabase-driver.jar ../metabase/plugins

```



Now you can start Metabase, and start adding Datomic databases



``` shell

cd ../metabase

lein run -m metabase.core

```



## Configuration EDN



When you configure a Datomic Database in Metabase you will notice a config field

called "Configuration EDN". Here you can paste a snippet of EDN which will

influence some of the Driver's behavior.



The EDN needs to represent a Clojure map. These keys are currently understood



- `:inclusion-clauses`

- `:tx-filter`

- `:relationships`



Other keys are ignored.



### `:inclusion-clauses`



Datomic does not have tables, but nevertheless the driver will map your data to

Metabase tables based on the attribute names in your schema. To limit results to

the right entities it needs to do a check to see if a certain entity logically

belongs to such a table.



By default these look like this



``` clojure

[(or [?eid :user/name]

     [?eid :user/password]

     [?eid :user/roles])]

```



In other words we look for entities that have any attribute starting with the

given prefix. This can be both suboptimal (there might be a single attribute

with an index that is faster to check), and it may be wrong, depending on your

setup.



So we allow configuring this clause per table. The configured value should be a

vector of datomic clauses. You have the full power of datalog available. Use the

special symbol `?eid` for the entity that is being filtered.



``` clojure

{:inclusion-clauses {"user" [[?eid :user/handle]]}}

```



### `:tx-filter`



The `datomic.api/filter` function allows you to get a filtered view of the

database. A common use case is to select datoms based on metadata added to

transaction entities.



You can set `:tx-filter` to any form that evaluates to a Clojure function. Make

sure any namespaces like `datomic.api` are fully qualified.



``` clojure

{:tx-filter

 (fn [db ^datomic.Datom datom]

   (let [tx-user (get-in (datomic.api/entity db (.tx datom)) [:tx/user :db/id])]

     (or (nil? tx-tenant) (= 17592186046521 tx-user))))}

```



### `:rules`



This allows you to configure Datomic rules. These then become available in the

native query editor, as well in `:inclusion-clauses` and `:relationships`.



``` clojure

{:rules

 [[(sub-accounts ?p ?c)

   [?p :account/children ?c]]

  [(sub-accounts ?p ?d)

   [?p :account/children ?c]

   (sub-accounts ?c ?d)]]}

```



### `:relationships`



This features allows you to add "synthetic foreign keys" to tables. These are

fields that Metabase will consider to be foreign keys, but in reality they are

backed by an arbitrary lookup path in Datomic. This can include reverse

reference (`:foo/_bar`) and rules.



To set up an extra relationship you start from the table where you want to add

the relationship, then give it a name, give the path of attributes and rules

needed to get to the other entity, and specifiy which table the resulting entity

belongs to.



``` clojure

{:relationships

 {;; foreign keys added to the account table

  :account

  {:journal-entry-lines

   {:path [:journal-entry-line/_account]

    :target :journal-entry-line}



   :subaccounts

   {:path [sub-accounts]

    :target :account}



   :parent-accounts

   {:path [_sub-accounts] ;; apply a rule in reverse

    :target :account}}



  ;; foreign keys added to the journal-entry-line table

  :journal-entry-line

  {:fiscal-year

   {:path [:journal-entry/_journal-entry-lines

           :ledger/_journal-entries

           :fiscal-year/_ledgers]

    :target :fiscal-year}}}}

```



## Status



FeatureSupported?Basics    {:source-table integer-literal}Yes    {:fields [& field]}Yes        [:field-id field-id]Yes        [:datetime-field local-field | fk unit]Yes    {:breakout [& concrete-field]}Yes        [:field-id field-id]Yes        [:aggregation 0]Yes        [:datetime-field local-field | fk unit]Yes    {:filter filter-clause}        [:and & filter-clause]Yes        [:or & filter-clause]Yes        [:not filter-clause]Yes        [:= concrete-field value & value]Yes        [:!= concrete-field value & value]Yes        [:< concrete-field orderable-value]Yes        [:> concrete-field orderable-value]Yes        [:<= concrete-field orderable-value]Yes        [:>= concrete-field orderable-value]Yes        [:is-null concrete-field]Yes        [:not-null concrete-field]Yes        [:between concrete-field min orderable-value max orderable-value]Yes        [:inside lat concrete-field lon concrete-field lat-max numeric-literal lon-min numeric-literal lat-min numeric-literal lon-max numeric-literal]Yes        [:starts-with concrete-field string-literal]Yes        [:contains concrete-field string-literal]Yes        [:does-not-contain concrete-field string-literal]Yes        [:ends-with concrete-field string-literal]Yes        [:time-interval field concrete-field n :current|:last|:next|integer-literal unit relative-datetime-unit]    {:limit integer-literal}Yes    {:order-by [& order-by-clause]}Yes:basic-aggregations    {:aggregation aggregation-clause}        [:count]Yes        [:count concrete-field]Yes        [:cum-count concrete-field]Yes        [:cum-sum concrete-field]Yes        [:distinct concrete-field]Yes        [:sum concrete-field]Yes        [:min concrete-field]Yes        [:max concrete-field]Yes        [:share filter-clause]:standard-deviation-aggregationsYes    {:aggregation aggregation-clause}Yes        [:stddev concrete-field]Yes:foreign-keysYes    {:fields [& field]}Yes        [:fk-> fk-field-id dest-field-id]Yes:nested-fields:set-timezone:expressions    {:fields [& field]}        [:expression]    {:breakout [& concrete-field]}        [:expression]    {:expressions {expression-name expression}}:native-parameters:expression-aggregations:nested-queriesYes    {:source-query query}Yes:binning:case-sensitivity-string-filter-optionsYes



## License



Copyright © 2019 Arne Brasseur



Licensed under the term of the Mozilla Public License 2.0, see LICENSE.