https://github.com/bilbottom/billiam-database
OLAP database project for life admin (feat. dbt).
https://github.com/bilbottom/billiam-database
dbt duckdb sql
Last synced: 11 months ago
JSON representation
OLAP database project for life admin (feat. dbt).
- Host: GitHub
- URL: https://github.com/bilbottom/billiam-database
- Owner: Bilbottom
- Created: 2023-03-18T22:08:31.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-08-10T09:03:08.000Z (over 1 year ago)
- Last Synced: 2024-08-10T10:22:20.844Z (over 1 year ago)
- Topics: dbt, duckdb, sql
- Language: Shell
- Homepage: https://bilbottom.github.io/billiam-database/
- Size: 6.62 MB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
[](https://www.python.org/downloads/)
[](https://github.com/astral-sh/uv)
[](https://www.docker.com/)
[](https://github.com/Bilbottom/billiam-database/actions/workflows/tests.yaml)
[](https://shields.io/badges/git-hub-last-commit)
[](https://github.com/prettier/prettier)
[](https://results.pre-commit.ci/latest/github/Bilbottom/billiam-database/main)
[](https://duckdb.org/)
[](https://www.metabase.com/)
---
# Billiam's Database 🧙♂️
OLAP database project for life admin.
## About
As part of my life admin, I keep track of:
- Every transaction I make at an item level (since 2018-01-18)
- On the job, what I'm working on every 15 minutes (since 2019-04-23)
Note that the job tracker is my [daily-tracker](https://github.com/Bilbottom/daily-tracker) project.
Documentation is hosted at:
- https://bilbottom.github.io/billiam-database
## Setup
Since this is a personal project, I don't expect anyone else to do this, but I'm still documenting it for myself (I have a very, very bad memory).
### Pre-requisites
This project requires the following three tools to be installed:
- [Python](https://www.python.org/downloads/)
- [uv](https://docs.astral.sh/uv/getting-started/installation/)
- [Docker](https://www.docker.com/) (only if you want to use [Metabase](https://www.metabase.com/))
The required versions are specified in the badges at the top of this README.
### Installation (SQLMesh)
After cloning the repo, install the dependencies and enable [pre-commit](https://pre-commit.com/):
```shell
uv sync --all-groups
pre-commit install --install-hooks
```
Pipeline changes need to be applied via a plan:
```
sqlmesh -p billiam_database plan
```
...and the pipeline can be run with the `run` command:
```
sqlmesh -p billiam_database run
```
The SQLMesh UI is great for doing these with a GUI:
```
sqlmesh -p billiam_database ui
```
### Installation (Metabase)
> [!WARNING]
>
> The DuckDB driver for Metabase is a community driver. This means that it might not work in all circumstances.
[Metabase](https://www.metabase.com/) is a tool for visualising data.
In this project, Metabase is run in a Docker container; run the following command:
```shell
# start
docker-compose --file docker-compose.yaml --project-name billiam-database up --detach
# stop
docker-compose --file docker-compose.yaml --project-name billiam-database down --remove-orphans
```
This will make Metabase available at:
- [http://localhost:3000](http://localhost:3000)
> [!WARNING]
>
> There are two important notes about the current Metabase configuration:
>
> - Since [DuckDB](https://duckdb.org/) only supports one connection at a time, you can't use Metabase and run the pipelines at the same time.
> - The Metabase data is stored locally in the `dockerfiles/metaduck-data` directory so that it persists between container restarts.