Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/semcod/dbos


https://github.com/semcod/dbos

Last synced: 8 days ago
JSON representation

Awesome Lists containing this project

README 

          
# Platform OS



A schema-driven runtime for data and applications. Not a backend, not an app.

PostgreSQL holds meaning, MIME-typed content tables hold payloads, and a set

of small services consume them through a published contract.



## What this actually is



Everything — devices, articles, scenarios, images, HTML pages, protocols —

lives as an `entities` row plus exactly one row in a `content_*` table

chosen by MIME. There's one content table per MIME type, which means any

service that cares about one MIME doesn't have to know about the others,

and you can `pg_dump -t content_markdown` to take your articles to a

different project without dragging along the rest of the database.



On top of the data there are:



- **sync-engine** — watches `./data/`, parses files by extension, routes each to the right `content_*` table

- **api-gateway** — generic `/api/entities/:id` that joins `entities` with the correct content table at runtime; plus CRUD for `storage_backends`, `protocol_gateways`, `inbound_sources`

- **command-bus** — persists every command, routes to a worker by `schemas.target_runtime`

- **worker-python** / **worker-php** — execute business logic

- **gen-jinja** (Python), **gen-twig** (PHP), **gen-handlebars** (Node) — three website generators, each reading the same tables, each declaring what it can render in `schemas.renderers[]`

- **ui-runtime** — schema-driven frontend

- **nginx-cdn** — reverse proxy with preview caching

- **vfs-webdav** — WebDAV skin over the same tables (mount from anywhere)

- **vfs-fuse** — real Linux FUSE mount

- **vfs-ftp** — FTP server (`:2121`) with passive mode support

- **vfs-imap** — IMAP4rev1 server (`:1143`) exposing entities as mailboxes/messages

- **vfs-pop3** — POP3 server (`:1110`) with UIDL/RETR/DELE

- **vfs-smtp** — SMTP gateway (`:2525`) ingesting mail as `mail/*` entities

- **storage-mirror** — tails `audit_log` and replays writes into SQLite / MySQL mirrors

- **sync-outbound** — reads `service_mappings` and pushes matching entities to filesystem targets (e.g. `./email/*.eml`)

- **connectors** — inbound pollers (filesystem, IMAP-pull, FTP-pull, SQL-mirror)



## Layout



```

platform/

├─ docker-compose.yml

├─ Makefile                       stack lifecycle, tests, runnable examples

├─ .env.example

├─ postgres/init/                 7 SQL files run on first boot

│  ├─ 01_core.sql                 extensions, users, ACL, audit_log

│  ├─ 02_registry.sql             schemas, mime_types, schema_paths, filesystem_map

│  ├─ 03_content_tables.sql       entities + content_json/yaml/xml/html/markdown/binary

│  ├─ 04_schemas_seed.sql         JSON Schemas + renderer declarations

│  ├─ 05_demo_data.sql            first rows in every content table

│  ├─ 06_connectors.sql           storage_backends, protocol_gateways, inbound_sources

│  └─ 07_mail_schema.sql          mail schema + schema_paths seed for SMTP/IMAP/POP3

├─ scripts/

│  ├─ demo.sh                     end-to-end walkthrough

│  ├─ export-service.sh           extract one service + its tables for reuse

│  └─ db-upgrade-protocols.sh     apply registry + mail schema to existing Postgres volumes

├─ libs/platform_storage/         shared storage contract + adapters (Pg, SQLite, MySQL)

├─ sync-engine/                   filesystem -> DB, MIME-routed

├─ api-gateway/                   REST + ACL + schema validation + registry CRUD

├─ command-bus/                   routes commands to workers by target_runtime

├─ workers/

│  ├─ python/                     FastAPI; handles create_device, etc.

│  └─ php/                        PHP CLI server; handles render_page

├─ generators/                    website generators (same contract, 3 techs)

│  ├─ gen-jinja/                  Python + FastAPI + Jinja2 + markdown

│  ├─ gen-twig/                   PHP + Twig

│  └─ gen-handlebars/             Node + Express + Handlebars

├─ ui-runtime/                    vanilla JS schema-driven UI

├─ cdn/nginx.conf                 reverse proxy

├─ vfs-webdav/                    WebDAV frontend over content_* tables

├─ vfs-fuse/                      FUSE frontend over content_* tables

├─ vfs-ftp/                       FTP gateway (passive mode, masquerade support)

├─ vfs-imap/                      IMAP4rev1 gateway

├─ vfs-pop3/                      POP3 gateway

├─ vfs-smtp/                      SMTP ingest gateway

├─ storage-mirror/                audit_log tailer -> SQLite/MySQL mirrors

├─ sync-outbound/                   service_mappings -> filesystem export

├─ connectors/                    inbound poller daemons

├─ sdk/                           Python / JS / PHP clients

├─ examples/                      runnable integration scenarios

│  ├─ 01-write-http-read-protocols/

│  ├─ 02-smtp-to-platform/

│  ├─ 03-sqlite-mirror/

│  ├─ 04-connectors-registry/

│  └─ 05-everything/

├─ data/                          watched by sync-engine (nested dirs supported)

│  ├─ articles/*.md

│  ├─ devices/*.json

│  ├─ pages/*.html

│  ├─ pages/page2/*.html         nested directory example

│  ├─ scenarios/*.yaml

│  └─ protocols/*.xml

├─ logs/                          info.txt, warnings.txt, error.txt

└─ mirror-data/                   SQLite mirror default path

```



## Quickstart



```bash

cp .env.example .env

make up-all          # core + protocols + connectors + mirrors

```



What happens on first boot:



1. Postgres runs the 7 init SQL files (schemas, demo data, registry, mail schema)

2. `sync-engine` picks up every file in `./data/` and writes it to the matching content table

3. All three generators wake up and publish `/capabilities` from `schemas.renderers[]`

4. UI at  (login `admin@platform.local` / `demo1234`)

   - File Browser: `#files` — tree, table, manager, grid views

   - Thumbnails: 64x64 PNG generated from file content (colors per MIME type)

   - Icon size control: 16px–64px, row height: compact/normal/spacious

   - Inline HTML preview + schema preview on entity pages

   - JSON syntax highlighting

5. CDN at , WebDAV at `http://localhost:8090`

6. Protocol gateways: FTP `:2121`, IMAP `:1143`, POP3 `:1110`, SMTP `:2525`

7. Logs in `./logs/` for debugging services



Then from a second terminal:



```bash

make examples        # run all 5 integration scenarios

./scripts/demo.sh    # end-to-end walkthrough

```



If you already have an old Postgres volume, run `make db-upgrade-protocols` before

starting the new services so registry tables and the mail schema exist.



## File Browser UI



The UI at `#files` provides 4 views for browsing entities:



| View      | URL hash                | Description                                    |

| --------- | ----------------------- | -----------------------------------------------|

| Tree      | `#files/tree`           | Hierarchical tree with expandable folders        |

| Table     | `#files/table`          | Columns: path, type, schema, MIME, updated, status |

| Manager   | `#files/manager`        | Two-column file manager with drag-and-drop move + edit/delete |

| Grid      | `#files/grid`           | Grouped by folder, thumbnail + filename cards  |



Controls:

- **Icon Size**: 16px / 24px / 32px / 48px / 64px — adjusts thumbnails everywhere

- **Row Height**: compact / normal / spacious — table and tree spacing



## Nested filesystem paths



Files in subdirectories are supported. `data/pages/page2/welcome.html` is stored with `external_id = pages/page2/welcome` (slash-separated). The UI displays paths as on disk.



```bash

# This entity exists at the nested path

curl -H "Authorization: Bearer $TOKEN" \

  "http://localhost:3000/api/entity?external_id=pages/page2/welcome"

```



## Thumbnails



Every file change triggers 64x64 PNG thumbnail generation via `sharp`:

- **Images**: resized actual image

- **Text/HTML/JSON/YAML/XML**: colored squares (color per MIME) with dot pattern indicating content length



Thumbnails are stored in the `thumbnails` table and served from `GET /api/thumbnail?external_id=...&size=64px`.



## Mount the platform as a filesystem



### WebDAV (easiest — works on Linux/Mac/Windows)



```bash

# Linux

sudo apt install davfs2

sudo mkdir -p /mnt/vfs

sudo mount -t davfs http://localhost:8090 /mnt/vfs

# user: admin  password: admin  (from .env)



ls /mnt/vfs

#   articles/  customers/  devices/  images/  pages/  protocols/  scenarios/



cat /mnt/vfs/articles/hello-platform.md      # streams content_markdown.body

echo "# new" > /mnt/vfs/articles/test.md     # upserts into content_markdown

```



macOS: Finder → *Go* → *Connect to Server* → `http://localhost:8090`

Windows: Explorer → *This PC* → *Map network drive* → `http://localhost:8090`



### FUSE (real Linux mount)



Only on a Linux host with `/dev/fuse`:



```bash

docker compose --profile fuse up -d vfs-fuse

ls ./vfs-mount/                              # shares the same folder with the container

```



Inside the container the mount is at `/mnt/vfs`, bind-mounted to

`./vfs-mount/` on the host so other processes see it too.



## The reusability recipe



Say you want `gen-jinja` in a completely different project next week:



```bash

./scripts/export-service.sh gen-jinja

# -> ./export/gen-jinja/

#       service/                source + Dockerfile

#       bundle.sql              schemas + mime_types + content_markdown + content_html

#       docker-compose.snippet  service block for the new project

```



Drop `bundle.sql` into the new project's DB, paste the compose snippet,

`docker compose up`. On first hit the service queries `mime_types` and

`schemas`, sees what it's registered for, starts handling requests. No code

change.



## Developing



### Watch a file go from disk to DB



```bash

cat > data/articles/my-post.md < |  storage-mirror    |

 python/php                               |  SQLite / MySQL    |

                                          +--------------------+

                                       ^

                                       |

                           +---------------------+

                           |     sync-engine     |

                           +----------+----------+

                                      |

                                   ./data/*

```



## Cheat sheet



| What                              | How                                                                 |

| --------------------------------- | ------------------------------------------------------------------- |

| Log in                            | `POST /auth/login  {email, password}` → JWT                         |

| List entities                     | `GET /api/entities?entity_type=device&limit=200`                    |

| Fetch entity + content            | `GET /api/entities/:external_id`  or `GET /api/entity?external_id=` |

| View rendered HTML                | `GET /api/entities/:external_id/html`                               |

| Thumbnail (64x64 PNG)             | `GET /api/thumbnail?external_id=pages/page2/welcome&size=64px`      |

| Update entity (move/edit)         | `PATCH /api/entity?external_id=... {external_id, content}`          |

| Delete entity                     | `DELETE /api/entity?external_id=...`                                |

| File Browser views                | `#files/tree`, `#files/table`, `#files/manager`, `#files/grid`      |

| Inspect routing                   | `GET /mime-types`  and  `GET /schemas`                              |

| Who renders schema X?             | `SELECT renderers FROM schemas WHERE id = 'article_v1';`            |

| Run a command                     | `POST /commands/create_device  {...}`                               |

| Audit                             | `GET /audit` (admin only)                                           |

| Registry CRUD (backends/gateways/sources) | `GET|POST|PATCH|DELETE /api/storage-backends` / `protocol-gateways` / `inbound-sources` |

| Mount as filesystem               | `mount -t davfs http://localhost:8090 /mnt/vfs`                     |

| Export one service                | `./scripts/export-service.sh gen-jinja`                             |

| View logs                         | `cat ./logs/info.txt`  `./logs/warnings.txt`  `./logs/error.txt`    |

| Run all examples                  | `make examples`                                                     |

| Upgrade existing DB               | `make db-upgrade-protocols`                                         |

| Install dev tools                 | `make install-dev`  (taskfile + testql)                             |

| List taskfile tasks               | `taskfile list`                                                     |

| Run TestQL smoke tests            | `testql run testql-scenarios/*.testql.toon.yaml`                    |



## Conventions that make this work



1. **Every entity declares a MIME.** `entities.primary_mime` points at

   `mime_types` which points at a content table. No service hardcodes

   which table holds what — they read the registry.

2. **Schemas carry `renderers[]`.** Generators advertise compatibility via

   data, not a central registry file.

3. **Filesystem layout is `schemas.fs_path_template`.** Change the

   template, the WebDAV/FUSE/sync-engine layouts all update together.

4. **Sources are always tracked.** Every row has `source ∈

   {filesystem, api, command, generator, system}`, and so does every

   `audit_log` entry. You always know who wrote what.



## Uniform protocols & storage plug-ins



Every protocol surface (WebDAV, FUSE, FTP, IMAP, POP3, SMTP) and every

inbound connector (filesystem, IMAP-pull, FTP-pull, SQL-mirror) implements

the same `EntityStore` contract from `libs/platform_storage/`. Add a new

protocol by writing a thin wrapper over that class — the data contract,

MIME routing and audit trail stay identical.



### Four registry tables



| Table               | What lives there                                      |

| ------------------- | ----------------------------------------------------- |

| `storage_backends`  | Databases/object-stores: postgres, sqlite, mysql, …   |

| `protocol_gateways` | Outbound surfaces: webdav, ftp, imap, pop3, smtp, …   |

| `inbound_sources`   | Pullers: filesystem, IMAP mailboxes, FTP dirs, SQL    |

| `service_mappings`  | Data-flow routing: source → target (DB → filesystem)    |



Seeded on first boot (`06_connectors.sql` + `08_service_mappings.sql`); add rows at runtime via UI or API.



### SDK (Python / JavaScript / PHP)



Programmatic access to entities, registry, and config:



```python

from dbos_client import DBOSClient

db = DBOSClient('http://localhost:3000')

db.login('admin@platform.local', 'demo1234')

db.create_entity('mail/hello', 'mail', 'mail_v1', {'title':'Hello','body':'World'})

db.create_service_mapping('imap-to-email', 'pg-primary', 'filesystem-email',

                          filter={'entity_type':'mail'},

                          transform={'format':'rfc5322','extension':'.eml'})

```



See `sdk/python/dbos_client.py`, `sdk/js/dbos_client.js`, `sdk/php/DBOSClient.php`.



### Start the extra protocols



```bash

docker compose --profile protocols up -d

#   vfs-ftp   :2121     (FTP, user/pass from FTP_USER/FTP_PASS)

#   vfs-imap  :1143     (IMAP4rev1)

#   vfs-pop3  :1110     (POP3)

#   vfs-smtp  :2525     (accepts incoming mail → entities)



docker compose --profile outbound up -d

#   sync-outbound       (exports DB entities → ./email/*.eml via service_mappings)

```



Example — read articles over IMAP with a mail client or `openssl`:



```bash

# Any IMAP client works. Account = localhost:1143, user 'admin', pass 'admin'.

# Mailbox "articles" lists every markdown entity; each message body IS the

# markdown, with entity metadata in RFC-5322 headers.

```



Example — upload a file via FTP, see it in the same DB:



```bash

lftp -u admin,admin localhost:2121

cd articles

put my-post.md      # -> content_markdown via platform_storage, source='ftp'

```



### Start the inbound pullers



```bash

docker compose --profile connectors up -d

```



Then register a remote source at runtime:



```sql

INSERT INTO inbound_sources (id, driver, endpoint, credentials_ref,

                             target_mime, id_template, config) VALUES

('support-mail', 'imap', 'imaps://mail.example.com:993', 'SUPPORT',

 'text/markdown', 'mail/{remote_id}',

 '{"mailbox":"INBOX","limit":50}');

```



Put `SUPPORT_USER`/`SUPPORT_PASS` in the env and the `connector-imap-pull`

daemon will begin polling it every minute, writing messages as `mail/*`

entities via the same `EntityStore.write()` used by `vfs-smtp`.



### Mirror to a different database



Declare a secondary backend and start the mirror daemon:



```sql

INSERT INTO storage_backends (id, driver, role, dsn) VALUES

('sqlite-mirror', 'sqlite', 'mirror', 'sqlite:///mirror-data/platform.sqlite');

```



```bash

docker compose --profile mirrors up -d storage-mirror

```



The daemon tails `audit_log` and replays every write into each mirror.

Both MySQL (`driver='mysql'`) and SQLite (`driver='sqlite'`) are

implemented; adding Mongo/Redis is one adapter class.



## What's deliberately not here



This is a usable dev scaffold, not a production stack. Things a production

deployment would add:



- Redis / Kafka for real async queues (commands currently execute inline)

- Object store (MinIO/S3) for `content_binary` when `storage_mode='object_store'`

- Observability (OTel traces, metrics, structured logs)

- Proper secrets management for JWT + DB credentials

- Connection pooling beyond what the Node/Python drivers provide by default

- Kubernetes manifests mirroring the compose topology



## License



Licensed under Apache-2.0.



## Status



_Last updated by [taskill](https://github.com/oqlos/taskill) at 2026-04-25 13:37 UTC_



| Metric | Value |

|---|---|

| HEAD | `d28c580` |

| Coverage | — |

| Failing tests | — |

| Commits in last cycle | 5 |



> Added documentation for a code analysis engine and a configuration management system; also introduced example files and additional service components.