{"id":51503681,"url":"https://github.com/puffball1567/flowlogbook","last_synced_at":"2026-07-07T22:05:45.092Z","repository":{"id":369099105,"uuid":"1287083486","full_name":"puffball1567/flowlogbook","owner":"puffball1567","description":"Run record, resume decision, and reuse-history primitives for FlowBrigade-style workflows.","archived":false,"fork":false,"pushed_at":"2026-07-03T14:51:11.000Z","size":30,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-03T16:29:37.951Z","etag":null,"topics":["execution-ledger","flow-control","flowbrigade","flowbrigade-toolkit","nim","postgresql","redis","resumable-workflows","sqlite","task-cache","valkey","workflow","workflow-engine"],"latest_commit_sha":null,"homepage":null,"language":"Nim","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/puffball1567.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-02T11:12:25.000Z","updated_at":"2026-07-03T14:56:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/puffball1567/flowlogbook","commit_stats":null,"previous_names":["puffball1567/flowlogbook"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/puffball1567/flowlogbook","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puffball1567%2Fflowlogbook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puffball1567%2Fflowlogbook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puffball1567%2Fflowlogbook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puffball1567%2Fflowlogbook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/puffball1567","download_url":"https://codeload.github.com/puffball1567/flowlogbook/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/puffball1567%2Fflowlogbook/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35244097,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["execution-ledger","flow-control","flowbrigade","flowbrigade-toolkit","nim","postgresql","redis","resumable-workflows","sqlite","task-cache","valkey","workflow","workflow-engine"],"created_at":"2026-07-07T22:05:44.397Z","updated_at":"2026-07-07T22:05:45.083Z","avatar_url":"https://github.com/puffball1567.png","language":"Nim","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FlowLogbook\n\nFlowLogbook is a small Nim library for recording repeatable task executions and\nmaking resume decisions.\n\nIt is not a workflow engine. It does not schedule tasks, run containers, or\nmanage a DAG. It focuses on one reusable layer:\n\n```text\ngiven a task identity and recorded outputs, should this run execute again or\nreuse a previous result?\n```\n\nThis is useful for workflow tools, batch jobs, AI pipelines, build tools, ETL\nsteps, and delivery systems that need auditable execution records.\n\n## Status\n\nFlowLogbook v0.2.0 is focused on execution records, resume decisions,\nadapter-friendly event history, and event-quality metrics. Within that scope,\nthe current version provides:\n\n- deterministic task fingerprints\n- structured run inputs and output artifacts\n- append-only in-memory ledger storage\n- SQLite ledger storage through a small internal SQLite C FFI adapter\n- PostgreSQL ledger storage through libpq\n- Redis/Valkey-compatible ledger storage through a small built-in RESP client\n- explicit schema version metadata for durable stores\n- latest-record lookup plus full history access\n- node and edge events for executed or externally observed flows\n- event metrics for timing coverage, metric density, node/edge event counts,\n  metric event counts, and total observed duration\n- FlowCaptain-style batch validation through `LogbookInput`\n- resume decisions with explicit reasons\n- reusable policy controls for output and digest requirements\n- focused tests for cache hit, miss, failed run, missing output, and invalid\n  fingerprints\n\n## Requirements\n\nCore in-memory usage only needs Nim.\n\nSQLite persistence uses the system SQLite C library and headers. On Ubuntu:\n\n```bash\nsudo apt install libsqlite3-dev\n```\n\nPostgreSQL persistence uses the system libpq runtime library. On Ubuntu:\n\n```bash\nsudo apt install libpq5\n```\n\nRedis/Valkey-compatible persistence only needs a reachable RESP server.\nFlowLogbook speaks RESP directly from Nim and does not require hiredis.\nThe current RESP adapter targets single-node servers. TLS, Sentinel, Cluster,\nand server-side Lua scripts are outside the built-in adapter scope.\n\n## Example\n\n```nim\nimport flowlogbook\n\nlet input = runInput(\n  taskName = \"thumbnail\",\n  command = \"convert input.png -resize 256x256 output.png\",\n  inputs = @[artifact(\"input.png\", \"sha256:source\")],\n  params = @[kv(\"size\", \"256\")]\n)\n\nvar ledger = initMemoryLedger()\nlet first = ledger.decide(input)\ndoAssert first.kind == rdkExecute\n\nlet completed = completedRecord(input, @[artifact(\"output.png\", \"sha256:out\")])\nledger.record(completed)\n\nlet second = ledger.decide(input)\ndoAssert second.kind == rdkReuse\ndoAssert ledger.attempts(fingerprint(input)) == 1\n```\n\nUse SQLite when records should survive process restarts:\n\n```nim\nimport flowlogbook/sqlite_store\n\nvar persistent = openSqliteLedger(\"flowlogbook.sqlite3\")\npersistent.record(completed)\nlet reused = persistent.decide(input)\npersistent.close()\n```\n\nPostgreSQL and Redis/Valkey-compatible storage are available as optional\nmodules:\n\n```nim\nimport flowlogbook/postgres_store\nimport flowlogbook/redis_store\n\nvar pg = openPostgresLedger(\"host=127.0.0.1 dbname=flowlogbook user=flowlogbook\")\nvar redis = openRedisLedger(host = \"127.0.0.1\", keyPrefix = \"flowlogbook\")\n```\n\nSQLite and PostgreSQL adapters expose explicit transactions for bulk writes:\n\n```nim\npersistent.beginTransaction()\ntry:\n  persistent.record(completed)\n  persistent.commitTransaction()\nexcept:\n  persistent.rollbackTransaction()\n  raise\n```\n\nReuse policy can be adjusted for tasks whose success is represented by a\nside effect rather than a file output:\n\n```nim\nvar policy = defaultReusePolicy()\npolicy.requireOutputs = false\n\nlet decision = ledger.decide(input, policy)\n```\n\nFlowLogbook can also record flow events that come from a runner, a web\nframework adapter, or an external workflow system:\n\n```nim\nledger.recordEvent(nodeEvent(\n  id = \"evt-1\",\n  source = \"nextflow-trace\",\n  flowId = \"pipeline\",\n  runId = \"run-1\",\n  variantId = \"A\",\n  nodeId = \"align\",\n  kind = fekNodeFinished,\n  status = rsCompleted,\n  durationMillis = 1200,\n  metrics = @[kv(\"records\", \"42\")]\n))\n```\n\nEvent metrics can be calculated from collected flow events:\n\n```nim\nlet metrics = events.eventMetrics()\necho metrics.timingCoverage\necho metrics.metricDensity\n```\n\nFor component integration, validate records and events as a batch before\npersisting them:\n\n```nim\nlet input = initLogbookInput(records = @[completed], events = @[\n  nodeEvent(\"evt-2\", \"runner\", \"flow\", \"run-1\", \"publish\", fekNodeFinished,\n    status = rsCompleted)\n])\nlet outcome = validate(input)\ndoAssert outcome.ok\n```\n\nMore examples are available in `examples/`:\n\n- `basic_resume.nim`\n- `sqlite_persistence.nim`\n- `postgres_persistence.nim`\n- `redis_persistence.nim`\n- `external_events.nim`\n\n## Relationship To FlowBrigade\n\nFlowBrigade controls execution while it is happening: retry, timeout, rate\nlimit, bulkhead, lock, and budget.\n\nFlowLogbook records what happened before and after execution: task identity,\noutputs, status, and whether an existing result can be reused.\n\nIt can also record observed node and edge events. This lets external tools such\nas workflow engines, web framework adapters, and business data collectors feed\nevidence into the same logbook model.\n\nThey are designed to compose without either project depending on the other.\n\n## Scope\n\nFlowLogbook deliberately stays below workflow engines. It can be used by a\nworkflow runtime, a web job runner, a build tool, an AI pipeline, or a delivery\nsystem without adopting a specific DSL, scheduler, container backend, or cloud\nservice.\n\nThe SQLite adapter keeps SQLite handles private to FlowLogbook. Public APIs\nuse FlowLogbook records, events, and JSON strings instead of exposing database\nclient handles. PostgreSQL and Redis/Valkey-compatible adapters follow the same public\nAPI shape. This keeps future C ABI bindings simpler and safer.\n\n## Security Notes\n\nFlowLogbook validates records and events before storing them. JSON import rejects\nunknown enum values, negative natural-number fields, and non-array values where\narrays are required.\n\nSQLite writes use prepared statements. PostgreSQL writes use libpq parameterized\nexecution. Redis/Valkey-compatible commands are encoded as RESP frames and do\nnot pass through a shell.\n\nCallers are still responsible for choosing trusted database paths, limiting\nuntrusted JSON input size before parsing, controlling filesystem permissions for\nSQLite files, protecting database credentials, and using TLS or private networks\nwhen connecting to external database servers.\n\nLive adapter tests are opt-in so normal test runs do not require local services:\n\n```bash\nFLOWLOGBOOK_TEST_REDIS=1 nimble test\nFLOWLOGBOOK_TEST_POSTGRES=1 FLOWLOGBOOK_POSTGRES_CONNINFO='host=127.0.0.1 dbname=flowlogbook_test user=flowlogbook' nimble test\n```\n\nFor a disposable PostgreSQL test database:\n\n```bash\ndocker run --rm --name flowlogbook-postgres \\\n  -e POSTGRES_PASSWORD=flowlogbook \\\n  -e POSTGRES_USER=flowlogbook \\\n  -e POSTGRES_DB=flowlogbook_test \\\n  -p 5432:5432 postgres:16\n\nFLOWLOGBOOK_TEST_POSTGRES=1 \\\nFLOWLOGBOOK_POSTGRES_CONNINFO='host=127.0.0.1 port=5432 dbname=flowlogbook_test user=flowlogbook password=flowlogbook' \\\nnimble test\n```\n\nBasic local benchmarks are available:\n\n```bash\nnimble bench\n```\n\nBenchmarks are intended as regression signals, not vendor-neutral database\nbenchmarks. The SQLite benchmark uses an explicit transaction because one\ndurable commit per record measures sync overhead more than ledger overhead.\n\n## Intellectual Property Notes\n\nFlowLogbook intentionally uses general, well-known ideas: deterministic hashes,\ntask records, output metadata, and explicit cache/resume decisions. It is not a\ncopy of Nextflow or any other workflow engine.\n\nSee [docs/ip-notes.md](docs/ip-notes.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpuffball1567%2Fflowlogbook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpuffball1567%2Fflowlogbook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpuffball1567%2Fflowlogbook/lists"}