{"id":48983387,"url":"https://github.com/bubustack/map-reduce-adapter-engram","last_synced_at":"2026-04-18T12:00:54.901Z","repository":{"id":352188481,"uuid":"1148069615","full_name":"bubustack/map-reduce-adapter-engram","owner":"bubustack","description":"Map-reduce adapter Engram for bobrapet — dynamic fan-out with child StoryRuns and result aggregation.","archived":false,"fork":false,"pushed_at":"2026-04-18T10:02:11.000Z","size":65,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-18T11:24:47.778Z","etag":null,"topics":["batch","bubustack","engram","fan-out","go","kubernetes","map-reduce","parallel"],"latest_commit_sha":null,"homepage":"https://bubustack.io/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bubustack.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["bubustack"]}},"created_at":"2026-02-02T14:43:38.000Z","updated_at":"2026-04-18T10:02:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/bubustack/map-reduce-adapter-engram","commit_stats":null,"previous_names":["bubustack/map-reduce-adapter-engram"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bubustack/map-reduce-adapter-engram","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bubustack%2Fmap-reduce-adapter-engram","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bubustack%2Fmap-reduce-adapter-engram/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bubustack%2Fmap-reduce-adapter-engram/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bubustack%2Fmap-reduce-adapter-engram/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bubustack","download_url":"https://codeload.github.com/bubustack/map-reduce-adapter-engram/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bubustack%2Fmap-reduce-adapter-engram/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31967993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","bubustack","engram","fan-out","go","kubernetes","map-reduce","parallel"],"created_at":"2026-04-18T12:00:25.304Z","updated_at":"2026-04-18T12:00:54.895Z","avatar_url":"https://github.com/bubustack.png","language":"Go","readme":"# 🧩 Map/Reduce Adapter Engram\n\nScales a Story over thousands of items by spinning up per-item StoryRuns and\noptionally reducing their outputs back into a summary.\n\n## 🌟 Highlights\n\n- Converts each item into a `StoryRun` via template-driven map logic.\n- Stores per-item outputs in shared storage and indexes them for downstream reduce steps.\n- Optional reduce story can summarize items or compute aggregates.\n- Maintains execution safety with retries, conditionals, and storage-based backpressure.\n\n## 🚀 Quick Start\n\n```bash\ngo test ./...\n```\n\nApply `Engram.yaml` and create a Story that references the template. Provide a\nruntime `items` array (inline or storage-backed) plus a `map.storyRef` or inline\n`map.story` definition.\n\n## ⚙️ Template Defaults (`Engram.spec.with`)\n\n| Field | Description |\n| --- | --- |\n| `defaultBatchSize` | Default number of items per batch when `map.batchSize` is omitted. |\n| `defaultConcurrency` | Default maximum number of in-flight StoryRuns when `map.concurrency` is omitted. |\n| `maxBatchSize` | Hard cap applied to batch size overrides. |\n| `maxConcurrency` | Hard cap applied to concurrency overrides. |\n| `defaultFailFast` | Default fail-fast behavior for the map stage. |\n| `defaultAllowFailures` | Default behavior for tolerating map-item failures without failing the adapter step. |\n| `defaultInlineResultsLimit` | Maximum number of per-item results to inline in `output.results` by default (`0` disables inlining). |\n| `pollInterval` | Default polling interval for child StoryRun completion. |\n| `defaultStoryRunTTLSeconds` | Default TTL for child StoryRun resources. |\n| `defaultStoryRunRetentionSeconds` | Default retention window for child StoryRuns. |\n\nThese values set bounds and defaults for runtime requests. The actual fan-out\npayload is supplied in the StepRun input.\n\n## 📥 Inputs\n\n| Field | Description |\n| --- | --- |\n| `items` | Required batch to process. Supports inline arrays, `$bubuStorageRef`, or `$bubuConfigMapRef`. |\n| `context` | Optional shared context injected into every mapped StoryRun. |\n| `inputs` | Optional shared inputs merged into every mapped StoryRun. |\n| `map.storyRef` | Reference to the Story template executed for each item. |\n| `map.story` | Inline Story spec alternative to `map.storyRef`. |\n| `map.inputs` | Additional per-item inputs merged into the map payload. |\n| `map.batchSize` | Per-request override for batch size. |\n| `map.concurrency` | Per-request override for maximum concurrent child StoryRuns. |\n| `map.failFast` | Stop the adapter after the first failed map item. |\n| `map.allowFailures` | Allow failed items without failing the parent step. |\n| `map.inlineResultsLimit` | Inline small per-item results directly in the adapter output. |\n| `map.pollInterval` | Override the polling interval for map child StoryRuns. |\n| `map.storyRunTTLSeconds` | TTL override for map child StoryRuns. |\n| `map.storyRunRetentionSeconds` | Retention override for map child StoryRuns. |\n| `reduce.storyRef` | Optional Story template reference for the reduce stage. |\n| `reduce.story` | Inline Story spec for the reduce stage. |\n| `reduce.inputs` | Additional inputs merged into the reduce payload. |\n| `reduce.pollInterval` | Polling interval override for the reduce StoryRun. |\n| `reduce.storyRunTTLSeconds` | TTL override for the reduce StoryRun. |\n| `reduce.storyRunRetentionSeconds` | Retention override for the reduce StoryRun. |\n\n## 📤 Outputs\n\n| Field | Description |\n| --- | --- |\n| `manifest` | Storage-backed manifest describing each mapped item, its child StoryRun, and the aggregate stats. |\n| `stats` | Object containing `total`, `succeeded`, and `failed` counters. |\n| `resultsPrefix` | Storage prefix that contains per-item outputs. |\n| `resultsIndex` | Storage-backed index of per-item status and output refs. |\n| `results` | Optional inline results when `inlineResultsLimit` allows them. |\n| `reduce` | Optional reduce-stage result metadata (`storyRun`, `phase`, `outputRef`, `error`). |\n\n## 🧪 Local Development\n\n- `go test ./...` – Unit and integration tests for batching and reduce flows.\n- `go vet ./...` – Ensure schema compliance before release.\n\n## 🤝 Community \u0026 Support\n\n- [Contributing](./CONTRIBUTING.md)\n- [Support](./SUPPORT.md)\n- [Security Policy](./SECURITY.md)\n- [Code of Conduct](./CODE_OF_CONDUCT.md)\n- [Discord](https://discord.gg/dysrB7D8H6)\n\n## 📄 License\n\nCopyright 2025 BubuStack.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","funding_links":["https://github.com/sponsors/bubustack"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbubustack%2Fmap-reduce-adapter-engram","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbubustack%2Fmap-reduce-adapter-engram","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbubustack%2Fmap-reduce-adapter-engram/lists"}