{"id":20578308,"url":"https://github.com/mikestefanello/backlite","last_synced_at":"2025-04-09T18:16:44.589Z","repository":{"id":249823397,"uuid":"818795475","full_name":"mikestefanello/backlite","owner":"mikestefanello","description":"Type-safe, persistent, embedded task queues and background job runner w/ SQLite","archived":false,"fork":false,"pushed_at":"2025-03-28T20:39:28.000Z","size":104,"stargazers_count":83,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-09T18:16:39.275Z","etag":null,"topics":["go","golang","job-queue","job-scheduler","jobs","queue","queues","scheduled-tasks","sqlite","sqlite3","task","task-manager","task-queue","task-runner","tasks"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mikestefanello.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-22T22:25:25.000Z","updated_at":"2025-03-28T20:36:30.000Z","dependencies_parsed_at":"2025-03-13T02:31:33.713Z","dependency_job_id":null,"html_url":"https://github.com/mikestefanello/backlite","commit_stats":null,"previous_names":["mikestefanello/backlite"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbacklite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbacklite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbacklite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikestefanello%2Fbacklite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mikestefanello","download_url":"https://codeload.github.com/mikestefanello/backlite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085323,"owners_count":21045139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","job-queue","job-scheduler","jobs","queue","queues","scheduled-tasks","sqlite","sqlite3","task","task-manager","task-queue","task-runner","tasks"],"created_at":"2024-11-16T06:12:22.171Z","updated_at":"2025-04-09T18:16:44.570Z","avatar_url":"https://github.com/mikestefanello.png","language":"Go","funding_links":[],"categories":["Messaging","Data Structures and Algorithms","消息"],"sub_categories":["Search and Analytic Databases","Queues","检索及分析资料库"],"readme":"## Backlite: Type-safe, persistent, embedded task queues and background job runner w/ SQLite\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/mikestefanello/backlite)](https://goreportcard.com/report/github.com/mikestefanello/backlite)\n[![Test](https://github.com/mikestefanello/backlite/actions/workflows/test.yml/badge.svg)](https://github.com/mikestefanello/backlite/actions/workflows/test.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Go Reference](https://pkg.go.dev/badge/github.com/mikestefanello/backlite.svg)](https://pkg.go.dev/github.com/mikestefanello/backlite)\n[![GoT](https://img.shields.io/badge/Made%20with-Go-1f425f.svg)](https://go.dev)\n[![Test coverage](https://raw.githubusercontent.com/wiki/mikestefanello/backlite/coverage.svg)](https://raw.githack.com/wiki/mikestefanello/backlite/coverage.html)\n\n\u003cp align=\"center\"\u003e\u003cimg alt=\"Logo\" src=\"https://raw.githubusercontent.com/mikestefanello/readmeimages/main/backlite/logo.png\" /\u003e\u003c/p\u003e\n\n## Table of Contents\n* [Introduction](#introduction)\n    * [Overview](#overview)\n    * [Origin](#origin)\n    * [Screenshots](#screenshots)\n    * [Status](#status)\n* [Installation](#installation)\n* [Features](#features)\n    * [Type-safety](#type-safety) \n    * [Persistence with SQLite](#persistence-with-sqlite)\n    * [Optional retention](#optional-retention)\n    * [Retry \u0026 Backoff](#retry--backoff)\n    * [Scheduled execution](#scheduled-execution)\n    * [Logging](#logging)\n    * [Nested tasks](#nested-tasks)\n    * [Graceful shutdown](#graceful-shutdown)\n    * [Transactions](#transactions)\n    * [No database polling](#no-database-polling)\n    * [Driver flexibility](#driver-flexibility)\n    * [Bulk inserts](#bulk-inserts)\n    * [Execution timeout](#execution-timeout)\n    * [Background worker pool](#background-worker-pool)\n    * [Web UI](#web-ui)\n    * [Panic recovery](#panic-recovery)\n* [Usage](#usage)\n    * [Client initialization](#client-initialization)\n    * [Schema installation](#schema-installation)\n    * [Declaring a Task type](#declaring-a-task-type)\n    * [Queue processor](#queue-processor)\n    * [Registering a queue](#registering-a-queue)\n    * [Adding tasks](#adding-tasks)\n    * [Starting the dispatcher](#starting-the-dispatcher)\n    * [Shutting down the dispatcher](#shutting-down-the-dispatcher)\n    * [Example](#example)\n* [Roadmap](#roadmap)\n\n## Introduction\n\n### Overview\n\nBacklite provides type-safe, persistent and embedded task queues meant to run within your application as opposed to an external message broker. A task can be of any type and each type declares its own queue along with the configuration for the queue. Tasks are automatically executed in the background via a configurable worker pool.\n\n### Origin\n\nThis project started shortly after migrating [Pagoda](https://github.com/mikestefanello/pagoda) to SQLite from Postgres and Redis. Redis was previously used to handle task queues and I wanted to leverage SQLite instead. Originally, [goqite](https://github.com/maragudk/goqite) was chosen, which is a great library and one that I took inspiration from, but I had a lot of different ideas for the overall approach and it lacked a number of features that I felt were needed.\n\n[River](https://github.com/riverqueue/river), an excellent, similar library built for Postgres, was also a major source of inspiration and ideas.\n\n### Screenshots\n\n\u003cimg src=\"https://raw.githubusercontent.com/mikestefanello/readmeimages/main/backlite/failed.png\" alt=\"Failed tasks\"/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/mikestefanello/readmeimages/main/backlite/task.png\" alt=\"Task details\"/\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/mikestefanello/readmeimages/main/backlite/task-failed.png\" alt=\"Task failed\"/\u003e\n\n### Status\n\nThis project is under active development, though all features outlined below are available and complete. No significant API or schema changes are expected at this time, but it is certainly possible.\n\n## Installation\n\nInstall by simply running: `go get github.com/mikestefanello/backlite`\n\n## Features\n\n### Type-safety\n\nNo need to deal with serialization and byte slices. By leveraging generics, tasks and queues are completely type-safe which means that you pass in your task type into a queue and your task processor callbacks will only receive that given type.\n \n### Persistence with SQLite\n\nWhen tasks are added to a queue, they are inserted into a SQLite database table to ensure persistence. \n\n### Optional retention\n\nEach queue can have completed tasks retained in a separate table for archiving, auditing, monitoring, etc. Options exist to retain all completed tasks or only those that failed all attempts. An option also exists to retain the task data for all tasks or only those that failed.\n\n### Retry \u0026 Backoff\n\nEach queue can be configured to retry tasks a certain number of times and to backoff a given amount of time between each attempt.\n\n### Scheduled execution\n\nWhen adding a task to a queue, you can specify a duration or specific time to wait until executing the task.\n\n### Logging\n\nOptionally log queue operations with a logger of your choice, as long as it implements the simple `Logger` interface, which `log/slog` does.\n\n```\n2024/07/21 14:08:13 INFO task processed id=0190d67a-d8da-76d4-8fb8-ded870d69151 queue=example duration=85.101µs attempt=1\n```\n\n### Nested tasks\n\nWhile processing a given task, it's easy to create another task in the same or a different queue, which allows you to nest tasks to create workflows. Use `FromContext()` with the provided context to get your initialized client from within the task processor, and add one or many tasks.\n\n### Graceful shutdown\n\nThe task dispatcher, which handles sending tasks to the worker pool for execution, can be shutdown gracefully by calling `Stop()` on the client. That will wait for all workers to finish for as long as the passed in context is not cancelled. The hard-stop the dispatcher, cancel the context passed in when calling `Start()`. See usage below.\n\n### Transactions\n\nTask creation can be added to a given database transaction. If you are using SQLite as your primary database, this provides a simple, robust way to ensure data integrity. For example, using the eCommerce app example, when inserting a new order into your database, the same transaction to be used to add a task to send an order notification email, and they either both succeed or both fail. Use the chained method `Tx()` to provide your transaction when adding one or multiple tasks.\n\n### No database polling\n\nSince SQLite only supports one writer, no continuous database polling is required. The task dispatcher is able to remain aware of new tasks and keep track of when future tasks are scheduled for, and thus only queries the database when it needs to.\n\n### Driver flexibility\n\nUse any SQLite driver that you'd like. This library only includes [go-sqlite3](https://github.com/mattn/go-sqlite3) since it is used in tests.\n\n### Bulk inserts\n\nInsert one or many tasks across one or many queues in a single operation.\n\n### Execution timeout\n\nEach queue can be configured with an execution timeout for processing a given task. The provided context will cancel after the time elapses. If you want to respect that timeout, your processor code will have to listen for the context cancellation.\n\n### Background worker pool\n\nWhen creating a client, you can specify the amount of goroutines to use to build a worker pool. This pool is created and shutdown via the dispatcher by calling `Start()` and `Stop()` on the client. The worker pool is the only way to process tasks; they cannot be pulled manually.\n\n### Web UI\n\nA simple web UI to monitor running, upcoming, and completed tasks is provided.\n\nTo run, pass your `*sql.DB` to `ui.NewHandler()` and register that to an HTTP handler, for example:\n\n```go\nmux := http.DefaultServeMux\nui.NewHandler(db).Register(mux)\nerr := http.ListenAndServe(\":9000\", mux)\n```\n\nThen visit the given port and/or domain in your browser (ie, `localhost:9000`).\n\nThe web CSS is provided by [tabler](https://github.com/tabler/tabler).\n\n### Panic recovery\n\nIf any of your task processors panics, the application with automatically recover, mark the task as failed, and store the panic message as the error message.\n\n## Usage\n\n### Client initialization\n\nFirst, open a connection to your SQLite database using the driver of your choice:\n\n```go\ndb, err := sql.Open(\"sqlite3\", \"data.db?_journal=WAL\u0026_timeout=5000\")\n```\n\nSecond, initialize a client:\n\n```go\nclient, err := backlite.NewClient(backlite.ClientConfig{\n    DB:              db,\n    Logger:          slog.Default(),\n    ReleaseAfter:    10 * time.Minute,\n    NumWorkers:      10,\n    CleanupInterval: time.Hour,\n})\n```\n\nThe configuration options are:\n\n* **DB**: The database connection.\n* **Logger**: A logger that implements the `Logger` interface. Omit if you do not want to log.\n* **ReleaseAfter**: The duration after which tasks claimed and passed for execution should be added back to the queue if a response was never received.\n* **NumWorkers**: The amount of goroutines to open which will process queued tasks.\n* **CleanupInterval**: How often the completed tasks database table will attempt to remove expired rows.\n\n### Schema installation\n\nUntil a more robust system is provided, to install the database schema, call `client.Install()`. This must be done prior to using the client. It is safe to call this if the schema was previously installed. The schema is currently defined in `internal/query/schema.sql`.\n\n### Declaring a Task type\n\nAny type can be a task as long as it implements the `Task` interface, which requires only the `Config() QueueConfig` method, used to provide information about the queue that these tasks will be added to. All fields should be exported. As an example, this is a task used to send new order email notifications:\n\n```go\ntype NewOrderEmailTask struct {\n    OrderID string\n    EmailAddress string\n}\n```\n\nThen implement the `Task` interface by providing the queue configuration:\n\n```go\nfunc (t NewOrderEmailTask) Config() backlite.QueueConfig {\n    return backlite.QueueConfig{\n        Name:        \"NewOrderEmail\",\n        MaxAttempts: 5,\n        Backoff:     5 * time.Second,\n        Timeout:     10 * time.Second,\n        Retention: \u0026backlite.Retention{\n            Duration:   6 * time.Hour,\n            OnlyFailed: false,\n            Data: \u0026backlite.RetainData{\n                OnlyFailed: true,\n            },\n        },\n    }\n}\n```\n\nThe configuration options are:\n\n* **Name**: The name of the queue. This must be unique otherwise registering the queue will fail.\n* **MaxAttempts**: The maximum number of times to try executing this task before it's consider failed and marked as complete.\n* **Backoff**: The amount of time to wait before retrying after a failed attempt at processing.\n* **Retention**: If provided, completed tasks will be retained in the database in a separate table according to the included options.\n    * **Duration**: How long to retain completed tasks in the database for. Omit to never expire.\n    * **OnlyFailed**: If true, only failed tasks will be retained.\n    * **Data**: If provided, the task data (the serialized task itself) will be retained.\n        * **OnlyFailed**: If true, the task data will only be retained for failed tasks.\n\n### Queue processor\n\nThe easiest way to implement a queue and define the processor is to use `backlite.NewQueue()`. This leverages generics in order to provide type-safety with a given task type. Using the example above:\n\n```go\nprocessor := func(ctx context.Context, task NewOrderEmailTask) error {\n    return email.Send(ctx, task.EmailAddress, fmt.Sprintf(\"Order %s received\", task.OrderID))\n}\n\nqueue := backlite.NewQueue[NewOrderEmailTask](processor)\n```\n\nThe parameter is the processor callback which is what will be called by the dispatcher worker pool to execute the task. If no error is returned, the task is considered successfully executed. If the task fails all attempts and the queue has retention enabled, the value of the error will be stored in the database.\n\nThe provided context will be set to timeout at the duration set in the queue settings, if provided. To get the client from the context, you can call `client := backlite.FromContext(ctx)`.\n\n### Registering a queue\n\nYou must register all queues with the client by calling `client.Register(queue)`. This will panic if duplicate queue names are registered.\n\n### Adding tasks\n\nTo add a task to the queue, simply pass one or many into `client.Add()`. You can provide tasks of different types. This returns a chainable operation which contains many options, that can be used as follows:\n\n```go\nerr := client.\n    Add(task1, task2).\n    Ctx(ctx).\n    Tx(tx).\n    At(time.Date(2024, 1, 5, 12, 30, 00)).\n    Wait(15 * time.Minute).\n    Save()\n```\n\nOnly `Add()` and `Save()` are required. Don't use `At()` and `Wait()` together as they override each other.\n\nThe options are:\n\n* **Ctx**: Provide a context to use for the operation.\n* **Tx**: Provide a database transaction to add the tasks to. You must commit this yourself then call `client.Notify()` to tell the dispatcher that the new task(s) were added. This may be improved in the future but for now it is required.\n* **At**: Don't execute this task until at least the given date and time.\n* **Wait**: Wait at least the given duration before executing the task.\n\n### Starting the dispatcher\n\nTo start the dispatcher, which will spin up the worker pool and begin executing tasks in the background, call `client.Start()`. The context you pass in must persist for as long as you want the dispatcher to continue working. If that is ever cancelled, the dispatcher will shutdown. See the next section for more details.\n\n### Shutting down the dispatcher\n\nTo gracefully shutdown the dispatcher, which will wait until all tasks currently being executed are finished, call `client.Stop()`. You can provide a context with a given timeout in order to give the shutdown process a set amount of time to gracefully shutdown. For example:\n\n```go\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\ndefer cancel()\nclient.Stop(ctx)\n```\n\nThis will wait up to 5 seconds for all workers to complete the task they are currently working on.\n\nIf you want to hard-stop the dispatcher, cancel the context that was provided when calling `client.Start()`.\n\n### Example\n\nTo see a working example, check out the example provided in [Pagoda](https://github.com/mikestefanello/pagoda/?tab=readme-ov-file#queues). When the app starts, a queue is defined and the dispatcher is started. There is a web route that includes a form which creates a task in the queue when it is submitted.\n\n## Roadmap\n\n- Expand testing\n- Hooks\n- Expand processor context to store attempt number, other data\n- Avoid needing to call Notify() when using transaction\n- Queue priority\n- Better handling of database schema, migrations\n- Store queue stats in a separate table?\n- Pause/resume queues\n- Benchmarks","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikestefanello%2Fbacklite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikestefanello%2Fbacklite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikestefanello%2Fbacklite/lists"}