{"id":21296903,"url":"https://github.com/reddec/wal","last_synced_at":"2025-03-15T17:27:28.810Z","repository":{"id":81702689,"uuid":"144737794","full_name":"reddec/wal","owner":"reddec","description":"Persistent queue on hashmap","archived":false,"fork":false,"pushed_at":"2018-11-09T16:05:00.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-09T00:06:41.220Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reddec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-14T15:18:56.000Z","updated_at":"2018-11-09T16:05:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"b2c0843c-ddc7-43ee-aa1f-72bbde161ba4","html_url":"https://github.com/reddec/wal","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reddec%2Fwal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reddec%2Fwal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reddec%2Fwal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reddec%2Fwal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reddec","download_url":"https://codeload.github.com/reddec/wal/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243764974,"owners_count":20344510,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-21T14:30:44.210Z","updated_at":"2025-03-15T17:27:28.769Z","avatar_url":"https://github.com/reddec.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Persistent queue and streaming\n\n[![](https://godoc.org/github.com/reddec/wal?status.svg)](https://godoc.org/github.com/reddec/wal)\n\n\n[API documentation](https://godoc.org/github.com/reddec/wal)\n\nThe main goal of the project is provide simple and convenient way of making local durable queue in applications.\n\nIdea is that if application has an local queue, the transport of messages can be anyone.\n\n![diag](https://user-images.githubusercontent.com/6597086/44100830-7a648f9e-a018-11e8-93da-7bba5e4bab3d.png)\n\nBuilt-in [storages](https://github.com/reddec/storages): in-memory, leveldb and else...\n\nBuilt-in processor:\n\n* HTTP client - http client for multiple endpoints with different delivery modes (everyone, at least one)\n\nBuilt-in strategy:\n\n* Repeat-with-delay - adds delay before new attempt if error appeared after processor\n* Ignore - ignore any errors\n\n## Basic usage\n\nSee godoc\n\n```go\n\nimport (\n    \"github.com/reddec/wal/mapqueue\"\n    \"github.com/reddec/wal/stream\"\n    \"context\"\n)\n\nfunc start(globalCtx context.Context) error {\n    // prepare storage\n\tstorage, err := mapqueue.NewLevelDbMap(\"./db\")\n\tif err != nil {\n\t\treturn error\n\t}\n\t// release resources on exit\n\tdefer storage.Close()\n\t// if storage is corrupted, error may appear\n\tqueue, err := mapqueue.NewMapQueue(storage)\n\tif err != nil {\n\t\treturn error\n\t}\n\n\t// create new stream, handler and then start\n\tsendStream := stream.New(queue).\n    \t\tContext(globalCtx).\n    \t\tStdLog(\"[stream] \").\n    \t\tProcess(func(ctx context.Context, data []byte) error {\n    \t\t\t// todo: here usually goes sending code or any other that may produce error\n    \t\t\treturn nil\n    \t\t}).Start()\n\n    // wait while will finish\n    return \u003c-sendStream.Done()\n}\n```\n\n## HTTP delivery\n\nSee cmd\n\nSupports modes:\n\n* Everyone - will fail if even one request failed\n* At-least-one - will pass if even one request was successful\n* At-most-one - will randomize urls and try one-by-one till first successful request, otherwise failed\n\n```go\nimport (\n    \"github.com/reddec/wal/mapqueue\"\n    \"github.com/reddec/wal/stream\"\n    \"github.com/reddec/wal/processor\"\n    \"github.com/reddec/storages/leveldbstorage\"\n    \"context\"\n)\n\nfunc start(globalCtx context.Context) error {\n    // prepare storage\n\tstorage, err := leveldbstorage.New(\"./db\")\n\tif err != nil {\n\t\treturn error\n\t}\n\t// release resources on exit\n\tdefer storage.Close()\n\n\t// if storage is corrupted, error may appear\n\tqueue, err := mapqueue.NewMapQueue(storage)\n\tif err != nil {\n\t\treturn error\n\t}\n\n\t// setup destinations\n\toutput := processor.NewHttpClient(\"http://example.com/\", \"http://serve.org/\").Build()\n\n\t// create new stream, handler and then start\n\tsendStream := stream.New(queue).\n    \t\tContext(globalCtx).\n    \t\tStdLog(\"[stream] \").\n    \t\tHandle(output).Start()\n    defer sendStream.Stop()\n\n    // todo: do some work\n    for i:=0; i\u003c100; i++{\n        err = queue.PutString(\"hello world\")\n        if err!= nil {\n            return err\n        }\n    }\n    // ...\n\n    return nil\n}\n```\n\n# CLI generators\n\n\n## typedqueue\n\nGenerates typed queue.\n\nUsage:\n\n    Usage of typedqueue:\n      -out string\n            Output file (default: \u003ctype name\u003e_storage.go)\n      -package string\n            Output package (default: same as in input file)\n      -type string\n            Type name to wrap\n\n\nEmbedded usage example:\n\n```go\n\ntype Sample struct {\n    // ...\n}\n\n//go:generate typedqueue -type Sample\n\n```\n\nwill produce (methods body omitted)\n\n```go\n\n// Typed queue for Sample\ntype SampleQueue struct {\n\tqueue *mapqueue.Queue\n}\n\n// Creates new queue for Sample\nfunc NewSampleQueue(queue *mapqueue.Queue) *SampleQueue {}\n\n// Put single Sample encoded in JSON into queue\nfunc (cs *SampleQueue) Put(item *Sample) error {}\n\n// Peek single Sample from queue and decode data as JSON\nfunc (cs *SampleQueue) Head() (*Sample, error) {}\n\n// Base (underline) queue\nfunc (cs *SampleQueue) Base() *mapqueue.Queue {}\n\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freddec%2Fwal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freddec%2Fwal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freddec%2Fwal/lists"}