{"id":22508943,"url":"https://github.com/viant/tapper","last_synced_at":"2025-08-03T13:31:03.551Z","repository":{"id":38301430,"uuid":"294538825","full_name":"viant/tapper","owner":"viant","description":"High performant transaction logger","archived":false,"fork":false,"pushed_at":"2024-11-12T14:30:35.000Z","size":53,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-11-12T15:30:44.136Z","etag":null,"topics":["golang","logger","transaction-processing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/viant.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-10T22:44:56.000Z","updated_at":"2024-11-12T14:30:39.000Z","dependencies_parsed_at":"2024-06-20T19:06:47.706Z","dependency_job_id":"661e32bc-92d7-4fbf-9959-43e6a6fd4651","html_url":"https://github.com/viant/tapper","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Ftapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Ftapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Ftapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viant%2Ftapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/viant","download_url":"https://codeload.github.com/viant/tapper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228547482,"owners_count":17935089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","logger","transaction-processing"],"created_at":"2024-12-07T01:26:17.636Z","updated_at":"2024-12-07T01:26:18.239Z","avatar_url":"https://github.com/viant.png","language":"Go","readme":"# High performant transaction logger for go\n\n[![Application operation performance metric.](https://goreportcard.com/badge/github.com/viant/tapper)](https://goreportcard.com/report/github.com/viant/tapper)\n[![GoDoc](https://godoc.org/github.com/viant/tapper?status.svg)](https://pkg.go.dev/github.com/viant/tapper/?tab=doc)\n\nThis library is compatible with Go 1.12+\n\nPlease refer to [`CHANGELOG.md`](CHANGELOG.md) if you encounter breaking changes.\n\n- [Introduction](#motivation)\n- [Usage](#usage)\n- [Configuration](#configuration)\n- [Messages](#messages)\n- [License](#license)\n- [Credits and Acknowledgements](#credits-and-acknowledgements)\n\n### Introduction\n\nThe goal of this project is to provide hyper performant, zero memory allocation transaction logger,\nthat can work with local and cloud file storage.\nTapper logger allow log rotation, where on each rotation additional logic can be delegated to the external web service or a shell script.\n\n\n### Usage\n\n```go\n\ncfg := \u0026config.Stream{\n    URL: \"/tmp/logfile.log\",\n    Rotation: \u0026config.Rotation{\n        EveryMs: 20000,\n        URL:     \"s3://my.bucket/data/logfile.log.[yyyyMMdd_HH]-%v\",\n    },\n}\nlogger, err := log.New(cfg, \"myID\", afs.New())\nif err != nil {\n    slog.Fatal(err)\n}\nprovider := msg.NewProvider(2048, 32)\nfor i :=0;i\u003c100;i++ {\n    message := provider.NewMessage()\n    message.PutString(\"k1\", \"value1\")\n    message.PutInt(\"k2\", 2)\n    message.PutStrings(\"k3\", []string{\"1\", \"3\"})\n    err = logger.Log(message)\n    if err != nil {\n        slog.Fatal(err)\n    }\n    message.Free()\n}\nlogger.Close()\n\n```\n\n### Configuration\n\n- **URL**:  location of main log stream\n- **FlushMod**: optional flush frequency (testing only, do not use on production)\n- **Codec**: optional compression codec (gzip) of main stream not recommended for production, compressing on rotation is much faster) \n\n- **Rotation**: optional rotation config where:\n    - **EveryMs**: rotation frequency in ms\n    - **Codec**:  optional compression codec (gzip) applied on log rotation.\n    - **URL**: rotation dest pattern\n    - **Emit**: optional rotation event notification vi URL or OS process (shell command) \n        * **URL** URL to call with specified parameters\n        * **Params** URL parameters (query string)\n        * **Name**: name of command to run\n        * **Args**: command arguments\n    - in Args or Params values you can use the following variables:\n        * $DestPath variable to refer to rotated absolute file name  \n        * $Dest to expand with dest URL \n        * $DestName to expand with simple file name \n        * $TimePath yyyy/mm/dd/hh rotation create time base path fragment\n\n##### Rotation URL pattern expression\n    - time expression placed in squere brackets: [yyyy-MM-dd_HH]\n    - logger ID - rotation seqence: %v\n\n##### Configuring rotation event with 3rd party web service\n\nThe following configuration drives rotation notification on http://127.0.0.1:8083 \nvi REST service as GET request.\n```yaml\n\nURL: /opt/app/logs/datastream1.log\nFlushMod: 1\nRotation:\n  EveryMs: 30000\n  URL: /opt/app/logs/datastream1.log.[yyyy-MM-dd_hh-mm-ss].%v\n  Emit:\n    URL: http://127.0.0.1:8083/log/datastream1\n    Params:\n      DestPath: $DestPath\n      DestName: $DestName\n      TimePath: $TimePath\n```\n\nSee [event consumer](emitter/consumer) service example.\n\n\n##### Configuring rotation event with a shell script\n\n```yaml\nURL: /opt/app/logs/datastream1.log\nFlushMod:\nRotation:\n  EveryMs: 10000\n  URL: /opt/app/logs/datastream1.log.[yyyy-MM-dd_hh-mm-ss].%v\n  Emit:\n    Command: /bin/bash\n    Args:\n      - /opt/mediator/script/sitelet.sh\n      - $DestPath\n      - $DestName\n      - $TimePath\n```\n\n### Messages\n\nTo reduce log message memory overhead, a message can be created by [Provider](msg/provider.go), which \nhandles data pooling. You can create a log message with the following snippet using json format.\n```go\nprovider := msg.NewProvider(avgMessageSize, concurrency, json.New) \nmessage := provider.NewMessage()\ndefer message.Free()\n```\n\nOne message is no longer needed Free method returns it back to the provider pool.\n\nLog message [support](io/stream.go) primitive and complex data structure.\n\n```go\nmeesage.Put([]byte{`\"k1\":\"raw data\"`})\nmeesage.PutString(\"k2\", \"v2\")\nmeesage.PutNonEmptyString(\"k2.1\", v)\nmeesage.PutB64EncodedBytes(\"k2.2\", rawData)\nmeesage.PutInt(\"k3\", 3)\nmeesage.PutFloat(\"k3\", 3.2)\nmeesage.PutBool(\"k3\", false)\nmeesage.PutInts(\"k4\", []int{1,2,3})\nmeesage.PutObject(\"k5\", object)\nmeesage.PutObjects(\"k6\", objects)\n```\n\nMessage Provider also support CSV type . In this case, the message provider constructor takes CSV message type. \n\n```go\nprovider := msg.NewProvider(avgMessageSize, concurrency, csv.New)\nmessage := provider.NewMessage()\ndefer message.Free()\n```\nOnly primitive and slice data types are supported in CSV message. \n\n### Benchmark\n\nBenchmark builds b.T x 1K message with 10 attrs and writes the log stream.\n\n```bash\nBenchmarkLogger_Log\nBenchmarkLogger_Log-16             \t  258068\t      4348 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkLogger_Log_Rotation\nBenchmarkLogger_Log_Rotation-16    \t  211363\t      5318 ns/op\t       5 B/op\t       0 allocs/op\n```\n\n\n## License\n\nThe source code is made available under the terms of the Apache License, Version 2, as stated in the file `LICENSE`.\n\nIndividual files may be made available under their own specific license,\nall compatible with Apache License, Version 2. Please see individual files for details.\n\n\n##  Credits and Acknowledgements\n\n**Library Author:** Adrian Witas\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviant%2Ftapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fviant%2Ftapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviant%2Ftapper/lists"}