{"id":19881112,"url":"https://github.com/f5/otel-arrow-adapter","last_synced_at":"2025-05-02T13:32:33.212Z","repository":{"id":39372862,"uuid":"506592925","full_name":"f5/otel-arrow-adapter","owner":"f5","description":"Adapter used to convert OTEL batches to/from OTEL Arrow batches in both directions.","archived":false,"fork":false,"pushed_at":"2023-11-12T16:13:31.000Z","size":10569,"stargazers_count":40,"open_issues_count":25,"forks_count":6,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-07T02:35:38.747Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/f5.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-23T10:22:04.000Z","updated_at":"2025-01-11T18:48:36.000Z","dependencies_parsed_at":"2024-06-18T21:18:29.952Z","dependency_job_id":"6eb07323-41f8-417e-af85-187d8f038218","html_url":"https://github.com/f5/otel-arrow-adapter","commit_stats":{"total_commits":325,"total_committers":6,"mean_commits":"54.166666666666664","dds":0.3630769230769231,"last_synced_commit":"cafc5b9a40363947430efea1cdc42b32c08ce170"},"previous_names":["lquerel/otel-arrow-adapter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f5%2Fotel-arrow-adapter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f5%2Fotel-arrow-adapter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f5%2Fotel-arrow-adapter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/f5%2Fotel-arrow-adapter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/f5","download_url":"https://codeload.github.com/f5/otel-arrow-adapter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252046281,"owners_count":21685984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T17:13:18.076Z","updated_at":"2025-05-02T13:32:31.311Z","avatar_url":"https://github.com/f5.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"** Important Note: This repository has been donated to and is now part of the OpenTelemetry project, specifically under [otel-arrow](https://github.com/open-telemetry/otel-arrow) and [otel-arrow-collector](https://github.com/open-telemetry/otel-arrow-collector) sub-projects.**\n \n# OTel Arrow Protocol implementation\n\nThis package is a reference implementation of the OTel Arrow protocol specified in this [OTEP](https://github.com/open-telemetry/oteps/blob/main/text/0156-columnar-encoding.md).\nAll OTLP entities are covered (metrics, logs, and traces) as well as all sub-elements such as events, links, gauge, sum, \nsummary, histograms, ... The overall goal is to optimize the compression ratio for telemetry data transmission as well \nas the end-to-end performance between telemetry data producers and receivers.\n\n**This package is still experimental and subject to change.** It is currently used by an [experimental OTLP/Arrow gRPC \nexporter and receiver](https://github.com/open-telemetry/experimental-arrow-collector).\n\nImportant links:\n- [OTEP](https://github.com/open-telemetry/oteps/blob/main/text/0156-columnar-encoding.md) - protocol specification.\n- [Donation](https://github.com/open-telemetry/community/issues/1332) - approved by the Technical Committee (repo not yet transferred in OTel org).\n- [Arrow Data Model](docs/data_model.md) - Mapping OTLP entities to Arrow Schemas.\n- [Benchmark results](docs/benchmarks.md) - Based on synthetic and production data.\n- [Validation process](docs/validation_process.md) - Encoding/Decoding validation process. \n- Articles describing some of the Arrow techniques used behind the scenes to optimize compression ratio and memory usage:\n  - [Data types, encoding, hierarchical data, denormalization](https://arrow.apache.org/blog/2023/04/11/our-journey-at-f5-with-apache-arrow-part-1/)\n  - [Adaptive Schemas and Sorting to Optimize Arrow Usage](https://arrow.apache.org/blog/2023/06/26/our-journey-at-f5-with-apache-arrow-part-2/)\n\n## Benchmark summary\n\nThe following chart shows the compressed message size (in bytes) as a function\nof the batch size for metrics (univariate), logs, and traces. The bottom of the\nchart shows the reduction factor for both the standard OTLP protocol (with ZSTD\ncompression) and the OTel Arrow protocol (ZSTD) in comparison with an\nuncompressed OTLP protocol.\n\n![compression_ratio](./docs/img/compression_ratio_summary_std_metrics.png)\n\nThe next chart follows the same logic but shows the results for multivariate\nmetrics (see left column).\n\n![compression_ratio](./docs/img/compression_ratio_summary_multivariate_metrics.png)\n\nFor more details, see the following [benchmark results](docs/benchmarks.md) page.\n \n## Phase 1 (current implementation)\n\nThis first step is intended to address the specific use cases of traffic reduction. Based on community feedback, many\ncompanies want to reduce the cost of transferring telemetry data over the Internet. By adding a collector that acts as\na point of integration and traffic conversion at the edge of a client environment, we can take advantage of the columnar\nformat to eliminate redundant data and optimize the compression rate. This is illustrated in the following diagram.\n\n![Traffic reduction use case](docs/img/traffic_reduction_use_case.png)\n\n\u003e Note 1: A fallback mechanism can be used to handle the case where the new protocol is not supported by the target. \n\u003e More on this mechanism in this [section](https://github.com/lquerel/oteps/blob/main/text/0156-columnar-encoding.md#protocol-extension-and-fallback-mechanism) of the OTEP. \n\nThe experimental collector implements on top of this library a new Arrow Receiver and Exporter able to fallback on\nstandard OTLP when needed. The following diagram is an overview of this integration. The internal representation of the\ndata has not been updated and this collector is still fundamentally row-oriented internally.\n\n![collector internal overview](docs/img/collector_internal_overview.png)\n\n\u003e Note 2: A future phase 2 of this project will focus on implementing end-to-end OTel Arrow to improve the overall\n\u003e performance.\n\n### Developers\n\nPull requests are welcome. For major changes, please open an issue\nfirst to discuss what you would like to change. For more information, please\nread [CONTRIBUTING](CONTRIBUTING.md).\n\n#### How to change the protobuf specification\n\nTo (re)generate the ArrowStreamService gRPC service, you need to install the `protoc` compiler and the `protoc-gen-grpc` plugin.\n```shell\ngo install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28\ngo install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2\nexport PATH=\"$PATH:$(go env GOPATH)/bin\"\ncd ./proto\n./generate.sh\n```\nOnce the `*.pb.go` files are generated, you need to replace the content of the `api/collector/arrow/v1` directory by the\ngenerated files present in the `./proto/api/collector/arrow/v1` directory.\n\n## Integration with the OpenTelemetry Collector\n\nThe integration of this package with the OpenTelemetry Collector is done in the following experimental repository:\n* [experimental-arrow-collector](https://github.com/open-telemetry/experimental-arrow-collector)\n\nThis above repository houses a fork of the entire core OpenTelemetry\nCollector, where the complete branch history is kept, including\n\"mainline\" Collector commits as well as Arrow-component development\ncommits.\n\nBecause that repository contains portions that are not part of the\nOTel-Arrow project, [the components are being maintained in this\nrepository](https://github.com/open-telemetry/experimental-arrow-collector/issues/48)\nuntil they can be merged into the\n[OpenTemetry-Collector-Contrib](github.com/open-telemetry/opentelemetry-collector-contrib)\nrepository.\n\nCollector components copied from that repository are currently\navailable in the\n[`./collector`](https://github.com/f5/otel-arrow-adapter/blob/main/collector/README.md)\nsub-package of this repository.\n\nExamples demonstrating how to configure and test an OpenTelemetry\nCollector with OTel-Arrow exporter and receiver components are located\nin `./collector/examples`, including:\n\n- [`examples/bridge`](https://github.com/f5/otel-arrow-adapter/tree/main/collector/examples/bridge):\n  A compression bridge between \"edge\" and \"saas\" collectors.\n- [`examples/metadata-bridge`](https://github.com/f5/otel-arrow-adapter/tree/main/collector/examples/metadata-bridge):\n  A compression bridge between \"edge\" and \"saas\" collectors with metadata support, allowing request headers to transit via OTel-Arrow.\n- [`examples/loopback`](https://github.com/f5/otel-arrow-adapter/tree/main/collector/examples/loopback):\n  A collector that writes Arrow to and from itself.\n- [`examples/recorder`](https://github.com/f5/otel-arrow-adapter/tree/main/collector/examples/recorder):\n  A collector with support for recording data files for diagnostic and benchmark purposes.\n- [`examples/synthesize`](https://github.com/f5/otel-arrow-adapter/tree/main/collector/examples/synthesize):\n  A collector with support for synthesizing telemetry data using a [telemetry-generator](https://github.com/lightstep/telemetry-generator) component.\n\n## License\n\nOTel Arrow Protocol Adapter is licensed under Apache 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ff5%2Fotel-arrow-adapter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ff5%2Fotel-arrow-adapter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ff5%2Fotel-arrow-adapter/lists"}