{"id":13436976,"url":"https://github.com/vectordotdev/vector","last_synced_at":"2025-05-12T05:32:48.654Z","repository":{"id":36977283,"uuid":"146327667","full_name":"vectordotdev/vector","owner":"vectordotdev","description":"A high-performance observability data pipeline.","archived":false,"fork":false,"pushed_at":"2025-05-09T20:18:41.000Z","size":121244,"stargazers_count":19486,"open_issues_count":2037,"forks_count":1738,"subscribers_count":155,"default_branch":"master","last_synced_at":"2025-05-12T02:45:51.157Z","etag":null,"topics":["events","forwarder","logs","metrics","observability","parser","pipeline","router","rust","stream-processing","vector"],"latest_commit_sha":null,"homepage":"https://vector.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vectordotdev.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":".github/SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-08-27T16:57:34.000Z","updated_at":"2025-05-12T02:01:17.000Z","dependencies_parsed_at":"2023-12-25T05:27:48.892Z","dependency_job_id":"a4836252-8fde-45cd-b375-97e92dac0aa8","html_url":"https://github.com/vectordotdev/vector","commit_stats":{"total_commits":10564,"total_committers":395,"mean_commits":"26.744303797468355","dds":0.8043354789852328,"last_synced_commit":"a75a043523cb3839c4c186719565aeeeceba01cf"},"previous_names":["timberio/vector"],"tags_count":106,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vectordotdev%2Fvector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vectordotdev","download_url":"https://codeload.github.com/vectordotdev/vector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253672734,"owners_count":21945482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["events","forwarder","logs","metrics","observability","parser","pipeline","router","rust","stream-processing","vector"],"created_at":"2024-07-31T03:00:53.448Z","updated_at":"2025-05-12T05:32:44.214Z","avatar_url":"https://github.com/vectordotdev.png","language":"Rust","readme":"\u003cp align=\"center\"\u003e\n  \u003cstrong\u003e\n    \u003ca href=\"https://vector.dev/docs/setup/quickstart/\"\u003eQuickstart\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026bull;\u0026nbsp;\u0026nbsp;\n    \u003ca href=\"https://vector.dev/docs/\"\u003eDocs\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026bull;\u0026nbsp;\u0026nbsp;\n    \u003ca href=\"https://vector.dev/guides/\"\u003eGuides\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026bull;\u0026nbsp;\u0026nbsp;\n    \u003ca href=\"https://vector.dev/components/\"\u003eIntegrations\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026bull;\u0026nbsp;\u0026nbsp;\n    \u003ca href=\"https://chat.vector.dev\"\u003eChat\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026bull;\u0026nbsp;\u0026nbsp;\n    \u003ca href=\"https://vector.dev/releases/latest/download/\"\u003eDownload\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026bull;\u0026nbsp;\u0026nbsp;\n    \u003ca href=\"https://rust-doc.vector.dev/\"\u003eRust Crate Docs\u003c/a\u003e\n  \u003c/strong\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"website/static/img/diagram.svg\" alt=\"Vector\"\u003e\n\u003c/p\u003e\n\n## What is Vector?\n\nVector is a high-performance, end-to-end (agent \u0026 aggregator) observability data\npipeline that puts you in control of your observability data.\n[Collect][docs.sources], [transform][docs.transforms], and [route][docs.sinks]\nall your logs and metrics to any vendors you want today and any other\nvendors you may want tomorrow. Vector enables dramatic cost reduction, novel\ndata enrichment, and data security where you need it, not where it is most\nconvenient for your vendors. Additionally, it is open source and up to 10x\nfaster than every alternative in the space.\n\nTo get started, follow our [**quickstart guide**][docs.quickstart] or [**install\nVector**][docs.installation].\n\n### Principles\n\n* **Reliable** - Built in [Rust][urls.rust], Vector's primary design goal is reliability.\n* **End-to-end** - Deploys as an [agent][docs.roles#agent] or [aggregator][docs.roles#aggregator]. Vector is a complete platform.\n* **Unified** - [Logs][docs.data-model.log], [metrics][docs.data-model.metric] (beta), and traces (coming soon). One tool for all of your data.\n\n### Use cases\n\n* Reduce total observability costs.\n* Transition vendors without disrupting workflows.\n* Enhance data quality and improve insights.\n* Consolidate agents and eliminate agent fatigue.\n* Improve overall observability performance and reliability.\n\n### Community\n\n* Vector is relied on by startups and enterprises like **Atlassian**, **T-Mobile**,\n  **Comcast**, **Zendesk**, **Discord**, **Fastly**, **CVS**, **Trivago**,\n  **Tuple**, **Douban**, **Visa**, **Mambu**, **Blockfi**, **Claranet**,\n  **Instacart**, **Forcepoint**, and [many more][urls.production_users].\n* Vector is **downloaded over 100,000 times per day**.\n* Vector's largest user **processes over 30TB daily**.\n* Vector has **over 100 contributors** and growing.\n\n## [Documentation](https://vector.dev/docs/)\n\n### About\n\n* [**Concepts**][docs.about.concepts]\n* [**Under the hood**][docs.about.under-the-hood]\n  * [**Architecture**][docs.under-the-hood.architecture] - [data model][docs.architecture.data-model] ([log][docs.data-model.log], [metric][docs.data-model.metric]), [pipeline model][docs.architecture.pipeline-model], [concurrency model][docs.architecture.concurrency-model], [runtime model][docs.architecture.runtime-model]\n  * [**Networking**][docs.under-the-hood.networking] - [ARC][docs.networking.adaptive-request-concurrency]\n  * [**Guarantees**][docs.under-the-hood.guarantees]\n\n### Setup\n\n* [**Quickstart**][docs.setup.quickstart]\n* [**Installation**][docs.setup.installation] - [operating systems][docs.installation.operating_systems], [package managers][docs.installation.package_managers], [platforms][docs.installation.platforms] ([Kubernetes][docs.platforms.kubernetes]), [manual][docs.installation.manual]\n* [**Deployment**][docs.deployment] - [roles][docs.deployment.roles], [topologies][docs.deployment.topologies]\n\n### Reference\n\n* **Configuration**\n  * [**Sources**][docs.configuration.sources] - [docker_logs][docs.sources.docker_logs], [file][docs.sources.file], [http][docs.sources.http], [journald][docs.sources.journald], [kafka][docs.sources.kafka], [socket][docs.sources.socket], and [many more...][docs.sources]\n  * [**Transforms**][docs.configuration.transforms] - [dedupe][docs.transforms.dedupe], [filter][docs.transforms.filter], [log_to_metric][docs.transforms.log_to_metric], [lua][docs.transforms.lua], [remap][docs.transforms.remap],\n    and [many more...][docs.transforms]\n  * [**Sinks**][docs.configuration.sinks] - [aws_cloudwatch_logs][docs.sinks.aws_cloudwatch_logs], [aws_s3][docs.sinks.aws_s3], [clickhouse][docs.sinks.clickhouse], [elasticsearch][docs.sinks.elasticsearch], [gcp_cloud_storage][docs.sinks.gcp_cloud_storage], and [many more...][docs.sinks]\n  * [**Enrichment Tables**][docs.configuration.enrichment_tables]\n  * [**Unit tests**][docs.configuration.tests]\n* [**Remap Language**][docs.reference.vrl]\n* [**API**][docs.reference.api]\n* [**CLI**][docs.reference.cli]\n\n### Administration\n\n* [**Management**][docs.administration.management]\n* [**Monitoring \u0026 observing**][docs.administration.monitoring]\n* [**Upgrading**][docs.administration.upgrading]\n* [**Validating**][docs.administration.validating]\n\n### Resources\n\n* [**Community**][urls.vector_community] - [chat][urls.vector_chat], [calendar][urls.vector_calendar], [@vectordotdev][urls.vector_twitter]\n* [**Releases**][urls.vector_releases]\n* **Policies** - [Code of Conduct][urls.vector_code_of_conduct], [Privacy][urls.vector_privacy_policy], [Releases][urls.vector_releases_policy], [Security][urls.vector_security_policy], [Versioning][urls.vector_versioning_policy]\n\n## Comparisons\n\n### Performance\n\nThe following performance tests demonstrate baseline performance between\ncommon protocols with the exception of the Regex Parsing test.\n\n| Test                                                                                                                   | Vector          | Filebeat | FluentBit       | FluentD   | Logstash  | SplunkUF        | SplunkHF |\n| ---------------------------------------------------------------------------------------------------------------------- | --------------- | -------- | --------------- | --------- | --------- | --------------- | -------- |\n| [TCP to Blackhole](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/tcp_to_blackhole_performance) | _**86mib/s**_   | n/a      | 64.4mib/s       | 27.7mib/s | 40.6mib/s | n/a             | n/a      |\n| [File to TCP](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/file_to_tcp_performance)           | _**76.7mib/s**_ | 7.8mib/s | 35mib/s         | 26.1mib/s | 3.1mib/s  | 40.1mib/s       | 39mib/s  |\n| [Regex Parsing](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/regex_parsing_performance)       | 13.2mib/s       | n/a      | _**20.5mib/s**_ | 2.6mib/s  | 4.6mib/s  | n/a             | 7.8mib/s |\n| [TCP to HTTP](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/tcp_to_http_performance)           | _**26.7mib/s**_ | n/a      | 19.6mib/s       | \u003c1mib/s   | 2.7mib/s  | n/a             | n/a      |\n| [TCP to TCP](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/tcp_to_tcp_performance)             | 69.9mib/s       | 5mib/s   | 67.1mib/s       | 3.9mib/s  | 10mib/s   | _**70.4mib/s**_ | 7.6mib/s |\n\nTo learn more about our performance tests, please see the [Vector test harness][urls.vector_test_harness].\n\n### Correctness\n\nThe following correctness tests are not exhaustive, but they demonstrate\nfundamental differences in quality and attention to detail:\n\n| Test                                                                                                                                 | Vector | Filebeat | FluentBit | FluentD | Logstash | Splunk UF | Splunk HF |\n| ------------------------------------------------------------------------------------------------------------------------------------ | ------ | -------- | --------- | ------- | -------- | --------- | --------- |\n| [Disk Buffer Persistence](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/disk_buffer_persistence_correctness) | **✓**  | ✓        |           |         | ⚠        | ✓         | ✓         |\n| [File Rotate (create)](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/file_rotate_create_correctness)         | **✓**  | ✓        | ✓         | ✓       | ✓        | ✓         | ✓         |\n| [File Rotate (copytruncate)](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/file_rotate_truncate_correctness) | **✓**  |          |           |         |          | ✓         | ✓         |\n| [File Truncation](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/file_truncate_correctness)                   | **✓**  | ✓        | ✓         | ✓       | ✓        | ✓         | ✓         |\n| [Process (SIGHUP)](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/sighup_correctness)                         | **✓**  |          |           |         | ⚠        | ✓         | ✓         |\n| [JSON (wrapped)](https://github.com/vectordotdev/vector-test-harness/tree/master/cases/wrapped_json_correctness)                     | **✓**  | ✓        | ✓         | ✓       | ✓        | ✓         | ✓         |\n\nTo learn more about our correctness tests, please see the [Vector test harness][urls.vector_test_harness].\n\n### Features\n\nVector is an end-to-end, unified, open data platform.\n\n|                     | **Vector** | Beats | Fluentbit | Fluentd | Logstash | Splunk UF | Splunk HF | Telegraf |\n| ------------------- | ---------- | ----- | --------- | ------- | -------- | --------- | --------- | -------- |\n| **End-to-end**      | **✓**      |       |           |         |          |           |           | ✓        |\n| Agent               | **✓**      | ✓     | ✓         |         |          | ✓         |           | ✓        |\n| Aggregator          | **✓**      |       |           | ✓       | ✓        |           | ✓         | ✓        |\n| **Unified**         | **✓**      |       |           |         |          |           |           | ✓        |\n| Logs                | **✓**      | ✓     | ✓         | ✓       | ✓        | ✓         | ✓         | ✓        |\n| Metrics             | **✓**      | ⚠     | ⚠         | ⚠       | ⚠        | ⚠         | ⚠         | ✓        |\n| Traces              | 🚧         |       |           |         |          |           |           |          |\n| **Open**            | **✓**      |       | ✓         | ✓       |          |           |           | ✓        |\n| Open-source         | **✓**      | ✓     | ✓         | ✓       | ✓        |           |           | ✓        |\n| Vendor-neutral      | **✓**      |       | ✓         | ✓       |          |           |           | ✓        |\n| **Reliability**     | **✓**      |       |           |         |          |           |           |          |\n| Memory-safe         | **✓**      |       |           |         |          |           |           | ✓        |\n| Delivery guarantees | **✓**      |       |           |         |          | ✓         | ✓         |          |\n| Multi-core          | **✓**      | ✓     | ✓         | ✓       | ✓        | ✓         | ✓         | ✓        |\n\n\n⚠ = Not interoperable, metrics are represented as structured logs\n\n---\n\n\u003cp align=\"center\"\u003e\n  Developed with ❤️ by \u003cstrong\u003e\u003ca href=\"https://datadoghq.com\"\u003eDatadog\u003c/a\u003e\u003c/strong\u003e - \u003ca href=\"https://github.com/vectordotdev/vector/security/policy\"\u003eSecurity Policy\u003c/a\u003e - \u003ca href=\"https://github.com/vectordotdev/vector/blob/master/PRIVACY.md\"\u003ePrivacy Policy\u003c/a\u003e\n\u003c/p\u003e\n\n[docs.about.concepts]: https://vector.dev/docs/about/concepts/\n[docs.about.under-the-hood]: https://vector.dev/docs/about/under-the-hood/\n[docs.administration.monitoring]: https://vector.dev/docs/administration/monitoring/\n[docs.administration.management]: https://vector.dev/docs/administration/management/\n[docs.administration.upgrading]: https://vector.dev/docs/administration/upgrading/\n[docs.administration.validating]: https://vector.dev/docs/administration/validating/\n[docs.architecture.concurrency-model]: https://vector.dev/docs/about/under-the-hood/architecture/concurrency-model/\n[docs.architecture.data-model]: https://vector.dev/docs/about/under-the-hood/architecture/data-model/\n[docs.architecture.pipeline-model]: https://vector.dev/docs/about/under-the-hood/architecture/pipeline-model/\n[docs.architecture.runtime-model]: https://vector.dev/docs/about/under-the-hood/architecture/runtime-model/\n[docs.configuration.sinks]: https://vector.dev/docs/reference/configuration/sinks/\n[docs.configuration.sources]: https://vector.dev/docs/reference/configuration/sources/\n[docs.configuration.tests]: https://vector.dev/docs/reference/configuration/tests/\n[docs.configuration.transforms]: https://vector.dev/docs/reference/configuration/transforms/\n[docs.configuration.enrichment_tables]: https://vector.dev/docs/reference/configuration/global-options/#enrichment_tables\n[docs.data-model.log]: https://vector.dev/docs/about/under-the-hood/architecture/data-model/log/\n[docs.data-model.metric]: https://vector.dev/docs/about/under-the-hood/architecture/data-model/metric/\n[docs.deployment.roles]: https://vector.dev/docs/setup/deployment/roles/\n[docs.deployment.topologies]: https://vector.dev/docs/setup/deployment/topologies/\n[docs.deployment]: https://vector.dev/docs/setup/deployment/\n[docs.installation.manual]: https://vector.dev/docs/setup/installation/manual/\n[docs.installation.operating_systems]: https://vector.dev/docs/setup/installation/operating-systems/\n[docs.installation.package_managers]: https://vector.dev/docs/setup/installation/package-managers/\n[docs.installation.platforms]: https://vector.dev/docs/setup/installation/platforms/\n[docs.installation]: https://vector.dev/docs/setup/installation/\n[docs.networking.adaptive-request-concurrency]: https://vector.dev/docs/about/under-the-hood/networking/arc/\n[docs.platforms.kubernetes]: https://vector.dev/docs/setup/installation/platforms/kubernetes/\n[docs.quickstart]: https://vector.dev/docs/setup/quickstart/\n[docs.reference.api]: https://vector.dev/docs/reference/api/\n[docs.reference.cli]: https://vector.dev/docs/reference/cli/\n[docs.reference.vrl]: https://vector.dev/docs/reference/vrl/\n[docs.roles#agent]: https://vector.dev/docs/setup/deployment/roles/#agent\n[docs.roles#aggregator]: https://vector.dev/docs/setup/deployment/roles/#aggregator\n[docs.setup.installation]: https://vector.dev/docs/setup/installation/\n[docs.setup.quickstart]: https://vector.dev/docs/setup/quickstart/\n[docs.sinks.aws_cloudwatch_logs]: https://vector.dev/docs/reference/configuration/sinks/aws_cloudwatch_logs/\n[docs.sinks.aws_s3]: https://vector.dev/docs/reference/configuration/sinks/aws_s3/\n[docs.sinks.clickhouse]: https://vector.dev/docs/reference/configuration/sinks/clickhouse/\n[docs.sinks.elasticsearch]: https://vector.dev/docs/reference/configuration/sinks/elasticsearch/\n[docs.sinks.gcp_cloud_storage]: https://vector.dev/docs/reference/configuration/sinks/gcp_cloud_storage/\n[docs.sinks]: https://vector.dev/docs/reference/configuration/sinks/\n[docs.sources.docker_logs]: https://vector.dev/docs/reference/configuration/sources/docker_logs/\n[docs.sources.file]: https://vector.dev/docs/reference/configuration/sources/file/\n[docs.sources.http]: https://vector.dev/docs/reference/configuration/sources/http/\n[docs.sources.journald]: https://vector.dev/docs/reference/configuration/sources/journald/\n[docs.sources.kafka]: https://vector.dev/docs/reference/configuration/sources/kafka/\n[docs.sources.socket]: https://vector.dev/docs/reference/configuration/sources/socket/\n[docs.sources]: https://vector.dev/docs/reference/configuration/sources/\n[docs.transforms.dedupe]: https://vector.dev/docs/reference/configuration/transforms/dedupe/\n[docs.transforms.filter]: https://vector.dev/docs/reference/configuration/transforms/filter/\n[docs.transforms.log_to_metric]: https://vector.dev/docs/reference/configuration/transforms/log_to_metric/\n[docs.transforms.lua]: https://vector.dev/docs/reference/configuration/transforms/lua/\n[docs.transforms.remap]: https://vector.dev/docs/reference/configuration/transforms/remap/\n[docs.transforms]: https://vector.dev/docs/reference/configuration/transforms/\n[docs.under-the-hood.architecture]: https://vector.dev/docs/about/under-the-hood/architecture/\n[docs.under-the-hood.guarantees]: https://vector.dev/docs/about/under-the-hood/guarantees/\n[docs.under-the-hood.networking]: https://vector.dev/docs/about/under-the-hood/networking/\n[urls.production_users]: https://github.com/vectordotdev/vector/issues/790\n[urls.rust]: https://www.rust-lang.org/\n[urls.vector_calendar]: https://calendar.vector.dev\n[urls.vector_chat]: https://chat.vector.dev\n[urls.vector_code_of_conduct]: https://github.com/vectordotdev/vector/blob/master/CODE_OF_CONDUCT.md\n[urls.vector_community]: https://vector.dev/community/\n[urls.vector_privacy_policy]: https://github.com/vectordotdev/vector/blob/master/PRIVACY.md\n[urls.vector_release_policy]: https://github.com/vectordotdev/vector/blob/master/RELEASING.md\n[urls.vector_releases]: https://vector.dev/releases/\n[urls.vector_releases_policy]: https://github.com/vectordotdev/vector/blob/master/RELEASES.md\n[urls.vector_security_policy]: https://github.com/vectordotdev/vector/security/policy\n[urls.vector_test_harness]: https://github.com/vectordotdev/vector-test-harness/\n[urls.vector_twitter]: https://twitter.com/vectordotdev\n[urls.vector_versioning_policy]: https://github.com/vectordotdev/vector/blob/master/VERSIONING.md\n[urls.vote_feature]: https://github.com/vectordotdev/vector/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc+label%3A%22type%3A+feature%22\n\n","funding_links":[],"categories":["应用","Rust","Applications","Observability","💻 Apps","HarmonyOS","rust","后端开发框架及项目","数据管道","Corporate and Analytical Applications","Logging","Tools","应用程序 Applications","Application Recommendation"],"sub_categories":["Observability","Regex","🌐 Networking and Internet","Windows Manager","管理面板","Data Integration and Specialized Solutions","Opentelemetry Vendors","可观察性 Observability","📊 Data Analysis","Observability Pipelines and Telemetry Processing"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvectordotdev%2Fvector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvectordotdev%2Fvector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvectordotdev%2Fvector/lists"}