{"id":15698671,"url":"https://github.com/xunleii/vector_jsonnet","last_synced_at":"2025-10-14T15:31:27.707Z","repository":{"id":55939463,"uuid":"258010420","full_name":"xunleii/vector_jsonnet","owner":"xunleii","description":"Jsonnet library to generate Vector configuration","archived":true,"fork":false,"pushed_at":"2022-11-24T08:47:27.000Z","size":96,"stargazers_count":6,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-10-24T11:49:38.060Z","etag":null,"topics":["jsonnet","library","vector"],"latest_commit_sha":null,"homepage":null,"language":"Jsonnet","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xunleii.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-22T20:17:57.000Z","updated_at":"2023-07-07T13:55:22.000Z","dependencies_parsed_at":"2023-01-23T13:46:10.780Z","dependency_job_id":null,"html_url":"https://github.com/xunleii/vector_jsonnet","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xunleii%2Fvector_jsonnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xunleii%2Fvector_jsonnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xunleii%2Fvector_jsonnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xunleii%2Fvector_jsonnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xunleii","download_url":"https://codeload.github.com/xunleii/vector_jsonnet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":236488463,"owners_count":19156933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jsonnet","library","vector"],"created_at":"2024-10-03T19:31:57.082Z","updated_at":"2025-10-14T15:31:22.414Z","avatar_url":"https://github.com/xunleii.png","language":"Jsonnet","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [vector.dev](vector.dev) jsonnet library\n\n[![Jsonnet](https://img.shields.io/badge/jsonnet-v0.15.0+-blue.svg)](https://github.com/google/jsonnet/releases)\n[![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/xunleii/vector_jsonnet?include_prereleases)](https://github.com/xunleii/vector_jsonnet/releases)\n[![GitHub Issues](https://img.shields.io/github/issues/xunleii/vector_jsonnet.svg)](https://github.com/xunleii/vector_jsonnet/issues)\n[![Contributions welcome](https://img.shields.io/badge/contributions-welcome-orange.svg)](#contributing)\n[![License](https://img.shields.io/badge/license-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n***vector_jsonnet*** *provides an alternative way of writing your TOML Vector configurations.*\n*Instead, you write list of component and link them through simple pipelines. This enables you to write reusable Vector components\nand pipelines that you can reuse or share. It can also be integrated with tools likes [tanka](tanka.dev) in order to deploy vector\ninstances on Kubernetes, with their TOML configuration, without writing a single TOML file.*\n\n## Getting started\n\n### Prerequisite\n\n**vector_jsonnet** requires Jsonnet.\n\n#### Mac OS X\n\nIf you do not have Homebrew installed, [install it now](https://brew.sh/).\nThen run:  `brew install jsonnet`\n\n#### Linux\n\nYou must build the binary. For details, [see the GitHub\nrepository](https://github.com/google/jsonnet).\n\n### Install\n\nThere is two way to install this library :\n\n#### From git\n\nFork or clone this repository, using a command such as:\n`git clone github.com/xunleii/vector_jsonnet`\n\n#### With [jsonnet-bundler](github.com/jsonnet-bundler/jsonnet-bundler) (recommended)\n\n`jb install https://github.com/xunleii/vector_jsonnet`\n\n## How to use\n\n### Import\n\nAfter installing this library, add the appropriate import statements for the library to your Jsonnet code:\n\n```jsonnet\n// assuming you have installed this lib with jb\nlocal vector = (import \"vendor/vector_jsonnet/vector.libsonnet\").vector;\n\n{ /* call vector.METHOD */ }\n```\n\n\u003e You can also import it by merging it with the root Jsonnet object\n\u003e\n\u003e ```jsonnet\n\u003e (import \"vendor/vector_jsonnet/vector.libsonnet\") +\n\u003e {\n\u003e   /* call $.vector.METHOD */\n\u003e }\n\u003e ```\n\n### Global settings *(vector.global)*\n\n\u003e Global options are relevant to Vector as a whole and apply to global behavior.\n\nTheses options are configurable in this section. However, the `.log_schema` cannot be managed by this library because it\nseems to be \"invalid\" by many TOML reader. Other settings like [TOML input verbosity](#toml-configtoml) can also be managed here.\n\n**Example:**\n\n```jsonnet\n// assuming you have installed this lib with jb\nlocal vector = (import \"vendor/vector_jsonnet/vector.libsonnet\").vector;\n\nvector.global({\n  data_dir: '/var/lib/vector'\n})\n```\n\nThanks to jsonnet, it also possible to override or disable fields\n\n```jsonnet\n// assuming you have installed this lib with jb\nlocal vector = (import \"vendor/vector_jsonnet/vector.libsonnet\").vector;\n\nvector.global({\n  data_dir: '/var/lib/vector'\n})\n.global({\n  data_dir:: '/var/lib/vector' // this field is now disabled\n})\n.global({\n  data_dir::: '/var/tmp/vector' // this field is now enabled and its value has changed\n})\n```\n\n### Components *(vector.components)*\n\n\u003e \"Component\" is the generic term we use for [sources](https://vector.dev/docs/about/concepts/#sources),\n[transforms](https://vector.dev/docs/about/concepts/#transforms), and [sinks](https://vector.dev/docs/about/concepts/#sinks).\nYou compose components to create pipelines, allowing you to ingest, transform, and send data.\n\nComponents are the main objects of Vector ; sharing them and use them must be as simple as possible. For that, `vector.components`\n is available and take only one parameter, an object containing the component definitions.\n\nA component definition is a pair of key/value: the component name and the Vector definition of the component (the format is almost\nthe same as the original Vector, but in JSON(NET) intead of TOML).\n\nHowever, there is two differences:\n\n- `inputs` field **MUST NOT** be defined because it is managed by the next part, `vector.pipelines`\n- `type` field **MUST NOT** be defined manually; this is the role of the helpers. Helpers are functions that add required *hidden*\nmetadata to the component, and are available at `vector.KIND.COMPONENT(definition)` (where `KIND` is one of `sources`, `transforms`\nand `sinks`, and `COMPONENT` refers to the Vector component name).\n\n**Example:**\n\n```jsonnet\nlocal vector = (import \"vendor/vector_jsonnet/vector.libsonnet\").vector;\n\nvector.global({ /* see global settings example */ })\n.components({\n  // Ingest data by t\n## License\n\n  // Tailing one or more files\n  apache_logs: vector.sources.file({\n    include: ['/var/log/apache2/*.log'],  // supports globbing\n    ignore_older: 86400,  // 1 day\n  }),\n\n  // Structure and parse the data\n  apache_parser: vector.transforms.regex_parser({\n    regex: '^(?P\u003chost\u003e[w.]+) - (?P\u003cuser\u003e[w]+) (?P\u003cbytes_in\u003e[d]+) [(?P\u003ctimestamp\u003e.*)] \"(?P\u003cmethod\u003e[w]+) (?P\u003cpath\u003e.*)\" (?P\u003cstatus\u003e[d]+) (?P\u003cbytes_out\u003e[d]+)$',\n  }),\n\n  // Sample the data to save on cost\n  apache_sampler: vector.transforms.sampler({\n    rate: 50,  // only keep 50%\n  }),\n\n  // Send structured data to a short-term storage\n  es_cluster: vector.sinks.elasticsearch({\n    host: 'http://79.12.221.222:9200',  // local or external host\n    index: 'vector-%Y-%m-%d',  // daily indices\n  }),\n\n  // Send structured data to a cost-effective long-term storage\n  s3_archives: vector.sinks.aws_s3({\n    region: 'us-east-1',\n    bucket: 'my-log-archives',\n    key_prefix: 'date=%Y-%m-%d',  // daily partitions, hive friendly format\n    compression: 'gzip',  // compress final objects\n    encoding: 'ndjson',  // new line delimited JSON\n    batch: {\n      max_size: 10000000,  // 10mb uncompressed\n    },\n  }),\n})\n```\n\n\u003e **NOTE:** The helpers are generated using the\n[`vector.spec.toml`](https://github.com/timberio/vector/blob/master/config/vector.spec.toml) file. If a component is not available\nin this file or if **vector_jsonnet** is not up to date, you can \"generate\" a custom component by using\n`vector.KIND.fn(COMPONENT, definition)` (where `KIND` is one of `sources`, `transforms` and `sinks`, and `COMPONENT` refers to\nthe Vector component name).\n\u003e\n\u003e *Example:* `vector.sources.fn('kubernetes', {})`\n\n### Pipelines *(vector.pipelines)*\n\n\u003e A \"pipeline\" is the end result of connecting [sources](https://vector.dev/docs/about/concepts/#sources),\n[transforms](https://vector.dev/docs/about/concepts/#transforms), and [sinks](https://vector.dev/docs/about/concepts/#sinks).\nYou can see a full example of a pipeline in the [configuration section](https://vector.dev/docs/setup/configuration/).\n\nA pipeline is, in this library, just a list of the components name defines in the [component section](#components-vectorcomponents).\nThis describe the *flow* of your events; it generally start with a `source` and end with a `sink`.\nAs a picture often speaks better than words, here an example how to define pipelines (using component defined above).\n\n**Example:**\n\n```jsonnet\nlocal vector = (import \"vendor/vector_jsonnet/vector.libsonnet\").vector;\n\nvector.global({ /* see global settings example */ })\n.components({ /* see components example */ })\n.pipelines([\n  ['apache_logs', 'apache_parser', 'apache_sampler', 'es_cluster'],\n  ['apache_logs', 'apache_parser', 's3_archives'],\n])\n```\n\n\u003e **WARN:** be careful when you define several pipelines with the same component; if you change something in the order of the\n\u003e component, all pipelines are also changed\n\u003e\n\u003e ```jsonnet\n\u003e .pipelines([\n\u003e   ['apache_logs', 'apache_parser', 'apache_sampler', 'es_cluster'],\n\u003e ])\n\u003e can be visualised like\n\u003e 'apache_logs' -\u003e 'apache_parser' -\u003e 'apache_sampler' -\u003e 'es_cluster'\n\u003e ```\n\u003e\n\u003e But\n\u003e\n\u003e ```jsonnet\n\u003e .pipelines([\n\u003e   ['apache_logs', 'apache_parser', 'apache_sampler', 'es_cluster'],\n\u003e   ['apache_logs', 'apache_sampler', 'apache_parser']\n\u003e ])\n\u003e can be visualised like\n\u003e         /--------------------------------------\n\u003e        v                                       \\\n\u003e 'apache_logs' -\u003e 'apache_parser' -\u003e 'apache_sampler' -\u003e 'es_cluster'\n\u003e         \\--------------------^\n\u003e ```\n\u003e\n\u003e which creates a loop in the pipeline and this is not handled by this library\n\n### Outputs\n\nWhen your configuration is ready, you can get the configuration in two format\n\n#### JSON *(CONFIG.json)*\n\nThe JSON output is mainly use for testing and debugging purpose, and must not be used to generates configuration for vector.\n\n#### TOML *(CONFIG.toml)*\n\nThe TOML output returns a string representing the full TOML configuration file.\nThe \"verbosity\" can be configured inside the global settings, through the *hidden* object `toml` :\n\n- `enable_intro: true` adds the [introduction part](https://github.com/xunleii/vector_jsonnet/blob/master/vector.out.toml.libsonnet#L79)\nat the head of the file. This `intro` can be override by adding `{assets+:{intro: 'NEW INTRO'}}` to your vector object.\n- `enable_headers: true` adds some [comments](https://github.com/xunleii/vector_jsonnet/blob/master/vector.out.toml.libsonnet#L97) to\nseparates sections into the final TOML. As the first parameter, theses comments can be override by adding\n`{assets+:{global: 'GLOBAL SEPARATOR', sources: 'SOURCES SEPARATOR', transforms: 'TRANSFORMS SEPARATOR', sinks: 'SINKS SEPARATOR'}}`\nto your vector object.\n- `enable_description` adds comments before each components with their `description` hidden fields.\n- `enable_multilines` enables to have multi-lines string instead of a single string with `\\n` on the final TOML. This features is mainly\nused to increase TOML readability for the [LUA component](https://vector.dev/docs/reference/transforms/lua/). But, because the multi-lines\nfeature implementation is heavy, it is disabled by default.\n\nExamples about theses settings are available in `examples/toml` ;\n\n- `jsonnet -S examples/toml/full.jsonnet` to see TOML with all settings enabled\n- `jsonnet -S examples/toml/mini.jsonnet` to see same vector configuration, with all settings disabled\n\n\u003e **WARN**: merging two vector objects erase TOML configuration (this avoid you to import unexpected TOML settings from module)\n\u003e\n\u003e ```jsonnet\n\u003e local vector = (import \"vendor/vector_jsonnet/vector.libsonnet\").vector;\n\u003e\n\u003e (\n\u003e   vector.global({\n\u003e     toml+:: {\n\u003e       enable_intro: true // TOML will be displayed with intro\n\u003e     },\n\u003e     data_dir: '/var/lib/vector'\n\u003e   }) +\n\u003e   vector.global({\n\u003e     data_dir: '/var/lib/vector'\n\u003e   })\n\u003e )\n\u003e .toml // TOML will be displayed without intro\n\u003e ```\n\u003e\n\n## Modules\n\nSee [examples/modular](examples/modular/README.md)\n\n## Contributing\n\nThanks for taking the time to start contributing!!\nJust before, please take a look of [CONTRIBUTING.md](/CONTRIBUTING.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxunleii%2Fvector_jsonnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxunleii%2Fvector_jsonnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxunleii%2Fvector_jsonnet/lists"}