{"id":19704698,"url":"https://github.com/datastax/pulsar-transformations","last_synced_at":"2025-07-29T20:35:14.847Z","repository":{"id":37789136,"uuid":"503806322","full_name":"datastax/pulsar-transformations","owner":"datastax","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-28T05:19:34.000Z","size":41427,"stargazers_count":10,"open_issues_count":7,"forks_count":8,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-28T06:26:47.749Z","etag":null,"topics":["apache-pulsar","streaming"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datastax.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-06-15T14:43:22.000Z","updated_at":"2024-10-05T08:37:48.000Z","dependencies_parsed_at":"2024-06-17T14:16:25.968Z","dependency_job_id":"ad319a63-d457-439b-bcf7-fd0cf057760e","html_url":"https://github.com/datastax/pulsar-transformations","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"purl":"pkg:github/datastax/pulsar-transformations","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-transformations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-transformations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-transformations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-transformations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datastax","download_url":"https://codeload.github.com/datastax/pulsar-transformations/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-transformations/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267754941,"owners_count":24139438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-pulsar","streaming"],"created_at":"2024-11-11T21:24:07.804Z","updated_at":"2025-07-29T20:35:14.805Z","avatar_url":"https://github.com/datastax.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pulsar transformations\n\nPulsar Transformations is a Pulsar Function that implements commonly done transformations on the data.\nThe intent is to provide a low-code approach, so you don't need to write code, understand Pulsar Schemas, \nor know one of the languages supported by Pulsar Functions to transform the data flowing in your Pulsar cluster.\nThe goal is also that the Pulsar Transformations Function is easy to use in the Pulsar cluster \nwithout having to install and operate other software.\nOnly basic transformations are available.\nFor more complex use cases, such as aggregation, joins and lookups,\nmore sophisticated tools such as SQL stream processing engines shall be used.\n\nCurrently available transformations are:\n* [cast](#cast): modifies the key or value schema to a target compatible schema.\n* [drop-fields](#drop-fields): drops fields from structured data.\n* [merge-key-value](#merge-key-value): merges the fields of KeyValue records where both the key and value are structured data with the same schema type.\n* [unwrap-key-value](#unwrap-key-value): if the record is a KeyValue, extract the KeyValue's key or value and make it the record value.\n* [flatten](#flatten): flattens structured data.\n* [drop](#drop): drops a record from further processing.\n* [compute](#compute): computes new properties, values or field values on the fly or replaces existing ones.\n\nPulsar Transformations requires Pulsar 2.11+ or Luna Streaming 2.10+ to run.\n\n## Configuration\n\nThe `TransformFunction` reads its configuration as `JSON` from the Function `userConfig` parameter in the format:\n\n```json\n{\n  \"steps\": [\n    {\n      \"type\": \"drop-fields\", \"fields\": \"keyField1,keyField2\", \"part\": \"key\"\n    },\n    {\n      \"type\": \"merge-key-value\"\n    },\n    {\n      \"type\": \"unwrap-key-value\"\n    },\n    {\n      \"type\": \"cast\", \"schema-type\": \"STRING\"\n    }\n  ]\n}\n```\n\nThe transformations are done in the order in which they appear in the `steps` array.\nEach step is defined by its `type` and uses its own arguments.\nAdditionally, each step can be dynamically toggled on or off\nby supplying a `when` condition that evaluates to true or false. \n\n\nThis example config applied on a `KeyValue\u003cAVRO, AVRO\u003e` input record with value `{key={keyField1: key1, keyField2: key2, keyField3: key3}, value={valueField1: value1, valueField2: value2, valueField3: value3}}` will return after each step:\n\n```\n{key={keyField1: key1, keyField2: key2, keyField3: key3}, value={valueField1: value1, valueField2: value2, valueField3: value3}}(KeyValue\u003cAVRO, AVRO\u003e)\n           |\n           | ”type\": \"drop-fields\", \"fields\": \"keyField1,keyField2”, \"part\": \"key”\n           |\n{key={keyField3: key3}, value={valueField1: value1, valueField2: value2, valueField3: value3}} (KeyValue\u003cAVRO, AVRO\u003e)\n           |\n           | \"type\": \"merge-key-value\"\n           |\n{key={keyField3: key3}, value={keyField3: key3, valueField1: value1, valueField2: value2, valueField3: value3}} (KeyValue\u003cAVRO, AVRO\u003e)\n           |\n           | \"type\": \"unwrap-key-value\"\n           |\n{keyField3: key3, valueField1: value1, valueField2: value2, valueField3: value3} (AVRO)\n           |\n           | \"type\": \"cast\", \"schema-type\": \"STRING\"\n           |\n{\"keyField3\": \"key3\", \"valueField1\": \"value1\", \"valueField2\": \"value2\", \"valueField3\": \"value3\"} (STRING)\n```\n\n## Type conversions\n\nSome step operations like `cast` or `compute` involve conversions from a type to another.\nWhen this happens the rules are:\n* timestamp, date and time related object conversions assume UTC time zone if it is not explicit.\n* date and time related object conversions to/from STRING use the RFC3339 format.\n* timestamp related object conversions to/from LONG and DOUBLE are done using the number of milliseconds since EPOCH (1970-01-01T00:00:00Z).\n* date related object conversions to/from INTEGER, LONG, FLOAT and DOUBLE are done using the number of days since EPOCH (1970-01-01).\n* time related object conversions to/from INTEGER, LONG and DOUBLE are done using the number of milliseconds since midnight (00:00:00).\n\n## Available steps\n\n### Cast\n\nTransforms the data to a target compatible schema.\nConversion are done using the rules described in [Type conversions](#type-conversions)\n\nStep name: `cast`\n\nParameters:\n\n| Name        | Description                                                                                                                                                                       |\n|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| schema-type | the target schema type.                                                                                                                                                           |\n| part        | when used with KeyValue data, defines if the transformation is done on the `key` or on the `value`. If `null` or absent the transformation applies to both the key and the value. |\n\n#### Example:\n\nUserConfig: `{\"steps\": [{\"type\": \"cast\", \"schema-type\": \"STRING\"}]}`\n\nInput: `{field1: value1, field2: value2} (AVRO)`\n\nOutput: `{\"field1\": \"value1\", \"field2\": \"value2\"} (STRING)`\n\n### Drop fields\n\nDrops fields of structured data (Currently only AVRO and JSON are supported).\n\nStep name: `drop-fields`\n\nParameters:\n\n| Name   | Description                                                                                                                                                                       |\n|--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| fields | the list of fields to drop separated by commas `,`                                                                                                                                |\n| part   | when used with KeyValue data, defines if the transformation is done on the `key` or on the `value`. If `null` or absent the transformation applies to both the key and the value. |\n\n#### Example\n\nUserConfig: `{\"steps\": [{\"type\": \"drop-fields\", \"fields\": \"password,other\"}]}`\n\nInput: `{name: value1, password: value2} (AVRO)`\n\nOutput: `{name: value1} (AVRO)`\n\n### Merge KeyValue\n\nMerges the fields of KeyValue records where both the key and value are structured types of the same schema type. (Currently only AVRO and JSON are supported).\n\nStep name: `merge-key-value`\n\nParameters: N/A\n\n##### Example\n\nUserConfig: `{\"steps\": [{\"type\": \"merge-key-value\"}]}`\n\nInput: `{key={keyField: key}, value={valueField: value}} (KeyValue\u003cAVRO, AVRO\u003e)`\n\nOutput: `{key={keyField: key}, value={keyField: key, valueField: value}} (KeyValue\u003cAVRO, AVRO\u003e)`\n\n### Unwrap KeyValue\n\nIf the record value is a KeyValue, extracts the KeyValue's key or value and make it the record value. \n\nStep name: `unwrap-key-value`\n\nParameters:\n\n| Name      | Description                                                                                 |\n|-----------|---------------------------------------------------------------------------------------------|\n| unwrapKey | by default, the value is unwrapped. Set this parameter to `true` to unwrap the key instead. |\n\n##### Example\n\nUserConfig: `{\"steps\": [{\"type\": \"unwrap-key-value\"}]}`\n\nInput: `{key={keyField: key}, value={valueField: value}} (KeyValue\u003cAVRO, AVRO\u003e)`\n\nOutput: `{valueField: value} (AVRO)`\n\n### Flatten\n\nConverts structured nested data into a new single-hierarchy-level structured data. \nThe names of the new fields are built by concatenating the intermediate level field names.\n\nStep name: `flatten`\n\n| Name        | Description                                                                                                                                                                       |\n|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| delimiter   | the delimiter to use when concatenating the field names (default: `_`)                                                                                                            |\n| part        | when used with KeyValue data, defines if the transformation is done on the `key` or on the `value`. If `null` or absent the transformation applies to both the key and the value. |\n\n### Drop\n\nDrops the record from further processing. Use in conjunction with `when` to selectively drop records.\n\nStep name: `drop`\n\nParameters:\n\n| Name | Description                                                                                         |\n|------|-----------------------------------------------------------------------------------------------------|\n| when | by default, the record is dropped. Set this parameter to selectively choose when to drop a message. |\n\n#### Example\n\nUserConfig: `{\"steps\": [{\"type\": \"drop\", \"when\": \"value.firstName == value1\"}]}`\n\nInput: `{firstName: value1, lastName: value2} (AVRO)`\n\nOutput: N/A. Record is dropped.\n\n##### Example\n\nUserConfig: `{\"steps\": [{\"type\": \"flatten\"}]}`\n\nInput: `{field1: {field11: value11, field12: value12}} (AVRO)`\n\nOutput: `{field1_field11: value11, field1_field12: value12} (AVRO)`\n\n### Compute\n\nComputes new properties, values or field values based on an `expression` evaluated at runtime. If the field already exists, it will be overwritten.\n\nStep name: `compute`\n\nParameters:\n\n| Name   | Description                                                                                                                                |\n|--------|--------------------------------------------------------------------------------------------------------------------------------------------|\n| fields | an array of JSON objects describing how to calculate the field values. The JSON object represents a `field` as described in the next table |\n\n| Name (field)             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |\n|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| name                     | the name of the field to be computed. Prefix with `key.` or `value.` to compute the fields in the key or value parts of the message. In addition, you can compute values on the following message headers [`destinationTopic`, `messageKey`, `properties.`]. Please note that properties is a map of key/value pairs that are referenced by the dot notation, for example `properties.key0`                                                                                                                                                                                                                                                                                                                                                                  |\n| expression               | supports the [Expression Language](#expression-language) syntax. It is evaluated at runtime and the result of the evaluation is assigned to the field.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |\n| type                     | the type of the computed field. This will translate to the schema type of the new field in the transformed message. The following types are currently supported [`STRING`, `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BOOLEAN`, `DATE`, `TIME`, `TIMESTAMP`, `LOCAL_DATE_TIME`, `LOCAL_TIME`, `LOCAL_DATE`, `INSTANT`]. For more details about each type, please check the next table. Conversions are done using the rules described in [Type conversions](#type-conversions). The `type` field is not required for the message headers [`destinationTopic`, `messageKey`, `properties.`] and `STRING` will be used. For the value and key, if it is not provided, then the type will be inferred from the result of the expression evaluation. |\n| optional (default: true) | if true, it marks the field as optional in the schema of the transformed message. This is useful when `null` is a possible value of the compute expression.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |\n\n| Name (field.type) | Input                                                                     | Pulsar Schema Type | AVRO Schema Type             | Expression Examples                                                                                                                                                    |\n|-------------------|---------------------------------------------------------------------------|--------------------|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `STRING`          | A unicode character sequence.                                             | `STRING`           | `string`                     | \"'first name'\", \"fn:str(value)\", \"fn:concat(value, '-suffix')\"                                                                                                         |\n| `INT8`            | An 8-bit integer.                                                         | `INT8`             | `int`                        | \"127\", \"1 + 1\"                                                                                                                                                         |\n| `INT16`           | A 16-bit integer.                                                         | `INT16`            | `int`                        | \"32768\"                                                                                                                                                                |\n| `INT32`           | A 32-bit integer.                                                         | `INT32`            | `int`                        | \"2147483647\"                                                                                                                                                           |\n| `INT64`           | A 64-bit integer.                                                         | `INT64`            | `int`                        | \"9223372036854775807\"                                                                                                                                                  |\n| `FLOAT`           | A 32-bit floating point.                                                  | `FLOAT`            | `float`                      | \"340282346638528859999999999999999999999.999999\", \"1.1 + 1.1\"                                                                                                          |\n| `DOUBLE`          | A 64-bit floating point.                                                  | `DOUBLE`           | `double`                     | \"1.79769313486231570e+308\"                                                                                                                                             |\n| `BOOLEAN`         | true or false                                                             | `BOOLEAN`          | `boolean`                    | \"true\", \"1 == 1\", \"value.stringField == 'matching string'\"                                                                                                             |\n| `DATE`            | A date without a time-zone.                                               | Not supported      | `date` (logical)             | \"'2022-10-02'\", \"19267\" (days)                                                                                                                                         |\n| `TIME`            | A time without a time-zone.                                               | `TIME`             | `time-millis` (logical)      | \"'10:15:30'\", \"36930000\" (millis) since 00:00:00                                                                                                                       |\n| `TIMESTAMP`       | A timestamp in UTC time-zone.                                             | `TIMESTAMP`        | `timestamp-millis` (logical) | \"'2022-10-02T01:02:03+02:00'\", \"1664665323000\" (millis), \"fn:now()\"                                                                                                    |\n| `INSTANT`         | A timestamp in UTC time-zone.                                             | `INSTANT`          | `timestamp-millis` (logical) | \"'2022-10-02T01:02:03+02:00'\", \"1664665323000\" (millis), \"fn:now()\"                                                                                                    |\n| `LOCAL_DATE`      | A date without a time-zone.                                               | `LOCAL_DATE`       | `date`  (logical)            | \"'2022-10-02'\", \"19267\" (days)                                                                                                                                         |\n| `LOCAL_TIME`      | A time without a time-zone.                                               | `LOCAL_TIME`       | `time-millis` (logical)      | \"'10:15:30'\", \"36930000\" (millis) since 00:00:00                                                                                                                       |\n| `LOCAL_DATE_TIME` | A timestamp without a time-zone.                                          | `LOCAL_DATE_TIME`  | `timestamp-millis` (logical) | \"'2022-10-02T01:02:03+02:00'\", \"1664665323000\" (millis)                                                                                                                |\n| `BYTES`           | A sequence of 8-bit unsigned bytes.                                       | `BYTES`            | `bytes`                      | \"'input'.bytes\"                                                                                                                                                        |\n| `DECIMAL`         | arbitrary-precision signed decimal number of the form unscaled × 10-scale | Not supported      | `decimal` (logical)          | \"fn:decimalFromUnscaled(value, 38)\", where value is a byte array containing the two's-complement representation of the unscaled integer value in big-endian byte order |\n\n##### Example 1\n\nUserConfig: `{\"steps\": [{\"type\": \"compute\", \"fields\":[\n                {\"name\": \"key.newKeyField\",   \"expression\" : \"5*3\", \"type\": \"INT32\"},\"\n                {\"name\": \"value.valueField\",  \"expression\" : \"fn:concat(value.valueField, '_suffix')\", \"type\": \"STRING\"}]}\n             ]}`\n\nInput: `{key={keyField: key}, value={valueField: value}} (KeyValue\u003cAVRO, AVRO\u003e)`\n\nOutput: `{key={keyField: key, newKeyField: 15}, value={valueField: value_suffix}} (KeyValue\u003cAVRO, AVRO\u003e)`\n\n##### Example 2\n\nUserConfig: `{\"steps\": [{\"type\": \"compute\", \"fields\":[\n                {\"name\": \"destinationTopic\",   \"expression\" : \"'routed'\"},\n                {\"name\": \"properties.k1\", \"expression\" : \"'overwritten'\"},\n                {\"name\": \"properties.k2\", \"expression\" : \"'new'\"}]}\n             ]}`\n\nInput: `{key={keyField: key}, value={valueField: value}} (KeyValue\u003cAVRO, AVRO\u003e), headers=destinationTopic: out1, propertes: {k1:v1}`\n\nOutput: `{key={keyField: key}, value={valueField: value}} (KeyValue\u003cAVRO, AVRO\u003e), headers=destinationTopic:routed, propertes: {k1:overwritten, k2:new}`\n\n### Expression Language\nIn order to support [Condition Steps](#conditional-steps) and the [Compute](#compute) Transform, an expression language is required to evaluate the conditional step `when` or the compute step `expression`.\nThe syntax is ([EL](https://javaee.github.io/tutorial/jsf-el001.html#BNAHQ)) like that uses the dot notation to access field properties or map keys.\nIt supports the following operators and functions:\n\n#### Operators\nThe Expression Language supports the following operators:\n* Arithmetic: +, - (binary), *, / and div, % and mod, - (unary)\n* Logical: and, \u0026\u0026, or, ||, not, !\n* Relational: ==, eq, !=, ne, \u003c, lt, \u003e, gt, \u003c=, ge, \u003e=, le.\n\n#### Functions\nUtility methods available under the `fn` namespace.\nFor example, to get the current timestamp, use 'fn:now()'.\nThe Expression Language supports the following functions:\n* `toDouble(input)`: Converts the input value to a DOUBLE number, If the input is `null`, it returns `null`.\n* `toInt(input)`: Converts the input value to an INTEGER number, If the input is `null`, it returns `null`.\n* `uppercase(input)`: Returns the string `input` uppercased, If the input is `null`, it returns `null`.\n* `lowercase(input)`: Returns the string `input` lowercased, If the input is `null`, it returns `null`.\n* `contains(input, value)`: Returns the boolean `true` if `value` exists in `input`. If `input` or `value` is `null`, it returns `false`. \n* `trim(input)`: Returns the `input` string with all leading and trailing spaces removed.\n* `concat(input1, input2)`: Returns a string concatenation of `input1` and `input2`. If either input is `null`, it is treated as an empty string.\n* `coalesce(value, valueIfNull)`: Returns `value` if it is not `null`, otherwise returns `valueIfNull`.\n* `replace(input, regex, replacement)`: Replaces each substring of `input` that matches the `regex` regular expression with `replacement`. See [Java's replaceAll](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/String.html#replaceAll(java.lang.String,java.lang.String)).\n* `str(input)`: Converts `input` to a string.\n* `toJson(input)`: Converts `input` to a JSON string.\n* `fromJson(input)`: Parse `input` as JSON.\n* `split(input, separatorExpression)`: Split the input to a list of strings, this is internally using the String.split() function. An empty input corresponds to an empty list. The input is convered to a String using the str() function.\n* `now()`: Returns the current timestamp.\n* `timestampAdd(input, delta, unit)`: Returns a timestamp formed by adding `delta` in `unit` to the `input` timestamp. \n  * `input` a timestamp to add to.\n  * `delta` a `long` amount of `unit` to add to `input`. Can be a negative value to perform subtraction.\n  * `unit` the string unit of time to add or subtract. Can be one of [`years`, `months`, `days`, `hours`, `minutes`, `seconds`, `millis`].\n* `decimalFromUnscaled(input, scale)`: Converts `input` to a `BigDecimal` with the given `scale`. \n  * `input` unscaled value of the BigDecimal. Can be any of STRING, INTEGER, LONG or Array of bytes containing the two's-complement representation in big-endian byte order.\n  * `scale` the scale of the `BigDecimal` to create.\n* `decimalFromNumber(input)`: Converts `input` to a `BigDecimal`.\n    * `input` value of the BigDecimal in DOUBLE or FLOAT. If INTEGER or LONG is provided, an unscaled BigDecimal value will be returned.\n* `filter(collection, expression)`: Returns a new collection containing only the elements of `collection` for which `expression` is `true`. The current element is available under the `record` variable. An example is fn:filter(value.queryResults, \"fn:toDouble(record.similarity) \u003e= 0.5\")\nFor all methods, if a parameter is not in the right type, a conversion will be done using the rules described in [Type conversions](#type-conversions).\nFor instance, you can do `fn:timestampAdd('2022-10-02T01:02:03Z', '42', 'hours'.bytes)`\n* `unpack(input, fieldsList)`: Returns a map containing the elements of `input`, for each field in the `fieldList` you will see an entry in the map. If the input is a string it is converted to a list using the `split()` function with the ',' separator\n\nWhen a function returns a timestamp, its type is `INSTANT`.\n\n#### Conditional Steps\n\nEach step accept an optional `when` configuration that is evaluated at step execution time against current record (i.e. the as seen by\nthe current step in the transformation pipeline). The `when` condition supports the [Expression Language](#expression-language) syntax. It provides access to the record attributes as follows:\n* `key`: the message key or the key portion of the record in a KeyValue schema.\n* `value`: the value portion of the record in a KeyValue schema, or the message payload itself.\n* `topicName`: the optional name of the topic which the record originated from (aka. Input Topic).\n* `destinationTopic`: the name of the topic on which the transformed record will be sent (aka. Output Topic).\n* `eventTime`: the optional timestamp attached to the record from its source. For example, the original timestamp attached to the pulsar message.\n* `properties`: the optional user-defined properties attached to record\n\nYou can use the `.` operator to access top level or nested properties on a schema-full `key` or `value`. For example, `key.keyField1` or `value.valueFiled1.nestedValueField`. You can also use to access different keys of the user defined properties. For example, `properties.prop1`.\n\n#### Example 1: KeyValue (KeyValue\u003cAVRO, AVRO\u003e)\n\n```json\n{\n  \"key\": {\n    \"compound\": {\n      \"uuid\": \"uuidValue\",\n      \"timestamp\": 1663616014\n    },\n    \"value\" : {\n      \"first\" : \"f1\",\n      \"last\" : \"l1\",\n      \"rank\" : 1,\n      \"address\" : {\n        \"zipcode\" : \"abc-def\"\n      }\n    }\n  }}\n```\n\n| when                                                         | Evaluates to |\n|--------------------------------------------------------------|--------------|\n| `\"key.compound.uuid == 'uudValue'\"`                          | True         |\n| `\"key.compound.timestamp \u003c= 10\"`                             | False        |\n| `\"value.first == 'f1' \u0026\u0026 value.last.toUpperCase() == 'L1'`   | True         |\n| `\"value.rank \u003c= 1 \u0026\u0026 value.address.substring(0, 3) == 'abc'` | True         |\n\n#### Example 2: (Primitive string schema with metadata)\n\n* Partition Key: `key1`\n* Source topic: `topic1`\n* User defined k/v: `{\"prop1\": \"p1\", \"prop2\": \"p2\"}`\n* Payload (String): `Hello world!`\n\n| when                                        | Evaluates to |\n|---------------------------------------------|--------------|\n| `\"key == 'key1' or topicName == 'topic1' \"` | True         |\n| `\"value == 'Hello world!'\"`                 | True         |\n| `\"properties.prop1 == 'p2'\"`                | False        |\n\n## Deployment\n\nSee [the Pulsar docs](https://pulsar.apache.org/fr/docs/functions-deploy) for more details on how to deploy a Function.\n\n### Deploy as a non built-in Function\n\n* Create a Transformation Function providing the path to the Pulsar Transformations NAR.\n```shell\npulsar-admin functions create \\\n--jar pulsar-transformations-2.0.0.nar \\\n--name my-function \\\n--inputs my-input-topic \\\n--output my-output-topic \\\n--user-config '{\"steps\": [{\"type\": \"drop-fields\", \"fields\": \"password\"}, {\"type\": \"merge-key-value\"}, {\"type\": \"unwrap-key-value\"}, {\"type\": \"cast\", \"schema-type\": \"STRING\"}]}'\n```\n\n### Deploy as a built-in Function\n\n* Put the Pulsar Transformations NAR in the `functions` directory of the Pulsar Function worker (or broker).\n```shell\ncp pulsar-transformations-2.0.0.nar $PULSAR_HOME/functions/pulsar-transformations-2.0.0.nar\n```\n* Restart the function worker (or broker) instance or reload all the built-in functions:\n```shell\npulsar-admin functions reload\n```\n* Create a Transformation Function with the admin CLI. The built-in function type is `transforms`.\n```shell\npulsar-admin functions create \\\n--function-type transforms \\\n--name my-function \\\n--inputs my-input-topic \\\n--output my-output-topic \\\n--user-config '{\"steps\": [{\"type\": \"drop-fields\", \"fields\": \"password\"}, {\"type\": \"merge-key-value\"}, {\"type\": \"unwrap-key-value\"}, {\"type\": \"cast\", \"schema-type\": \"STRING\"}]}'\n```\n\n### Deploy the Transformation Function coupled with a Pulsar Sink\n\n*This requires Datastax Luna Streaming 2.10.1.6+.*\n\nThanks to [PIP-193](https://github.com/apache/pulsar/issues/16739) it's possible to execute a function inside a sink process, removing the need of temporary topics.\n\n* Create a Pulsar Sink instance with `transform-function` Transformation Function with the admin CLI.\n```shell\npulsar-admin sinks create \\\n--sink-type \u003csink_type\u003e \\\n--inputs my-input-topic \\\n--tenant public \\\n--namespace default \\\n--name my-sink \\\n--transform-function \"builtin://transforms\" \\\n--transform-function-config '{\"steps\": [{\"type\": \"drop-fields\", \"fields\": \"password\"}, {\"type\": \"merge-key-value\"}, {\"type\": \"unwrap-key-value\"}, {\"type\": \"cast\", \"schema-type\": \"STRING\"}]}'\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fpulsar-transformations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatastax%2Fpulsar-transformations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fpulsar-transformations/lists"}