{"id":21482792,"url":"https://github.com/ibmstreams/streamsx.transform","last_synced_at":"2025-03-17T09:22:40.501Z","repository":{"id":36190880,"uuid":"40495080","full_name":"IBMStreams/streamsx.transform","owner":"IBMStreams","description":"Toolkit that contains basic building block operators that efficiently transform data from one format to another","archived":false,"fork":false,"pushed_at":"2020-11-10T09:19:14.000Z","size":910,"stargazers_count":0,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"develop","last_synced_at":"2025-01-23T18:50:36.705Z","etag":null,"topics":["ibm-streams","stream-processing","toolkit"],"latest_commit_sha":null,"homepage":"https://ibmstreams.github.io/streamsx.transform","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IBMStreams.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-10T17:07:49.000Z","updated_at":"2020-11-10T10:45:19.000Z","dependencies_parsed_at":"2022-08-02T12:58:40.262Z","dependency_job_id":null,"html_url":"https://github.com/IBMStreams/streamsx.transform","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBMStreams%2Fstreamsx.transform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBMStreams%2Fstreamsx.transform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBMStreams%2Fstreamsx.transform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBMStreams%2Fstreamsx.transform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IBMStreams","download_url":"https://codeload.github.com/IBMStreams/streamsx.transform/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244006303,"owners_count":20382443,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ibm-streams","stream-processing","toolkit"],"created_at":"2024-11-23T12:36:48.477Z","updated_at":"2025-03-17T09:22:40.472Z","avatar_url":"https://github.com/IBMStreams.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# streamsx.transform\nThis toolkit contains general-purpose operators that do tuple manipulation.  Currently, the only operator it contains is Modify, but operators that perform joins and aggregations fit would fit in this toolkit.\n\nGeneral-purpose operators that perform manipulations on streams without looking at the tuples should go into streamsx.plumbing. \n\n## Modify\n\nThe Modify operator can be used as a replacement for Functor when the input types and output types are the same.   For large tuples, using Modify may improve performance, and will never hurt performance.\n\nFunctor creates a new output tuple from the input tuple, while Modify modifies the input tuple and place it on the output stream.  When tuples are large and the modification is small, Functor can result is a lot of data copying, which can impact performance.  \n\n\nAs an example, consider application in tests/timing.  The application applies Functor or Modify three times in a row on the input tuple, incrementing one attribute each time.  The input tuple has three attributes, one of which is a list.  When the list is small, there is little difference between `Functor` and `Modify`.  When the list is large, the performance difference can be quite large.  Here's some numbers from a run on my machine: \n\n| Operator | list size 1 | list size 100 |\n-----------|-------------|---------------|\nModify     |    18.8s    | 23.8s         |\nFunctor    |    22.8s    | 123s          |\n\n\n\n## Using the Modify Operator to Reduce Copying\nWe've made the `Modify` operator available as part of [streamsx.transform](http://ibmstreams.github.io/streamsx.transform/) on Github. It is used like Functor, but unlike Functor, it modifies its input tuple rather than creating a new one. Because it submits the same tuple it received, it's limited to the case where the input type and output type are the same. Because it doesn't copy tuple attributes, using `Modify` instead of `Functor` can speed up your application, especially when your tuple size is large and the application uses `partitionColocation` statements. The first your first step in optimizing your application is to go [here](https://ibmstreams.github.io/streamsx.transform/docs/knowledge/overview/). Trying to eliminate extra copies should only come after the more basic steps described there. The rest of this post dives into the guts of Streams to explain why this can make a difference, and why we had to create a separate operator rather than modifying `Functor`.\n\n## Functor vs Filter\n\nLet me start by looking at `Functor`. `Functor` lets you transform one tuple into another, and it also comes with a filter parameter to let you drop tuples you don't want to pass through. Let's say your input tuple and output tuple are the same, but you just want to filter out some of them. You could use a `Functor`:\n\n\u003cpre\u003estream\u003cData\u003e Filtered = Functor(Data) {\n    param filter: x \u003e 5;\n}\n\u003c/pre\u003e\n\nor you could use a `Filter`\n\n\u003cpre\u003estream\u003cData\u003e Filtered = Filter(Data) {\n   param filter: x \u003e 5;\n}\n\u003c/pre\u003e\n\nLet's pause for a moment. What's the difference between the two? While the two snippets look very similar, the generated C++ is different. From Filter, notice that the input tuple itself is submitted:\n\n\u003cpre\u003e    IPort0Type const \u0026 iport$0 = static_cast\u003cIPort0Type const \u0026\u003e(tuple);\n    if (lit$0)\n        submit(tuple, 0);\n\u003c/pre\u003e\n\nFrom Functor, notices that a new tuple is created, and the new tuple is submitted:\n\n\u003cpre\u003e    IPort0Type const \u0026 iport$0 = static_cast\u003cIPort0Type const \u0026\u003e(tuple);\n    if (!(lit$0) )\n        return;\n    { OPort0Type otuple(iport$0); submit(otuple, 0); }\n\u003c/pre\u003e\n\nThe difference is Filter operator passes on the same tuple it received, while the `Functor` makes a new tuple. When the operator is connected to its downstream operator via a partition colocation statement (or because they are in a standalone) and the the tuple size is large, the `Functor` is much more expensive because it is copying every attribute in the tuple, while `Filter` is not. I measured this performance difference for input tuple containing a list of 100 integers where the filter expressions were true. The `Filter` is about three times as as fast `Functor`--for 100,000,000 tuples, about 17 seconds for `Filter` and 45 for `Functor`. `Modify` came out of a desire to have an operator that could be used like `Functor` but didn't create new tuples.\n\n## Introducing `Modify`\n\nTo speed up the case when the tuples are large and only a small change is being made, I created the `Modify` operator. It's used exactly like the `Functor`, except that its input type and output type must be the same. Internally, though, unlike `Functor` it modifies the input tuple in place and then sends it on. Let's say we write something like this:\n\n\u003cpre\u003estream\u003cData\u003e Increment = Modify(Data) {\noutput Increment:\n   x = x +1;\n}\n\nHere's Modify:\u003c/pre\u003e\n\n\u003cpre\u003e       iport$0.set_x(SPL::int32(iport$0.get_x() + lit$0)); \n       submit(iport$0,0);\n\u003c/pre\u003e\n\nfrom an equivalent use of Functor. Note the newly created otuple.\n\n\u003cpre\u003e    OPort0Type otuple(SPL::int32(iport$0.get_x() + lit$0), iport$0.get_lotsOfData()); \n    submit (otuple, 0);\n\u003c/pre\u003e\n\nIn my quick test, this can result in a significant performance improvement for large tuple sizes--23 seconds for `Modify` versus 45 seconds for `Functor`. The next section is where we really dive into the guts of Streams.\n\n### Why do we need a new operator?\n\nIt is tempting to think we can add this modify-in-place capability to `Functor`--that is, if the input type and output type are the same, generate code for `Modify`, and otherwise, generate code as it does now. This would give us one operator that works like `Modify` when input and output types are the same, and `Functor` otherwise. But doing this would actually slow down applications in some circumstances, and to explain why, I'm going to have to describe a feature of the Streams runtime you've probably been using but may never have noticed. Consider the following application:\n\n\u003cpre\u003estream\u003cint32 x,int32 y\u003e Data = Beacon() {\noutput Data:\n   x = 0,y=0;\n}\n\nstreams\u003cint32 x,int32 y\u003e IncrX = Modify(Data) {\noutput IncrX:\n    x=x+1;\n}\n\nstreams\u003cint32 x,int32 y\u003e IncrY = Modify(Data) {\n   output IncrY: \n      y=y+1;\n} \n\u003c/pre\u003e\n\nYou'd expect that all the tuples on IncrX stream have `x=1` and `y=0` and that all the tuples on the IncrY stream have `x=0` and `y=1`. When the `Beacon` is in a different PE than `IncrX` and `IncrY`, the tuple from the `Beacon` is copied to `IncrX` and `IncrY`. But when we use `partitionColocation` to put them inside a shared PE, or when its run in standalone mode, tuples are passed by reference between operators. To ensure this doesn't result in the same tuple being changed by IncrX and IncrY, in order to prevent the output from having a tuple with both x and y incremented, the SPL runtime needs to make a copy of the tuples on the Data stream, to ensure that the same C++ object isn't sent to both `IncrX` and `IncrY`. But now let's look at that same app with `Functor` instead of `Modify`:\n\n\u003cpre\u003ecomposite NoRuntimeCopy {\n\ngraph\nstream\u003cint32 x,int32 y\u003e Data = Beacon() {\noutput Data:\n   x = 0,y=0;\n}\n\nstreams\u003cint32 x,int32 y\u003e IncrX = Functor(Data) {\noutput IncrX:\n    x=x+1;\n}\n\nstreams\u003cint32 x,int32 y\u003e IncrY = Functor(Data) {\n   output IncrY: \n      y=y+1;\n} \n}\n\u003c/pre\u003e\n\nIn this case, the SPL runtime does NOT need to copy the tuples on the Data stream. Since `Functor` does not modify its input tuples, it can send the same tuple to `IncrX` and to `IncrY`, and there won't be a tuple with `x==1` and `y==1`. And, in fact, the SPL runtime does not make a copy in this case, and uses the same tuple. This brings up the question: How does the SPL runtime know whether it's dealing with an operator like `Modify` for which it needs to make a copy, or an operator like `Functor`? To do make that decision, it uses the operator model. Remember that that operator model is the same for all invocations of the operator, no matter its input or output tuples. One of the properties of an input port in the operator model is `tupleMutationAllowed`. If we allow `Functor` to sometimes change its input tuple, then we'd have to change the tuple mutation allowed to be true, and the runtime wouldn't know when it was safe to skip a copy, and we'd do extra copying in some cases. As far as the SPL runtime is concerned, there's one big difference between `Functor` and `Modify`: `Functor` agrees not to change or update its input, but `Modify` makes no such promises.\n\n## Final notes\n\nThis post can be summed as use `Modify` instead of `Functor` when your input and output types are the same. But the broader message here is to be aware of when the Streams runtime and operators are copying tuples. This post is actually a simplification; the SPL runtime uses more information than just the `tupleMutationAllowed` attribute to decide whether a copy is necessary. See [here](https://www.ibm.com/support/knowledgecenter/nl/SSCRJU_4.3.0/com.ibm.streams.dev.doc/doc/str_portmutability.html) for details.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibmstreams%2Fstreamsx.transform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fibmstreams%2Fstreamsx.transform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibmstreams%2Fstreamsx.transform/lists"}