{"id":19646580,"url":"https://github.com/amzn/rheoceros","last_synced_at":"2025-07-13T06:36:34.458Z","repository":{"id":46822990,"uuid":"465503341","full_name":"amzn/rheoceros","owner":"amzn","description":"Cloud-based AI / ML workflow and data application development framework","archived":false,"fork":false,"pushed_at":"2024-08-20T16:52:04.000Z","size":2616,"stargazers_count":17,"open_issues_count":0,"forks_count":9,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-05T09:11:36.823Z","etag":null,"topics":["ai","aws","aws-emr","aws-glue","aws-lambda","bring-your-own-account","cloud","data-science","event-based","feature-engineering","flow","low-code-framework","machine-learning","pyspark","sagemaker-notebook","sagemaker-notebook-instance","scala-spark","serverless","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amzn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-02T23:36:02.000Z","updated_at":"2025-04-04T02:11:37.000Z","dependencies_parsed_at":"2024-06-01T01:55:28.138Z","dependency_job_id":"b85446c1-07f9-4df5-96cb-acdbf7d78c79","html_url":"https://github.com/amzn/rheoceros","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amzn%2Frheoceros","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amzn%2Frheoceros/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amzn%2Frheoceros/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amzn%2Frheoceros/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amzn","download_url":"https://codeload.github.com/amzn/rheoceros/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251338545,"owners_count":21573573,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","aws","aws-emr","aws-glue","aws-lambda","bring-your-own-account","cloud","data-science","event-based","feature-engineering","flow","low-code-framework","machine-learning","pyspark","sagemaker-notebook","sagemaker-notebook-instance","scala-spark","serverless","spark"],"created_at":"2024-11-11T14:39:20.408Z","updated_at":"2025-04-28T15:30:41.878Z","avatar_url":"https://github.com/amzn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RheocerOS\n\nA cloud-based, lightweight framework designed to push the state-of-art in AI team workflows. It was designed and \nimplemented by one of AI teams at Amazon, as part of an \"innovation speed/momentum\" initiative.\n\n* Introduces a new development methodology for cloud-based big-data/ML applications with the highest level of infrastructure/cloud abstraction and automation\n* Provides a common development and runtime platform for both scientists and engineers\n  * Accelerates transition from experimentation into enterprise grade AI applications\n* Auto-provisions and updates isolated model-driven (AWS) application(s) in your own account\n\nIn RheocerOS, events as Signals are first-class citizens. Entire workflows are created with a new paradigm that makes\nyou think more functional and more in sync with the actual/natural flow of information among abstracted cloud entities \nsuch as datasets, timers, metrics, alarms, ML models from your cloud stack. \n\nProgramming experience is all about creating and binding those entities dynamically in high-level Python code.\n\nTherefore, this new programming paradigm naturally eliminates the complexities in:\n* backfilling\n* sandboxing / cloning\n* code or data sharing / seamless collaboration \n* parallel branch executions and programmatic control of those executions from different endpoints (e.g notebooks, desktops)\n* hybrid use of compute resources (do ETL with Spark on AWS Glue, with PrestoSQL on Athena, run inlined code on Lambda) across different nodes in the same application  \n\n### Who can use it?\n\nCurrently anyone who has access to an AWS account can start experimenting with an application RheocerOS within 10 mins, \nwithout leaving their popular Python development environment. Without an AWS account, unit-test modules can be run and analyzed to get an idea about the development experience.\n\nIn AI/big-data/data-science teams, it honors the use-cases and the needs of all stakeholders and the possible interactions\nbetween them:\n* Scientists\n* Engineers\n* PMs\n* Upstream and downstream teams\n\nRheocerOS implicitly enforces/applies AWS best practices for all those stakeholders.\nAllows data-mesh / ecosystem formation, easy data-sharing across production systems or their personal stacks.\n\n## High-Level Workflow\n\n![Workflow](./doc/resources/RheocerOS_High_Level_Workflow.drawio.png \"High Level Workflow\")\n\n### Development Experience\n\n* Basic Python knowledge\n* In big-data nodes, use:\n  * PySpark/SQL (via Glue v1, v2, v3, EMR [beta])\n  * Spark/SQL (via Glue [Scala] v1, v2, v3, EMR [beta])\n  * PrestoSQL (currently via AWS Athena)\n  * Or contribute to the framework: new driver!\n* Basic understanding and intuition around CW metrics/alarms\n* Write code anywhere and run anywhere\n\n#### Setup\n\nSee [DEVELOPMENT.md](./doc/DEVELOPMENT.md)\n\n\n#### Application Code\n\nSome examples from [examples](./examples)\n\nYou should be able to run these examples directly from within rheoceros\npackage or in your own application Python package that relies on rheoceros.\n\nCredentials:\n\n* As described within the sample codes below, these examples rely on default AWS credentials on your machine (e.g ~/.aws/credentials or env variables used by boto).\n* If you want to pass the credentials programmatically, then use 'access_id' and 'access_key' parameters of AWSApplication class.\n* AWS Entity represented by the credentials should have the rights to:\n  * create/delete/modify the prospective dev-role of the application which is to be created\n    * The IAM role is named in this format: ```\u003capp_name\u003e-\u003caws_region\u003e-IntelliFlowDevRole```. You can use this to complete the policy of your admin user.\n  * Or do anything / admin :)\n  * Note: so we can call it as the bootstrapper entity as well, RheocerOS creates the dev-role (if not exists) and assumes into it as soon as possible within the init sequence of AWSApplication\n\nWarning: If you are on a Sagemaker notebook, just make sure that the IAM role has the same permissions listed above.\n\n* Example 1\n\nSample data-pipeline.\n\n```python\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n\nimport intelliflow.api_ext as flow\nfrom intelliflow.api_ext import *\nimport logging\n\nlog = flow.init_basic_logging(root_level=logging.WARN)\n\n# automatically reads default credentials (and creates this app in that AWS account)\n# default credentials should belong to an entity that can either:\n#  - do everything (admin, use it for quick prototyping!)\n#  - can create/delete roles (dev role of this app) that would have this app-name and IntelliFlow in it\n#  - can get/assume its dev-role (won't help if this is the first run)\napp = AWSApplication(\"dxg-proto\", \"us-east-1\")\n\ntimer_signal_daily = app.add_timer(id=\"daily_timer\"\n                                   # https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule-schedule.html#rate-expressions\n                                   , schedule_expression=\"rate(1 day)\"  # you can use CRON tab here as well\n                                   , time_dimension_id=\"day\")\n\npdex = app.marshal_external_data(\n  external_data_desc=S3Dataset(\"111222333444\",\n                               \"dxg-dev-external-data-poc\",\n                               \"pdex\",\n                               # placeholder for 'region'\n                               \"{}\",\n                               # placeholder for 'day'\n                               \"{}\",\n                               dataset_format=DataFormat.JSON)\n  # in spark code, this will be the default DataFrame name/alias (but you can overwrite it in 'inputs' if you use map instead of list)\n  , id=\"pdex\"\n  , dimension_spec= {\n    # partition 1\n    'region': {\n      'type': DimensionType.STRING,\n\n      # partition 2\n      'day': {\n        'type': DimensionType.DATETIME,\n        'granularity': DatetimeGranularity.DAY,  # redundant because IF's default is day\n        'format': '%Y-%m-%d' # format of day partition in resource (s3) path, e.g 2021-06-21\n      }\n\n    }\n  }\n  # this dataset currently does not use any 'completion' file.\n  # But declaring this will at least eliminate any update from the upstream partition causing event processing/trigger in this pipeline.\n  # By using this today, framework will not allow this to be used in an event-based manner. why? because this file '_SUCCESS' is\n  # not being generated today. If we don't declare this and use 'pdex' signal in an event-based manner, it will cause\n  # event ingestion (and evaluation for) for each partition file! Idempotency is there but if events are spread out\n  # then they might cause re-calculations.\n  , protocol=SignalIntegrityProtocol(\"FILE_CHECK\", {\"file\": \"_SUCCESS\"})\n)\n\ntpdex = app.create_data(id=\"tpdex\",  # default DataFrame ID in downstream nodes (if not overwritten)\n                        inputs=[pdex['NA']['*'].ref.range_check(True), # range_check -\u003e True, make sure that timer signal indicated daily partition will be available before the execution, otherwise block\n                                timer_signal_daily],\n                        # ref -\u003e dont wait for a new event, range_check: make sure []\n                        compute_targets=[\n                          BatchCompute(\n                            scala_script(\"\"\"                                 \n\n                                          pdex\n                                            .filter(\n                                              (col(\"addressId\") isNotNull)\n                                                \u0026\u0026 (col(\"requestId\") isNotNull)\n                                                \u0026\u0026 (col(\"asin\") isNotNull)\n                                                \u0026\u0026 (col(\"entityId\") isNotNull)\n                                                \u0026\u0026 (col(\"deliveryOption\") isNotNull)\n                                            )\n                                            .select(\n                                              pdex.col(\"*\"),\n                                              explode(col(\"deliveryOption\")).as(\"flatDeliveryOption\")\n                                            )\n                                            .select(\n                                              \"datestamp\",\n                                              \"entityId\",\n                                              \"requestId\",\n                                              \"marketplaceId\",\n                                              \"asin\",\n                                              \"sessionId\",\n                                              \"merchant_id\",\n                                              \"client_id\",\n                                              \"addressId\",\n                                              \"offerSku\"\n                                            )\n                                            .distinct()\n                                               \"\"\"\n                                         # ,\n                                         # external_library_paths=[\"s3://BUCKET_FOR_JAR/FOLDER/SUPER-JAR-NAME.jar\"]\n                                         ),\n                            lang=Lang.SCALA,\n                            GlueVersion=\"2.0\",\n                            # Glue specific job params (see boto API)\n                            WorkerType=GlueWorkerType.G_1X.value,\n                            NumberOfWorkers=50\n                          )\n                        ])\n\napp.activate()\n\n# emulate the injection of 'timer_signal_daily' input as an event with a specific partition value\napp.process(timer_signal_daily['2021-09-20'], target_route_id=tpdex)\n\napp.poll(tpdex['NA']['2021-09-20'])\n\n# app.execute(tpdex['NA']['2021-09-20'])\n\n```\n\nYou or one of your team-mates (if authorized by you) can run the following code in her/his notebook or IDE:\n\n```python\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n\nimport intelliflow.api_ext as flow\nfrom intelliflow.api_ext import *\nimport logging\nfrom datetime import datetime, timedelta\n\nlog = flow.init_basic_logging(root_level=logging.WARN)\n\napp = AWSApplication(\"dxg-proto\", \"us-east-1\")\n\n# retrieve the reference for tpdex node\ntpdex_node = app['tpdef']\n\n# in a new Jupyter cell\n# execute  it for tomorrow for the same region (blocking call by default)\napp.execute(tpdex_node['NA']['2021-09-21'])\n\n# in a new Jupyter cell\n# backfill the node for 60 days by consuming the entire AWS Glue Capacity in the account\nfor i in range(60):\n  # don't wait for executions, fork new batch compute execution in AWS\n  app.execute(tpdex_node['NA'][datetime(2021,09,21) + timedelta(1)], wait=False)\n\n# in a new Jupyter cell\n# JOIN logic (as in multi-threading)\n# wait for all executions\nfor i in range(60):\n    path, records = app.poll(tpdex_node['NA'][datetime(2021,09,21) + timedelta(1)])\n    assert path, f\"Execution failed! Failed records {records!r}\"\n```\n\nIf you are done with the application (maybe 1 week later), just terminate and delete it (wipe it out from AWS):\n\n```python\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n\nimport intelliflow.api_ext as flow\nfrom intelliflow.api_ext import *\n\napp = AWSApplication(\"dxg-proto\", \"us-east-1\")\napp.terminate() # resources except the storage (data) + IAM roles\napp.delete() # data + IAM roles\n```\n\nThere will be no trace of that application in your account\n\n* Example 2\n\nSingle-branch, basic backfilling demonstration with dummy compute targets for your experimentation.\n\nRecursive execution mechanism that can even be used for generic 'backfilling'.\n\nCrawl up the chain, \"uptream/ancestor\" nodes. Orchestration and event-propagation in cloud.\n\n```python\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n\nimport intelliflow.api_ext as flow\nfrom intelliflow.api_ext import *\nfrom intelliflow.utils.test.inlined_compute import NOOPCompute\n\nfrom datetime import datetime, timedelta\n\nflow.init_basic_logging()\n\ntoday = datetime.now()\n\napp = AWSApplication(\"backfill-1\", \"us-east-1\", access_id=\u003cAWS_ACCESS_ID\u003e, access_key=\u003cAWS_ACCESS_KEY\u003e)\n \na = app.create_data(\"A\",\n                    inputs=[app.add_timer(\"T\", \"rate(1 day)\")],\n                    compute_targets=[NOOPCompute])\n\nb = app.create_data(\"B\", inputs=[a[:-2]], compute_targets=[NOOPCompute])\n\napp.execute(b[today], wait=True, recursive=True)\n\nassert app.poll(b[today])[0], \"Execution must have been successful!\"\nassert app.poll(a[today - timedelta(1)])[0], \"Recursive execution on node 'A' could not be found!\"\n\napp.pause()\n```\n\nAnother backfilling example with a relatively more complex topology:\n\n```python\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n\nimport intelliflow.api_ext as flow\nfrom intelliflow.api_ext import *\n\nfrom intelliflow.utils.test.inlined_compute import NOOPCompute\n\nfrom datetime import datetime, timedelta\n\nflow.init_basic_logging()\n\ntoday = datetime.now()\napp = AWSApplication(\"backfill-2\", \"us-east-1\")\n \next = app.marshal_external_data(\n    S3Dataset(account_id=\"111222333444\",\n              bucket=\"bucket-exec-recurse\",\n              folder=\"data\", \n              \"{}\") # represent the partition format in S3\n    , id=\"ext_data_1\"\n    , dimension_spec={\n        'day': {\n            'type': DimensionType.DATETIME,\n            'format': '%Y-%m-%d'\n        }\n    }\n    , protocol=SignalIntegrityProtocol(\"FILE_CHECK\", {\"file\": \"_SUCCESS\"})\n)\n\n# note that \"A\" depends on a range of 'NOW - 2' partitions on 'ext_data_1'\na = app.create_data(\"A\",\n                    inputs=[\n                        ext[:-2].nearest(),  # scan last two days and pick the most recent (nearest) partition\n                        app.add_timer(\"T\", \"rate(1 day)\", time_dimension_id=\"day\")  # having the partition name as 'day' here auto-links its dimension with S3 dataset's day dimension/partition from above\n                    ],\n                    compute_targets=[NOOPCompute])\n\n# note that \"B\" depends on a range of 'NOW - 2' partitions on 'ext_data_1'\nb = app.create_data(\"B\", inputs=[ext[:-2]], compute_targets=[NOOPCompute])\n\nc = app.create_data(\"C\", inputs=[ext], compute_targets=[NOOPCompute])\n\n# use 'nearest' (which would trigger execution on nearest, TIP partition if entire range is missing)\n# and 'ref' (which would still go through entire range check when 'recursive'=True)\nd = app.create_data(\"D\", inputs=[a[:-15].nearest(), b[:-2].ref, c], compute_targets=[NOOPCompute])\n\napp.activate()\n\ntoday = datetime.now()\n\n# rheoceros \"just make sure that d for today is ready\"\n# note: if the python process crashes, re-run the framework will hook up with the ongoing execution chain in orchestration with minimum redundant job ru-runs\napp.execute(d[today], wait=True, recursive=True)\n\n# ancestor \"a, b, c\" data will also be ready at this point\n\napp.pause()\n```\n\n* Example 3\n\nA sample pipeline that makes a hybrid use of different runtimes (versions x languages).\n\n```python\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n\nimport intelliflow.api_ext as flow\nfrom intelliflow.api_ext import *\n\nflow.init_basic_logging()\n\napp = AWSApplication(\"scala-python\", \"us-east-1\")\n\nd_ad_orders_na = app.glue_table(\"dex_ml_catalog\", \"d_ad_orders_na\")\n\nrepeat_d_ad_orders_na_scala = app.create_data(id=\"REPEAT_AD_ORDERS_IN_SCALA\",\n                                        inputs=[d_ad_orders_na],\n                                        compute_targets=[\n                                            BatchCompute(\n                                                scala_script(\"d_ad_orders_na.limit(100)\"\n                                                             #, external_library_paths=[\"\u003cYOUR_S3_FILE_PATH\u003e\"]\n                                                             ),\n                                                lang=Lang.SCALA,\n                                                GlueVersion=\"3.0\",\n                                                # Glue specific params (see boto API)\n                                                WorkerType=GlueWorkerType.G_1X.value,\n                                                NumberOfWorkers=50,\n                                                Timeout=3 * 60  # 3 hours\n                                            )\n                                        ])\n\nrepeat_d_ad_orders_na_python = app.create_data(id=\"REPEAT_AD_ORDERS_IN_PYTHON\",\n                                               inputs=[d_ad_orders_na],\n                                               compute_targets=[\n                                                   BatchCompute(\n                                                       \"output = d_ad_orders_na.limit(100)\",\n                                                       GlueVersion=\"3.0\",\n                                                       WorkerType=GlueWorkerType.G_1X.value,\n                                                       NumberOfWorkers=50,\n                                                       Timeout=3 * 60  # 3 hours\n                                                   )\n                                               ])\n\nrepeat_d_ad_orders_na_scala_complex = app.create_data(id=\"REPEAT_AD_ORDERS_IN_SCALA_COMPLEX\",\n                                              inputs=[d_ad_orders_na],\n                                              compute_targets=[\n                                                  BatchCompute(\n                                                      scala_script(\"\"\"import com.amazon.dexmlbladeglue.jobs.TrendFeaturesJob\n                                                                 \n                                                                    val order_day = dimensions(\"order_day\")\n                                                                    val my_param = args(\"my_param\")\n                                                                    // input dataframes are available both as local variables and also within\n                                                                    // the spark context\n                                                                    // other variables are also available (spark, inputs, inputTable, execParams)\n                                                                    TrendFeatureWorker.main(args)\n                                                                   \"\"\"\n                                                                   , external_library_paths=[\"s3://\u003cYOUR_S3_FILE_PATH\u003e\"]\n                                                                   ),\n                                                      lang=Lang.SCALA,\n                                                      GlueVersion=\"2.0\",\n                                                      WorkerType=GlueWorkerType.G_1X.value,\n                                                      NumberOfWorkers=50,\n                                                      Timeout=3 * 60,  # 3 hours\n                                                      my_param=\"PARAM1\"\n                                                  )\n                                              ])\n\napp.activate(allow_concurrent_executions=False)\n\napp.process(d_ad_orders_na[\"2021-01-14\"])\n\napp.poll(repeat_d_ad_orders_na_scala[\"2021-01-14\"])\napp.poll(repeat_d_ad_orders_na_python[\"2021-01-14\"])\n\napp.pause()\n```\n\n* Example 4\n\nA simple workflow that has integrated alarming and metrics configuration using the framework provided node/route metrics:\n\nTIP: when the application is activated, go to AWS Console \u003e Cloudwatch -\u003e Dashboards and see the app specific dashboard.\n\n```python\n# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# SPDX-License-Identifier: MIT-0\n\nfrom datetime import datetime, timedelta\n\nimport time\n\nimport intelliflow.api_ext as flow\nfrom intelliflow.api_ext import *\nfrom intelliflow.core.platform.compute_targets.email import EMAIL\nfrom intelliflow.core.signal_processing.definitions.metric_alarm_defs import AlarmDimension\nfrom intelliflow.utils.test.inlined_compute import NOOPCompute\n\nflow.init_basic_logging()\n\n# automatically reads default credentials\n# default credentials should belong to an entity that can either:\n#  - do everything (admin, use it for quick prototyping!)\n#  - can create/delete roles (dev role of this app) that would have this app-name and IntelliFlow in it\n#  - can get/assume its dev-role (won't help if this is the first run)\napp = AWSApplication(\"alarm-demo\", \"us-east-1\")\n\nroute_id = \"metric_emitting_node\"\n\n#1- let's create a sample node with no inputs. we will execute this manually only.\ntest_node = app.create_data(id=route_id,\n                            compute_targets=[\n                                InlinedCompute(\n                                    # this will be executed in AWS\n                                    lambda x, y, z: int(\"FAIL!\")\n                                )\n                            ])\n\n# get orchestration metrics specific to this node\ntest_node_metrics = app.get_route_metrics(test_node)\n\ntest_node_routing_metrics_map = test_node_metrics[RoutingTable]\n#for alias, metric_signal in test_node_routing_metrics_map.items():\n#    print(f\"Metric signal alias for test_node: {alias}\")\n#    print(\"Actual/concrete metric stats:\")\n#    print(metric_signal.describe())\n\ntest_node_state_metric_signal = test_node_routing_metrics_map[\"routing_table.receive.hook\" + \".\" + route_id]\n\ntest_node_failure_alarm = app.create_alarm(id=\"metric_emitting_node_failure\",\n                                   target_metric_or_expression=test_node_state_metric_signal[RoutingHookInterface.Execution.IExecutionFailureHook.__name__][MetricStatistic.SUM][MetricPeriod.MINUTES(5)],\n                                   number_of_evaluation_periods=1,\n                                   number_of_datapoint_periods=1,\n                                   comparison_operator=AlarmComparisonOperator.GreaterThanOrEqualToThreshold,\n                                   # since we are using logical/conditional operator in metric math, the output series only contains 1s or 0s.\n                                   # so threshold will be 1 to detect.\n                                   threshold=1\n                                   )\n\ntest_node_failure_reactor = app.create_data(id=\"test_node_failure_reactor\",\n                                            inputs=[test_node_failure_alarm[\"ALARM\"]],\n                                            compute_targets=[\n                                                EMAIL(sender=\"if-test-list@amazon.com\",  # sender from the same account (see './doc/EMAIL_ACTION_SETUP_AND_INFO.md')\n                                                      recipient_list=[\"foo-ml-engineers@amazon.com\"]).action()\n                                            ],\n                                            # we are overwriting dimension spec here since the default dimension spec\n                                            # from input alarm would have 'time' dimension format in seconds \"%Y-%m-%d %H:%M:%S\"\n                                            # which would be very hard to track with poll API using datetime.now(), example:\n                                            #     path, records=app.poll(test_node_failure_reactor['ALARM'][datetime.now()])\n                                            #\n                                            # othwerwise this parameter can be easily ignored, see 'test_node_monitor_reactor' below.\n                                            output_dimension_spec={\n                                                AlarmDimension.STATE_TRANSITION.value: {\n                                                       \"type\": \"STRING\",\n                                                       AlarmDimension.TIME.value: {\n                                                          \"type\": \"DATETIME\",\n                                                          \"format\": \"%Y-%m-%d\" # this is the reason we are defining the spec here\n                                                       }\n                                                }\n                                            })\n\n\ntest_node_state_timing_signal = test_node_routing_metrics_map[\"routing_table.receive.hook.time_in_utc\" + \".\" + route_id]\n\n# detect if execution is longer than hour (60 * 60 secs)\ntest_node_exec_duration_alarm = app.create_alarm(id=\"metric_emitting_long_execution\",\n                                           target_metric_or_expression=\"(end1 - start) \u003e 3600 OR (end2 - start) \u003e 3600\",\n                                           metrics={\n                                               # both in utc (seconds) and over a window of 3 hours\n                                               \"start\": test_node_state_metric_signal[RoutingHookInterface.Execution.IExecutionBeginHook.__name__][MetricStatistic.SUM][MetricPeriod.MINUTES(3 * 60)],\n                                               \"end1\": test_node_state_metric_signal[RoutingHookInterface.Execution.IExecutionSuccessHook.__name__][ MetricStatistic.SUM][MetricPeriod.MINUTES(3 * 60)],\n                                               \"end2\": test_node_state_metric_signal[ RoutingHookInterface.Execution.IExecutionFailureHook.__name__][ MetricStatistic.SUM][MetricPeriod.MINUTES(3 * 60)]\n                                           },\n                                           number_of_evaluation_periods=1,\n                                           number_of_datapoint_periods=1,\n                                           comparison_operator=AlarmComparisonOperator.GreaterThanOrEqualToThreshold,\n                                           threshold=1,\n                                           default_actions=AlarmDefaultActionsMap(\n                                               ALARM_ACTIONS=set(),\n                                               OK_ACTIONS=set(),\n                                               INSUFFICIENT_DATA_ACTIONS=set()\n                                           )\n                                        )\n\ntest_node_monitor = app.create_composite_alarm(id=\"test_node_monitor\",\n                                               # use '|, \u0026, not, (, )' on RheocerOS alarm objects to build complex alarm_rules (refer AWS Cloudwatch - Composite Alarm Rules)\n                                               alarm_rule=test_node_failure_alarm | test_node_exec_duration_alarm)\n\ntest_node_monitor_reactor = app.create_data(id=\"test_node_monitor_reactor\",\n                                            # in plain english: \"when this monitor (composite alarm) goes to ALARM run my 'compute_targets'\"\n                                            inputs=[test_node_monitor[\"ALARM\"]],\n                                            compute_targets=[\n                                                NOOPCompute  # replace with Spark, PrestoSQL, Lambda code, Email,..\n                                            ])\n\napp.activate()\n\n\n# TIP: go to AWS console and observe the status of alarms (especially 'test_node_monitor') in auto-generated Cloudwatch Dashboard.\n\n# TEST STRATEGY 1: use natural application flow starting from auto-emitted orchestration metrics via manual node exec,\n# these metrics will trigger alarms, then those alarms will trigger downstream reactor nodes.\n#\n# now start the chain of events\n# - start a forced execution on test node\ntry:\n    app.execute(test_node)\n    assert False, \"test_node execution must have failed!\"\nexcept RuntimeError:\n    pass\n\n# it must have failed\npath, failure_compute_records = app.poll(test_node)\nassert not path and failure_compute_records\n\n# - then (in AWS) failure emits test_node_state_metric_signal with the following metric stat:\n#      {\n#           \"MetricName\": \"IExecutionFailureHook,\n#           \"Dimensions\": {\n#               \"RouteID\": \"metric_emitting_node\"\n#           }\n#      }\n#\n# - this metric triggers test_node_failure_alarm (AlarmName=\"metric_emitting_node_failure\")\n# - and this alarm triggers:\n#    1- test_node_monitor (AlarmName=\"test_node_monitor\")\n#    2- More importantly, the reactor node test_node_failure_reactor created to send out an EMAIL\n#\n# so let's poll the reactor node programmatically (it should also send out an email)\nwhile True:\n    path, records = app.poll(test_node_failure_reactor['ALARM'][datetime.now()])\n\n    if not path and not records:\n        # no execution detected yet [poll immediately returned (None, None)], let's poll again\n        time.sleep(5)\n    else:\n        assert path, \"trigger/execution on 'test_node_failure_reactor' must have succeeded\"\n        break\n\napp.pause()\n```\n\n* \"Hello World\" Example\n \nCode:\n[hello_world.py](https://github.com/amzn/rheoceros/blob/main/examples/hello_world.py \"Hello World Example\")\n\n![Hello World Example](./doc/resources/RheocerOS_HelloWorld_Example_1.drawio.png \"Hello World Example Flow\")\n\n* Scheduled \"Hello World\" Example\n\nModify the previous \"Hello World\" example and use a timer signal to turn it into a periodically executed workflow:\n\n![Hello World Scheduled](./doc/resources/RheocerOS_HelloWorld_Daily.drawio.png \"Hello World Example Flow - Scheduled\")\n\n\n### The Flow\n\nHaving just reviewed the examples above, we can now see the possible bindings (I/O) relationship between\nall the APIs exposed by an \u003cAWSApplication\u003e object.\n\n![API Flow](./doc/resources/RheocerOS_Development_API_Flow.drawio.png \"API Flow\")\n\nRefer;\n\n[AWSApplication](https://github.com/amzn/rheoceros/blob/main/src/intelliflow/api_ext.py#L700 \"AWSApplication class\")\n\n[Application](https://github.com/amzn/rheoceros/blob/main/src/intelliflow/core/application/application.py#L86 \"Application class\")\n\n### Activation / Runtime\n\nEach time you call AWSApplication::activate or AWSApplication::execute on a node of an updated topology, the activation\nsequence begins (e.g including but not limited to (1) cloud resource provisioning, (2) bindings, (3) deployment of your entire dependency bundle/workking to compute resources):\n\n* Application runtime and its cloud resources are isolated and updated each time an activation happens.\n   * Cloud resources adapt to your business logic and topology \n* Fully serverless/parallelized orchestration\n\nFor the curious:\n\n* Go to AWS Console and search for resources (DDB, Glue, Lambda, IAM, CW, Athena, etc) that contain your app ID in their names.\n\n## Current State and Vision\n\nRheocerOS aims to span the entire ML workflow life-cycle, extending the same programming model within the same application:\n\n* Phase 1: Core workflow generation / Feature engineering (datasets, basic AWS compute, big-data compute, alarming-metrics, etc)\n* Phase 2: Create and access other entities (e.g Model build on Sagemaker)\n* Phase 3: Inference (model inference, business rules as inference time nodes) \n\nCurrent state of the framework captures \"Core workflow generation\" only.\n\n## Example Areas of Contribution\n\n* Tooling or front-end development around the core framework\n  * It can easily power the backend of any new web/desktop application\n* Documentation\n* Core framework\n* New \u003cBatchCompute\u003e drivers\n* New \u003cComputeTarget\u003e implementations\n\nNote: Our documentation on GitHub is \"in progress\". Our priority is to provide the complete user guide and API documentation first and then move onto detailed contributor guide\non areas like the internal design of the core framework.\n\n## User Guide\n\nSee [USER_GUIDE.md](./doc/user_guide/USER_GUIDE.md)\n\n## Development\n\nSee [DEVELOPMENT.md](./doc/DEVELOPMENT.md)\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is licensed under the Apache-2.0 License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famzn%2Frheoceros","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famzn%2Frheoceros","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famzn%2Frheoceros/lists"}