{"id":17848464,"url":"https://github.com/betodealmeida/senor-octopus","last_synced_at":"2025-03-20T07:31:53.556Z","repository":{"id":49098438,"uuid":"350548313","full_name":"betodealmeida/senor-octopus","owner":"betodealmeida","description":"An application to move data around","archived":false,"fork":false,"pushed_at":"2023-04-24T03:01:28.000Z","size":228,"stargazers_count":15,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-28T16:00:29.424Z","etag":null,"topics":["airbyte","home-automation","ifttt","mqtt","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/betodealmeida.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.rst","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-23T01:56:30.000Z","updated_at":"2024-01-30T14:38:34.000Z","dependencies_parsed_at":"2024-10-27T23:10:48.202Z","dependency_job_id":"7104873f-2e60-4de4-a47c-db051f3e265e","html_url":"https://github.com/betodealmeida/senor-octopus","commit_stats":{"total_commits":107,"total_committers":1,"mean_commits":107.0,"dds":0.0,"last_synced_commit":"362d4cb8d5ac7fea620c2b4d46e807bb614a59bd"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betodealmeida%2Fsenor-octopus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betodealmeida%2Fsenor-octopus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betodealmeida%2Fsenor-octopus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betodealmeida%2Fsenor-octopus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/betodealmeida","download_url":"https://codeload.github.com/betodealmeida/senor-octopus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244066220,"owners_count":20392405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airbyte","home-automation","ifttt","mqtt","python"],"created_at":"2024-10-27T22:22:23.961Z","updated_at":"2025-03-20T07:31:48.548Z","avatar_url":"https://github.com/betodealmeida.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"=============\nsenor-octopus\n=============\n\n.. image:: https://coveralls.io/repos/github/betodealmeida/senor-octopus/badge.svg?branch=main\n   :target: https://coveralls.io/github/betodealmeida/senor-octopus?branch=main\n.. image:: https://img.shields.io/cirrus/github/betodealmeida/senor-octopus\n   :target: https://cirrus-ci.com/github/betodealmeida/senor-octopus\n   :alt: Cirrus CI - Base Branch Build Status\n.. image:: https://badge.fury.io/py/senor-octopus.svg\n   :target: https://badge.fury.io/py/senor-octopus\n.. image:: https://img.shields.io/pypi/pyversions/senor-octopus\n   :alt: PyPI - Python Version\n\nThey say there are only 2 kinds of work: you either move information from one place to another, or you move mass from one place to another.\n\n**Señor Octopus is an application that moves data around**. It reads a YAML configuration file that describes how to connect **nodes**. For example, you might want to measure your internet speed every hour and store it in a database:\n\n.. code-block:: yaml\n\n    speedtest:\n      plugin: source.speedtest\n      flow: -\u003e db\n      schedule: @hourly\n\n    db:\n      plugin: sink.db.postgresql\n      flow: speedtest -\u003e\n      user: alice\n      password: XXX\n      host: localhost\n      port: 5432\n      dbname: default\n\nNodes are connected by the ``flow`` attribute. The ``speedtest`` node is connected to the ``db`` node because it points to it:\n\n.. code-block:: yaml\n\n    speedtest:\n      flow: -\u003e db\n\nThe ``db`` node, on the other hand, listens to events from the ``speedtest`` node:\n\n.. code-block:: yaml\n\n    db:\n      flow: speedtest -\u003e\n\nWe can also use ``*`` as a wildcard, if we want a node to connect to all other nodes, or specify a list of nodes:\n\n.. code-block:: yaml\n\n    speedtest:\n      flow: -\u003e db, log\n\n    db:\n      flow: \"* -\u003e\"\n\nNote that in YAML we need to quote attributes that start with an asterisk.\n\nRunning Señor Octopus\n=====================\n\nYou can save the configuration above to a file called ``speedtest.yaml`` and run:\n\n.. code-block:: bash\n\n    $ pip install senor-octopus\n    $ srocto speedtest.yaml\n\nEvery hour the ``speedtest`` **source** node will run, and the results will be sent to the ``db`` **sink** node, which writes them to a Postgres database.\n\nHow to these results look like?\n\nEvents\n======\n\nSeñor Octopus uses a very simple but flexible data model to move data around. We have nodes called **sources** that create a stream of events, each one like this:\n\n.. code-block:: python\n\n    class Event(TypedDict):\n        timestamp: datetime\n        name: str\n        value: Any\n\nAn event has a **timestamp** associated with it, a **name**, and a **value**. Note that the value can be anything!\n\nA **source** will produce a stream of events. In the example above, once per hour the ``speedtest`` source will produce events like these:\n\n.. code-block:: python\n\n    [\n        {\n            'timestamp': datetime.datetime(2021, 5, 11, 22, 16, 26, 812083, tzinfo=datetime.timezone.utc),\n            'name': 'hub.speedtest.download',\n            'value': 16568200.018792046,\n        },\n        {\n            'timestamp': datetime.datetime(2021, 5, 11, 22, 16, 26, 812966, tzinfo=datetime.timezone.utc),\n            'name': 'hub.speedtest.upload',\n            'value': 5449607.159468643,\n        },\n        {\n            'timestamp': datetime.datetime(2021, 5, 11, 22, 16, 26, 820369, tzinfo=datetime.timezone.utc),\n            'name': 'hub.speedtest.client',\n            'value': {\n                'ip': '173.211.12.32',\n                'lat': '37.751',\n                'lon': '-97.822',\n                'isp': 'Colocation America Corporation',\n                'isprating': '3.7',\n                'rating': '0',\n                'ispdlavg': '0',\n                'ispulavg': '0',\n                'loggedin': '0',\n                'country': 'US',\n            }\n        },\n        ...\n    ]\n\nThe events are sent to **sinks**, which consume the stream. In this example, the ``db`` sink will receive the events and store them in a Postgres database.\n\nEvent-driven sources\n====================\n\nIn the previous example we configured the ``speedtest`` source to run hourly. Not all sources need to be scheduled, though. We can have a source that listens to a given topic in `MQTT \u003chttps://mqtt.org/\u003e`_, eg:\n\n.. code-block:: yaml\n\n    mqtt:\n      plugin: source.mqtt\n      flow: -\u003e db\n      topics:\n        - \"srocto/feeds/#\"\n      host: localhost\n      port: 1883\n      username: bob\n      password: XXX\n      message_is_json: true\n\nThe source above will immediately send an event to the ``db`` node every time a new message shows up in the topic wildcard ``srocto/feeds/#``, so it can be written to the database — a super easy way of persisting a message queue to disk!\n\nBatching events\n===============\n\nThe example above is not super efficient, since it writes to the database every time an event arrives. Instead, we can easily **batch** the events so that they're accumulated in a queue and processed every, say, 5 minutes:\n\n.. code-block:: yaml\n\n    db:\n      plugin: sink.db.postgresl\n      flow: speedtest, mqtt -\u003e\n      batch: 5 minutes\n      user: alice\n      password: XXX\n      host: localhost\n      port: 5432\n      dbname: default\n\nWith the ``batch`` parameter any incoming events are stored in a queue for the configured time, and processed by the sink together. Any pending events in the queue will still be processed if ``srocto`` terminates gracefully (eg, with ``ctrl+C``).\n\nFiltering events\n================\n\nMuch of the flexibility of Señor Octopus comes from a third type of node, the **filter**. Filters can be used to not only filter data, but also format it. For example, let's say we want to turn on some lights at sunset. The ``sun`` source will send events with a value of \"sunset\" or \"sunrise\" every time one occurs:\n\n.. code-block:: python\n\n    {\n        'timestamp': ...,\n        'name': 'hub.sun',\n        'value': 'sunset',\n    }\n\nThe ``tuya`` sink can be used to control a smart switch, but in order to turn it on it expects an event that looks like this:\n\n.. code-block:: python\n\n    {\n        'timestamp': ...,\n        'name': ...,\n        'value': 'on',\n    }\n\nWe can use the ``jinja`` filter to ignore \"sunrise\" events, and to convert the \"sunset\" value into \"on\":\n\n\n.. code-block:: yaml\n\n    sun:\n      plugin: source.sun\n      flow: -\u003e sunset\n      latitude: 38.3\n      longitude: -123.0\n\n    sunset:\n      plugin: filter.jinja\n      flow: sun -\u003e lights\n      template: \u003e\n        {% if event['value'] == 'sunset' %}\n          on\n        {% endif %}\n\n    lights:\n      plugin: sink.tuya\n      flow: sunset -\u003e\n      device: \"Porch lights\"\n      email: charlie@example.com\n      password: XXX\n      country: \"1\"\n      application: smart_life\n\nWith this configuration the ``sunset`` filter will drop any events that don't have a value of \"sunset\". And for those events that have, the value will be replaced by the string \"on\" so it can activate the lights in the ``lights`` node.\n\nThrottling events\n=================\n\nSometimes we want to limit the number of events being consumed by a sink. For example, imagine that we want to use Señor Octopus to monitor air quality using an `Awair Element \u003chttps://www.getawair.com/home/element\u003e`_, sending us an SMS when the score is below a given threshold. We would like the SMS to be sent at most once every 30 minutes, and only between 8am and 10pm.\n\nHere's how we can do that:\n\n.. code-block:: yaml\n\n    awair:\n      plugin: source.awair\n      flow: -\u003e bad_air\n      schedule: 0/10 * * * *\n      access_token: XXX\n      device_type: awair-element\n      device_id: 12345\n\n    bad_air:\n      plugin: filter.jinja\n      flow: awair -\u003e sms\n      template: \u003e\n        {% if\n           event['timestamp'].astimezone().hour \u003e= 8 and\n           event['timestamp'].astimezone().hour \u003c= 21 and\n           event['name'] == 'hub.awair.score' and\n           event['value'] \u003c 80\n        %}\n          Air quality score is low: {{ event['value'] }}\n        {% endif %}\n\n    sms:\n      plugin: sink.sms\n      flow: bad_air -\u003e\n      throttle: 30 minutes\n      account_sid: XXX\n      auth_token: XXX\n      from: \"+18002738255\"\n      to: \"+15558675309\"\n\nIn the example above, the ``awair`` source will fetch air quality data every 10 minutes, and send it to ``bad_air``. The filter checks for the hour, to prevent sending an SMS from 10pm to 8am, and checks the air quality score — if it's lower than 80 it will reformat the value of the event to a nice message, eg:\n\n    \"Air quality score is low: 70\"\n\nThis is then sent to the ``sms`` sink, which has a ``throttle`` of 30 minutes. The throttle configuration will prevent the sink from running more than once every 30 minutes, to avoid spamming us with messages in case the score remains low.\n\nPlugins\n=======\n\nSeñor Octopus supports an increasing list of plugins, and it's straightforward to add new ones. Each plugin is simply a function that produces, processes, or consumes a stream.\n\nHere's the ``random`` source, which produces random numbers:\n\n.. code-block:: python\n\n    async def rand(events: int = 10, prefix: str = \"hub.random\") -\u003e Stream:\n        for _ in range(events):\n            yield {\n                \"timestamp\": datetime.now(timezone.utc),\n                \"name\": prefix,\n                \"value\": random.random(),\n            }\n\nThis is the full source code for the ``jinja`` filter:\n\n.. code-block:: python\n\n    async def jinja(stream: Stream, template: str) -\u003e Stream:\n        _logger.debug(\"Applying template to events\")\n        tmpl = Template(template)\n        async for event in stream:\n            value = tmpl.render(event=event)\n            if value:\n                yield {\n                    \"timestamp\": event[\"timestamp\"],\n                    \"name\": event[\"name\"],\n                    \"value\": value,\n                }\n\nAnd this is the ``sms`` sink:\n\n.. code-block:: python\n\n    async def sms(\n        stream: Stream, account_sid: str, auth_token: str, to: str, **kwargs: str\n    ) -\u003e None:\n        from_ = kwargs[\"from\"]\n        client = Client(account_sid, auth_token)\n        async for event in stream:\n            _logger.debug(event)\n            _logger.info(\"Sending SMS\")\n            client.messages.create(body=str(event[\"value\"]).strip(), from_=from_, to=to)\n\nAs you can see, a source is an async generator that yields events. A filter receives the stream with additional configuration parameters, and also returns a stream. And a sink receives a stream with additional parameters, and returns nothing.\n\nSources\n~~~~~~~\n\nThe current plugins for sources are:\n\n- `source.awair \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/awair.py\u003e`_: Fetch air quality data from Awair Element monitor.\n- `source.crypto \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/crypto.py\u003e`_: Fetch price of cryptocurrencies from cryptocompare.com.\n- `source.mqtt \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/mqtt.py\u003e`_: Subscribe to messages on one or more MQTT topics.\n- `source.rand \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/rand.py\u003e`_: Generate random numbers between 0 and 1.\n- `source.speed \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/speed.py\u003e`_: Measure internet speed.\n- `source.sqla \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/sqla.py\u003e`_: Read data from database.\n- `source.static \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/static.py\u003e`_: Generate static events.\n- `source.stock \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/stock.py\u003e`_: Fetch stock price form Yahoo! Finance.\n- `source.sun \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/sun.py\u003e`_: Send events on sunrise and sunset.\n- `source.udp \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/udp/main.py\u003e`_: Listens to UDP messages on a given port.\n- `source.weatherapi \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/weatherapi.py\u003e`_: Fetch weather forecast data from weatherapi.com.\n- `source.whistle \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sources/whistle.py\u003e`_: Fetch device information and location for a Whistle pet tracker.\n\nFilters\n~~~~~~~\n\nThe existing filters are very similar, the main difference being how you configure them:\n\n- `filter.combine \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/filters/combine.py\u003e`_: Aggregate multiple events into a single one.\n- `filter.format \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/filters/format.py\u003e`_: Format an event stream based using Python string formatting.\n- `filter.jinja \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/filters/jinja.py\u003e`_: Apply a Jinja2 template to events.\n- `filter.jsonpath \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/filters/jpath.py\u003e`_: Filter event stream based on a JSON path.\n- `filter.serialize \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/filters/serialize.py\u003e`_: Serialize payload to JSON or YAML.\n- `filter.deserialize \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/filters/deserialize.py\u003e`_: Deserialize payload from JSON or YAML.\n\nSinks\n~~~~~\n\nThese are the current sinks:\n\n- `sink.log \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sinks/log.py\u003e`_: Send events to a logger.\n- `sink.mqtt \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sinks/mqtt.py\u003e`_: Send events as messages to an MQTT topic.\n- `sink.pushover \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sinks/pushover.py\u003e`_: Send events to the Pushover mobile app.\n- `sink.slack \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sinks/slack.py\u003e`_: Send messages to a Slack channel.\n- `sink.sms \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sinks/sms.py\u003e`_: Send SMS via Twilio.\n- `sink.tuya \u003chttps://github.com/betodealmeida/senor-octopus/blob/main/src/senor_octopus/sinks/tuya.py\u003e`_: Send commands to a Tuya/Smart Life device.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbetodealmeida%2Fsenor-octopus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbetodealmeida%2Fsenor-octopus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbetodealmeida%2Fsenor-octopus/lists"}