{"id":13936749,"url":"https://github.com/plecto/motorway","last_synced_at":"2025-07-19T22:31:45.464Z","repository":{"id":1169234,"uuid":"26050270","full_name":"plecto/motorway","owner":"plecto","description":"Cloud ready pure-python streaming data pipeline library","archived":false,"fork":false,"pushed_at":"2023-08-16T09:47:27.000Z","size":394,"stargazers_count":154,"open_issues_count":4,"forks_count":19,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-08-08T23:24:25.295Z","etag":null,"topics":["motorway","pipeline","python","streaming"],"latest_commit_sha":null,"homepage":"http://motorway.readthedocs.org/en/latest/index.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/plecto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-11-01T12:40:28.000Z","updated_at":"2024-02-07T13:30:01.000Z","dependencies_parsed_at":"2024-05-02T22:56:45.455Z","dependency_job_id":null,"html_url":"https://github.com/plecto/motorway","commit_stats":{"total_commits":308,"total_committers":10,"mean_commits":30.8,"dds":0.5454545454545454,"last_synced_commit":"a911fa996d4f47c1b4aef7c82a03b05e436c9c21"},"previous_names":[],"tags_count":77,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plecto%2Fmotorway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plecto%2Fmotorway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plecto%2Fmotorway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plecto%2Fmotorway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/plecto","download_url":"https://codeload.github.com/plecto/motorway/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226686730,"owners_count":17666928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["motorway","pipeline","python","streaming"],"created_at":"2024-08-07T23:02:57.645Z","updated_at":"2024-11-27T04:31:24.822Z","avatar_url":"https://github.com/plecto.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"motorway\n========\n\nRequires python 3.7\n\nTests: [![Circle CI](https://circleci.com/gh/plecto/motorway.svg?style=svg)](https://circleci.com/gh/plecto/motorway)\n\nMotorway is a real-time data pipeline, much like Apache Storm - but made in Python :-) We use it over at Plecto and we're really happy with it - but we're continously developing it. The reason why we started this project was that we wanted something similar to Storm, but without Zookeeper and the need to take the pipeline down to update the topology.\n\n# Epic web interface\n\n![Screenshot](https://www.dropbox.com/s/v614jtz0u1h9hrs/Screenshot%202016-07-29%2014.28.26.png?dl=1)\n\n# Amazing Selling points!\n\n- No need to \"upload\" topologies (in particular, no need to stop the old topology before launching the new one)\n- Possibility to work tigthly with our python codebase\n- \"Cloud compatible\" - should be able to run in AWS Auto Scaling Groups. No manual setup required for scaling and no external requirements such as Zookeeper that also do not run very nice in the Auto Scaling Groups.\n\n# Extraordinary algorithm\n\nMotorway re-implemented the same [algorithm to store message state](https://storm.incubator.apache.org/documentation/Acking-framework-implementation.html) as Apache Storm, which is brilliant. \n\nUnlike with Storm where you submit a topology to an existing cluster, with Motorway you simply add a new node with the new code and take down the other afterwards. If you want to be able to use Motorway in a HA environment (and you probably want to), you should consider running a dedicated \"master node\" which only handles discovery - in that way nodes can come and go as needed.\n\n**New:** Now with pypy support for double speed!\n\n# Use with Django\n\nCan easily be integrated with django, if you define the pipeline (as seen below) in a management command. However, large pipelines might result in a high number of connections to your DB.\n\n\nWord Count Example\n==================\n\n```python\nclass WordRamp(Ramp):\n    sentences = [\n        \"Oak is strong and also gives shade.\",\n        \"Cats and dogs each hate the other.\",\n        \"The pipe began to rust while new.\",\n        \"Open the crate but don't break the glass.\",\n        \"Add the sum to the product of these three.\",\n        \"Thieves who rob friends deserve jail.\",\n        \"The ripe taste of cheese improves with age.\",\n        \"Act on these orders with great speed.\",\n        \"The hog crawled under the high fence.\",\n        \"Move the vat over the hot fire.\",\n    ]\n\n    def next(self):\n        yield Message(uuid.uuid4().int, self.sentences[random.randint(0, len(self.sentences) -1)])\n        \nclass SentenceSplitIntersection(Intersection):\n    def process(self, message):\n        for word in message.content.split(\" \"):\n            yield Message.new(message, word, grouping_value=word)\n        self.ack(message)\n\n\nclass WordCountIntersection(Intersection):\n    def __init__(self):\n        self._count = defaultdict(int)\n        super(WordCountIntersection, self).__init__()\n\n    @batch_process(wait=2, limit=500)\n    def process(self, messages):\n        for message in messages:\n            self._count[message.content] += 1\n            self.ack(message)\n        print self._count\n\nclass WordCountPipeline(Pipeline):\n    def definition(self):\n        self.add_ramp(WordRamp, 'sentence')\n        self.add_intersection(SentenceSplitIntersection, 'sentence', 'word')\n        self.add_intersection(WordCountIntersection, 'word')\n\n\nWordCountPipeline().run()\n```\n\nIntegrations\n============\n\nCurrent list of integrations:\n\n- Salesforce (consumer, batch + real-time)\n- Recurly (consumer)\n- Amazon SQS (consumer + producer)\n- Amazon Kinesis (consumer + producer)\n- SQL Servers (via SQLAlchemy)\n\nLook in motorway/contrib/ for these addons and feel free to contribute additional ones.\n\nInsights? No problem!\n============\nMotorway can be instrumented using New Relics python agent. Just run it using newrelic-admin and motorway \nwill start sending metrics. You can find them in New Relic as non-web transactions.\n\n\nLicense\n=======\n   Copyright 2014-2021 Plecto ApS\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplecto%2Fmotorway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fplecto%2Fmotorway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplecto%2Fmotorway/lists"}