{"id":22968426,"url":"https://github.com/irc-sphere/hyperstream","last_synced_at":"2025-08-13T09:34:31.763Z","repository":{"id":62569987,"uuid":"72214668","full_name":"IRC-SPHERE/HyperStream","owner":"IRC-SPHERE","description":"HyperStream","archived":false,"fork":false,"pushed_at":"2024-04-08T16:07:29.000Z","size":21071,"stargazers_count":13,"open_issues_count":14,"forks_count":5,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-04-29T19:08:48.939Z","etag":null,"topics":["compute-engine","hyperstream","sphere","stream-processing","streaming-data","workflow-engine"],"latest_commit_sha":null,"homepage":"https://irc-sphere.github.io/HyperStream/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IRC-SPHERE.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-28T14:31:05.000Z","updated_at":"2024-04-29T19:08:48.940Z","dependencies_parsed_at":"2022-11-03T17:15:39.286Z","dependency_job_id":null,"html_url":"https://github.com/IRC-SPHERE/HyperStream","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IRC-SPHERE%2FHyperStream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IRC-SPHERE%2FHyperStream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IRC-SPHERE%2FHyperStream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IRC-SPHERE%2FHyperStream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IRC-SPHERE","download_url":"https://codeload.github.com/IRC-SPHERE/HyperStream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229754920,"owners_count":18119138,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compute-engine","hyperstream","sphere","stream-processing","streaming-data","workflow-engine"],"created_at":"2024-12-14T21:19:15.734Z","updated_at":"2024-12-14T21:19:16.547Z","avatar_url":"https://github.com/IRC-SPHERE.png","language":"Python","readme":".. figure:: https://cdn.rawgit.com/IRC-SPHERE/HyperStream/dfbac332/hyperstream.svg\n   :alt: HyperStream logo\n\n   HyperStream logo\n\nHyperStream\n===========\n\n|DOI| |Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby|\n|Build Status| |Dependency Status| |Test Coverage| |Issue Count|\n|Documentation Status|\n\nHyperstream is a large-scale, flexible and robust software package for\nprocessing streaming data.\n\n-  HyperStream `homepage \u003chttps://irc-sphere.github.io/HyperStream/\u003e`__\n-  Tutorial\n   `notebooks \u003chttp://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/\u003e`__\n-  Gitter `chat room \u003chttps://gitter.im/IRC-SPHERE-HyperStream/Lobby\u003e`__\n-  Developer\n   `documentation \u003chttp://hyperstream.readthedocs.io/en/latest/\u003e`__\n\nHyperstream overcomes the limitations of other computational engines and\nprovides high-level interfaces to execute complex nesting, fusion, and\nprediction both in online and offline forms in streaming environments.\nAlthough developed specifically for SPHERE, Hyperstream is a general\npurpose tool that is well-suited for the design, development, and\ndeployment of algorithms and predictive models in a wide space of\nsequential predictive problems.\n\nThis software has been designed from the outset to be\ndomain-independent, in order to provide maximum value to the wider\ncommunity. Key aspects of the software include the capability to create\ncomplex interlinked workflows, and a computational engine that is\ndesigned to be \"compute-on-request\", meaning that no unnecessary\nresources are used.\n\nInstallation\n============\n\nDocker images\n-------------\n\nIf you do not want to install all the packages separately you can use\nour Docker bundle available\n`here \u003chttps://github.com/IRC-SPHERE/Hyperstream-Dockerfiles\u003e`__.\n\nLocal machine\n-------------\n\nInstall via pip\n\n::\n\n    pip install hyperstream\n    python -c 'from hyperstream import HyperStream'\n\nTo get the latest version\n\n::\n\n    pip install -U git+git://github.com/IRC-SPHERE/HyperStream.git#egg=hyperstream\n    pip install -r requirements.txt\n\nOr clone the repository\n\n.. code:: bash\n\n    git clone git@github.com:IRC-SPHERE/HyperStream.git\n    cd HyperStream\n    virtualenv venv\n    . venv/bin/activate\n    pip install -r requirements.txt\n    python -c 'from hyperstream import HyperStream'\n\nAdditionally, one of the requirements to run Hyperstream is a MongoDB\nserver. By default, Hyperstream tries to connect to the port 27017 on\nthe localhost.\n\nTo access via a cloud provider, you might try mLab, MongoDB Atlas, or \nsearch for a different one. If you are using mLab, here are a few notes:\n\n- Set up an account via the mLab web site instructions. When asked to \npick a server type (e.g. Amazon, Google, etc) you can just choose free \noption of 500MB. This is more than enough to get started.\n- mLab will ask you to create a database; any name is fine, but make \nsure you write down what it is.\n- After creating a database, note that you’ll need to create at least \none database user in order to access the database.\n- You can test your database connection using MongoDB’s built-in command \nline tools\n\nTo install MongoDB locally go to the `official\ndocumentation \u003chttps://docs.mongodb.com/manual/installation/\u003e`__. E.g.\nin a Debian OS it is possible to install with the following command\n\n.. code:: bash\n\n    sudo apt-get install mongodb\n\nOnce the MongoDB server is installed, it can be started with the\nfollowing command\n\n.. code:: bash\n\n    service mongod start\n\nRunning tests\n=============\n\nRun the following command\n\n::\n\n    nosetests\n\nNote that for the MQTT logging test to succeed, you will need to have an\nMQTT broker running (e.g. Mosquitto). For example:\n\n::\n\n    docker run -ti -p 1883:1883 -p 9001:9001 toke/mosquitto\n\nor on OSX you will need pidof and mosquitto:\n\n::\n\n    brew install pidof\n    brew install mosquitto\n    brew services start mosquitto\n\nTutorials\n=========\n\nThe following tutorials show how to use HyperStream in a step-by-step\nguide.\n\n-  `Tutorial 1:\n   Introduction \u003chttps://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_01.ipynb\u003e`__\n-  `Tutorial 2: Creating\n   tools \u003chttps://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_02.ipynb\u003e`__\n-  `Tutorial 3: Stream\n   composition \u003chttps://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_03.ipynb\u003e`__\n-  `Tutorial 4: Real-time\n   streams \u003chttps://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_04.ipynb\u003e`__\n-  `Tutorial 5:\n   Workflows \u003chttps://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_05.ipynb\u003e`__\n\n.. code:: python\n\n        from hyperstream import HyperStream, StreamId, TimeInterval\n        from hyperstream.utils import utcnow, UTC\n        from datetime import timedelta\n\n        hs = HyperStream(loglevel=20)\n        M = hs.channel_manager.memory\n        T = hs.channel_manager.tools\n        clock = StreamId(name=\"clock\")\n        clock_tool = T[clock].window().last().value()\n        ticker = M.get_or_create_stream(stream_id=StreamId(name=\"ticker\"))\n        now = utcnow()\n        before = (now - timedelta(seconds=30)).replace(tzinfo=UTC)\n        ti = TimeInterval(before, now)\n        clock_tool.execute(sources=[], sink=ticker, interval=ti, alignment_stream=None)\n        print(list(ticker.window().tail(5)))\n\n        [StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=\u003cUTC\u003e), value=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=\u003cUTC\u003e)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=\u003cUTC\u003e), value=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=\u003cUTC\u003e)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=\u003cUTC\u003e), value=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=\u003cUTC\u003e)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=\u003cUTC\u003e), value=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=\u003cUTC\u003e)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=\u003cUTC\u003e), value=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=\u003cUTC\u003e))]\n\nHyperStream Viewer\n==================\n\nThe `HyperStream\nViewer \u003chttps://github.com/IRC-SPHERE/HyperStreamViewer\u003e`__ is a\npython/Flask web-app for interacting with HyperStream. In order to keep\nHyperStream to a minimum, this web-app is released as a separate\nrepository that takes the core as a dependency.\n\nLicense\n=======\n\nThis code is released under the `MIT\nlicense \u003chttps://github.com/IRC-SPHERE/HyperStream/blob/master/LICENSE\u003e`__.\n\nAcknowledgements\n================\n\nThis work has been funded by the UK Engineering and Physical Sciences\nResearch Council (EPSRC) under Grant\n`EP/K031910/1 \u003chttp://gow.epsrc.ac.uk/NGBOViewGrant.aspx?GrantRef=EP/K031910/1\u003e`__\n- \"SPHERE Interdisciplinary Research Collaboration\".\n\n.. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.242227.svg\n   :target: https://doi.org/10.5281/zenodo.242227\n.. |Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby| image:: https://badges.gitter.im/IRC-SPHERE-HyperStream/Lobby.svg\n   :target: https://gitter.im/IRC-SPHERE-HyperStream/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge\n.. |Build Status| image:: https://travis-ci.org/IRC-SPHERE/HyperStream.svg?branch=master\n   :target: https://travis-ci.org/IRC-SPHERE/HyperStream\n.. |Dependency Status| image:: https://www.versioneye.com/user/projects/58e423cb26a5bb005220301e/badge.svg?style=flat-square\n   :target: https://www.versioneye.com/user/projects/58e423cb26a5bb005220301e\n.. |Test Coverage| image:: https://codeclimate.com/github/IRC-SPHERE/HyperStream/badges/coverage.svg\n   :target: https://codeclimate.com/github/IRC-SPHERE/HyperStream/coverage\n.. |Issue Count| image:: https://codeclimate.com/github/IRC-SPHERE/HyperStream/badges/issue_count.svg\n   :target: https://codeclimate.com/github/IRC-SPHERE/HyperStream\n.. |Documentation Status| image:: https://readthedocs.org/projects/hyperstream/badge/?version=latest\n   :target: http://hyperstream.readthedocs.io/en/latest/?badge=latest\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firc-sphere%2Fhyperstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Firc-sphere%2Fhyperstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Firc-sphere%2Fhyperstream/lists"}