https://github.com/irc-sphere/hyperstream
HyperStream
https://github.com/irc-sphere/hyperstream
compute-engine hyperstream sphere stream-processing streaming-data workflow-engine
Last synced: 9 months ago
JSON representation
HyperStream
- Host: GitHub
- URL: https://github.com/irc-sphere/hyperstream
- Owner: IRC-SPHERE
- License: mit
- Created: 2016-10-28T14:31:05.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2024-04-08T16:07:29.000Z (about 2 years ago)
- Last Synced: 2024-04-29T19:08:48.939Z (almost 2 years ago)
- Topics: compute-engine, hyperstream, sphere, stream-processing, streaming-data, workflow-engine
- Language: Python
- Homepage: https://irc-sphere.github.io/HyperStream/
- Size: 20.1 MB
- Stars: 13
- Watchers: 10
- Forks: 5
- Open Issues: 14
-
Metadata Files:
- Readme: README
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
.. figure:: https://cdn.rawgit.com/IRC-SPHERE/HyperStream/dfbac332/hyperstream.svg
:alt: HyperStream logo
HyperStream logo
HyperStream
===========
|DOI| |Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby|
|Build Status| |Dependency Status| |Test Coverage| |Issue Count|
|Documentation Status|
Hyperstream is a large-scale, flexible and robust software package for
processing streaming data.
- HyperStream `homepage `__
- Tutorial
`notebooks `__
- Gitter `chat room `__
- Developer
`documentation `__
Hyperstream overcomes the limitations of other computational engines and
provides high-level interfaces to execute complex nesting, fusion, and
prediction both in online and offline forms in streaming environments.
Although developed specifically for SPHERE, Hyperstream is a general
purpose tool that is well-suited for the design, development, and
deployment of algorithms and predictive models in a wide space of
sequential predictive problems.
This software has been designed from the outset to be
domain-independent, in order to provide maximum value to the wider
community. Key aspects of the software include the capability to create
complex interlinked workflows, and a computational engine that is
designed to be "compute-on-request", meaning that no unnecessary
resources are used.
Installation
============
Docker images
-------------
If you do not want to install all the packages separately you can use
our Docker bundle available
`here `__.
Local machine
-------------
Install via pip
::
pip install hyperstream
python -c 'from hyperstream import HyperStream'
To get the latest version
::
pip install -U git+git://github.com/IRC-SPHERE/HyperStream.git#egg=hyperstream
pip install -r requirements.txt
Or clone the repository
.. code:: bash
git clone git@github.com:IRC-SPHERE/HyperStream.git
cd HyperStream
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
python -c 'from hyperstream import HyperStream'
Additionally, one of the requirements to run Hyperstream is a MongoDB
server. By default, Hyperstream tries to connect to the port 27017 on
the localhost.
To access via a cloud provider, you might try mLab, MongoDB Atlas, or
search for a different one. If you are using mLab, here are a few notes:
- Set up an account via the mLab web site instructions. When asked to
pick a server type (e.g. Amazon, Google, etc) you can just choose free
option of 500MB. This is more than enough to get started.
- mLab will ask you to create a database; any name is fine, but make
sure you write down what it is.
- After creating a database, note that you’ll need to create at least
one database user in order to access the database.
- You can test your database connection using MongoDB’s built-in command
line tools
To install MongoDB locally go to the `official
documentation `__. E.g.
in a Debian OS it is possible to install with the following command
.. code:: bash
sudo apt-get install mongodb
Once the MongoDB server is installed, it can be started with the
following command
.. code:: bash
service mongod start
Running tests
=============
Run the following command
::
nosetests
Note that for the MQTT logging test to succeed, you will need to have an
MQTT broker running (e.g. Mosquitto). For example:
::
docker run -ti -p 1883:1883 -p 9001:9001 toke/mosquitto
or on OSX you will need pidof and mosquitto:
::
brew install pidof
brew install mosquitto
brew services start mosquitto
Tutorials
=========
The following tutorials show how to use HyperStream in a step-by-step
guide.
- `Tutorial 1:
Introduction `__
- `Tutorial 2: Creating
tools `__
- `Tutorial 3: Stream
composition `__
- `Tutorial 4: Real-time
streams `__
- `Tutorial 5:
Workflows `__
.. code:: python
from hyperstream import HyperStream, StreamId, TimeInterval
from hyperstream.utils import utcnow, UTC
from datetime import timedelta
hs = HyperStream(loglevel=20)
M = hs.channel_manager.memory
T = hs.channel_manager.tools
clock = StreamId(name="clock")
clock_tool = T[clock].window().last().value()
ticker = M.get_or_create_stream(stream_id=StreamId(name="ticker"))
now = utcnow()
before = (now - timedelta(seconds=30)).replace(tzinfo=UTC)
ti = TimeInterval(before, now)
clock_tool.execute(sources=[], sink=ticker, interval=ti, alignment_stream=None)
print(list(ticker.window().tail(5)))
[StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=), value=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=), value=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=), value=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=), value=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=), value=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=))]
HyperStream Viewer
==================
The `HyperStream
Viewer `__ is a
python/Flask web-app for interacting with HyperStream. In order to keep
HyperStream to a minimum, this web-app is released as a separate
repository that takes the core as a dependency.
License
=======
This code is released under the `MIT
license `__.
Acknowledgements
================
This work has been funded by the UK Engineering and Physical Sciences
Research Council (EPSRC) under Grant
`EP/K031910/1 `__
- "SPHERE Interdisciplinary Research Collaboration".
.. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.242227.svg
:target: https://doi.org/10.5281/zenodo.242227
.. |Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby| image:: https://badges.gitter.im/IRC-SPHERE-HyperStream/Lobby.svg
:target: https://gitter.im/IRC-SPHERE-HyperStream/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
.. |Build Status| image:: https://travis-ci.org/IRC-SPHERE/HyperStream.svg?branch=master
:target: https://travis-ci.org/IRC-SPHERE/HyperStream
.. |Dependency Status| image:: https://www.versioneye.com/user/projects/58e423cb26a5bb005220301e/badge.svg?style=flat-square
:target: https://www.versioneye.com/user/projects/58e423cb26a5bb005220301e
.. |Test Coverage| image:: https://codeclimate.com/github/IRC-SPHERE/HyperStream/badges/coverage.svg
:target: https://codeclimate.com/github/IRC-SPHERE/HyperStream/coverage
.. |Issue Count| image:: https://codeclimate.com/github/IRC-SPHERE/HyperStream/badges/issue_count.svg
:target: https://codeclimate.com/github/IRC-SPHERE/HyperStream
.. |Documentation Status| image:: https://readthedocs.org/projects/hyperstream/badge/?version=latest
:target: http://hyperstream.readthedocs.io/en/latest/?badge=latest