https://github.com/noflo/noflo-runtime-msgflo

NoFlo runtime for AMQP/MQTT
https://github.com/noflo/noflo-runtime-msgflo

fbp-runtime msgflo noflo

Last synced: 3 months ago
JSON representation

NoFlo runtime for AMQP/MQTT

• Host: GitHub
• URL: https://github.com/noflo/noflo-runtime-msgflo
• Owner: noflo
• Created: 2015-01-31T05:00:37.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2020-05-23T12:03:19.000Z (about 6 years ago)
• Last Synced: 2025-09-26T03:03:21.418Z (8 months ago)
• Topics: fbp-runtime, msgflo, noflo
• Language: CoffeeScript
• Size: 217 KB
• Stars: 10
• Watchers: 7
• Forks: 8
• Open Issues: 20
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGES.md

Awesome Lists containing this project

README

          

noflo-runtime-msgflo

====================

[NoFlo](https://noflojs.org) runtime, designed for use with [msgflo](https://github.com/the-grid/msgflo).

Loads a NoFlo graph and registers it as a msgflo participant.

The exported ports of the NoFlo network then becomes accessible through message queues

([AMQP](http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol) or [MQTT](http://en.wikipedia.org/wiki/MQTT)),

and msgflo can coordinate how multiple such networks send data between eachother.

Will eventually allow a single entry-point for

[FBP runtime protocol](http://noflojs.org/documentation/protocol) clients (like [Flowhub](http://flowhub.io))

to access NoFlo FBP networks that span multiple nodes.

Usage

------

Using the commandline tool

    noflo-runtime-msgflo --name myworker --graph project/WorkerGraph --broker amqp://foo.cloudamqp.com/bar

Altenatively one can use the embedding API, see [src/mount.coffee](./src/mount.coffee)

    runtime = require 'noflo-runtime-msgflo'

    options =

      name: 'gssmeasure'

      graph: 'the-grid-api/PrecomputeGss',

      basedir: __dirname

      broker: config.amqp.url,

    rt = new runtime.mount.Mounter options

    rt.start (err, rt) ->

      # started

## Debugging

noflo-runtime-msgflo supports [flowtrace](https://github.com/flowbased/flowtrace) allows to trace & store the execution of the NoFlo network,

so you can debug any issues that would occur.

You can enable tracing using `--trace` commandline argument,

or via the `trace:start` [FBP runtime protocol](http://noflojs.org/documentation/protocol/#trace) message.

    noflo-runtime-msgflo --graph project/MyMainGraph --trace

To trigger dumping a flowtrace locally, send the `SIGUSR2` Unix signal

    kill -SIGUSR2 $PID_OF_PROCESS

    ... Wrote flowtrace to: /tmp/1151020-12063-ami5vq.json

Or, to trigger a flowtrace remotely, send a `trace:dump` FBP protocol message to the queue of participant.

    msgflo-send-message --queue .fbp.$participantId.receive --json '{ "protocol":"trace", "command":"dump", "payload":{ "graph":"default", "type":"flowtrace.json"} }'

Then, assuming `.fbp.$participantId.send` has been bound to queue `my-flowtraces`, one could download it

    msgflo-dump-message --queue my-flowtraces --amount 1

You can now use various flowtrace tools to introspect the data.

For instance, you can get a human readable log using `flowtrace-show`

    flowtrace-show /tmp/1151020-12063-ami5vq.json

    -> IN repeat CONN

    -> IN repeat DATA hello world

    -> IN stdout CONN

    -> IN stdout DATA hello world

    -> IN repeat DISC

    -> IN stdout DISC

Optimizing startup time

----

Since NoFlo 0.7 [FBP manifest](https://github.com/flowbased/fbp-manifest) 

is used, making it possible to cache components which turns process startup

faster due to less disk IO.

To cache graph's components, first install `noflo` and run

`noflo-cache-preheat` to create the FBP manifest file `fbp.json`. Then run the

graph passing the cache parameter:

    noflo-runtime-msgflo --cache true --name myworker --graph project/WorkerGraph --broker amqp://foo.cloudamqp.com/bar

TODO

-----

### 0.3

* Implement [FBP protocol over msgflo](https://github.com/noflo/noflo-runtime-msgflo/issues/30) transport