{"id":19340202,"url":"https://github.com/flowbased/fbp-spec","last_synced_at":"2025-04-23T02:31:19.473Z","repository":{"id":30436066,"uuid":"33989141","full_name":"flowbased/fbp-spec","owner":"flowbased","description":"Data-driven testing of FBP components and graphs","archived":false,"fork":false,"pushed_at":"2023-01-16T08:02:49.000Z","size":2704,"stargazers_count":32,"open_issues_count":26,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-12T10:04:56.859Z","etag":null,"topics":["bdd","fbp-protocol","tdd","test-runner","testing"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flowbased.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-15T11:08:48.000Z","updated_at":"2024-05-19T14:26:29.000Z","dependencies_parsed_at":"2023-01-14T16:57:30.574Z","dependency_job_id":null,"html_url":"https://github.com/flowbased/fbp-spec","commit_stats":null,"previous_names":[],"tags_count":51,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowbased%2Ffbp-spec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowbased%2Ffbp-spec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowbased%2Ffbp-spec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flowbased%2Ffbp-spec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flowbased","download_url":"https://codeload.github.com/flowbased/fbp-spec/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250357621,"owners_count":21417315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bdd","fbp-protocol","tdd","test-runner","testing"],"created_at":"2024-11-10T03:25:37.254Z","updated_at":"2025-04-23T02:31:19.245Z","avatar_url":"https://github.com/flowbased.png","language":"JavaScript","readme":"fbp-spec\n========\n\nA runtime-independent test framework for Flow Based Programming (FBP) component and graphs,\nusing declarative, data-driven testing.\n\nOne can use fbp-spec to do testing at multiple levels,\neach approximately corresponding to the different architectural levels of Flow Based Programming:\n\n* Unit (FBP component/subgraph)\n* Integration (FBP graph)\n* System (FBP runtime)\n\n## Status:\n\n**In production**\n\n* Used by several projects, including [Ingress Table](https://github.com/c-base/ingress-table) and [BIG IoT NoFlo bridge](https://github.com/flowhub/bigiot-bridge)\n* Tested with several FBP runtimes: [NoFlo](https://noflojs.org), [MicroFlo](https://microflo.org), [Python example](https://github.com/flowbased/protocol-examples), [MsgFlo](http://msgflo.org)\n* Runners available for contious integration (CLI, Mocha) and interactively (in [Flowhub](https://flowhub.org))\n\n## Purpose \u0026 Scope\n\nNote: `fbp-spec` is intended for use by application and component-library developers.\n\nThe following is considered out-of-scope:\n\n* Testing conformance with the FBP protocol. Instead use [fbp-protocol](https://github.com/flowbased/fbp-protocol)\n* Testing an FBP runtime/engine itself. Instead use a testing framework for your particular runtime language/environment.\n\n## License\n\nThe [MIT license](./LICENSE.md)\n\n# Usage\n\n## Installing\n\nSet up fbp-spec as an NPM dependency\n\n    npm install --save-dev fbp-spec\n\nor, install it globally. Useful if you just want the commandline tool.\n\n    npm install -g fbp-spec\n\n## Writing tests\n\nEach declared test suite loads an FBP component (or graph) fixture,\nand runs a set of test cases by sending a set of input data\nto input ports and verifying the output data against the expected results.\n\n```YAML\nname: \"Simple example of passing tests\"\ntopic: \"core/Repeat\"\nfixture:\n type: 'fbp'\n data: |\n  INPORT=it.IN:IN\n  OUTPORT=f.OUT:OUT\n  it(core/Repeat) OUT -\u003e IN f(core/Repeat)\n\ncases:\n-\n  name: 'sending a boolean'\n  assertion: 'should repeat the same'\n  inputs:\n    in: true\n  expect:\n    out:\n      equals: true\n-\n  name: 'sending a number'\n  assertion: 'should repeat the same'\n  inputs:\n    in: 1000\n  expect:\n    out:\n      equals: 1000\n```\n\n#### Multiple ports\n\nYou can send data to multiple inports and check expectations on multiple ports per testcase:\n\n```YAML\n-\n  name: '1 active track toggled high'\n  assertion: 'should give value1 color'\n  inputs:\n    tracks: 1\n    animation: [\n      0, # track idx\n      \"0xEE00EE\", # val0\n      \"0xAA00AA\", # val1\n      200, # period\n      50, # dutycycle\n      0, # offset\n      500 ] # duration\n    clock: 250\n  expect:\n    clock:\n     equals: 250\n    value:\n     equals: [0, 0x00AA] # FIXME: truncated\n```\n\n#### Sequence of packets\n\nFor testing components with state, you can sending multiple input packets\n[in sequence](https://github.com/flowbased/fbp-spec/blob/master/examples/sequences.yaml).\n\n```YAML\n-\n  name: 'sequence of data using spacy notation'\n  assertion: 'should pass'\n  inputs:\n    -\n      in: true\n    -\n      in: false\n  expect:\n    - \n      out:\n        equals: true\n    -\n      out:\n        equals: false \n```\n\n#### Extract data using path\n\nWith `path` you can specify a [JSONPath](http://goessner.net/articles/JsonPath/)\nto extract the piece(s) of data the assertions will be ran against:\n\n```YAML\n-\n  name: 'select single value'\n  assertion: 'should pass'\n  inputs:\n    in: { outer: { inner: { foo: 'bar' } } }\n  expect:\n    out:\n      path: '$.outer.inner.foo'\n      equals: 'bar'\n-\n  name: 'selecting many correct values'\n  assertion: 'should pass'\n  inputs:\n    in:\n      outer:\n        first: { foo: 'bar' }\n        second: { foo: 'bar' }\n  expect:\n    out:\n      path: '$.outer.*.foo'\n      equals: 'bar'\n```\n\n#### Skipping tests\n\nSetting `skip` property on a testcase or suite, will cause it to not be ran.\nShould contain a message of the reason for skipping.\n\n```YAML\n-\n  name: 'a test that is skipped'\n  assertion: 'will not be ran'\n  inputs:\n    in: 1000\n  expect:\n    out:\n      equals: 1000\n  skip: 'not implemented yet'\n```\n\n#### Using fixtures\n\nOne can use testing-specific components in the fixture, to simplify\ndriving the unit under test with complex inputs and performing complex assertions.\n\n```YAML\nfixture:\n type: 'fbp'\n data: |\n  INPORT=imagename.IN:NAME\n  INPORT=testee.PARAM:PARAM\n  INPORT=reference.IN:REFERENCE\n  OUTPORT=compare.OUT:SIMILARITY\n\n  generate(test/GenerateTestImage) OUT -\u003e IN testee(my/Component)\n  testee OUT -\u003e ACTUAL compare(test/CompareImage)\n  reference(test/ReadReferenceImage) OUT -\u003e REFERENCE compare\ncases:\n-\n  name: 'testing complex data with custom components fixture'\n  assertion: 'should pass'\n  inputs:\n    name: someimage\n    param: 100\n    reference: someimage-100-result\n  expect:\n    similarity:\n      above: 0.99\n```\n\n#### Supported assertions\n\nInstead of `equals` you can use any of the supported assertion predicates. Examples include:\n\n    type\n    above\n    below\n    contains\n    haveKeys\n    includeKeys\n\nFor a full set of assertions, see [the schema](https://github.com/flowbased/fbp-spec/blob/master/schemata/expectation.yaml)\n\n#### More\n\nA comprehensive set of examples can be found under [./examples](./examples).\nFor the detailed definition of the dataformat for tests, see [schemata/](./schemata/).\n\n\n## Running tests with fbp-spec commandline tool\n\nThe simplest and most universal way of running tests is with the `fbp-spec` commandline tool.\n\n    $ fbp-spec --address ws://localhost:3333 examples/multisuite-failandpass.yaml\n\n    MultiSuite, failing tests\n      sending a boolean with wrong expect\n        should fail: ✗ Error: expected true to deeply equal false\n      sending a number with wrong expect\n        should fail: ✗ Error: expected 1000 to deeply equal 1003\n    MultiSuite, passing tests\n      sending a boolean\n        should repeat the same: ✓\n      sending a number\n        should repeat the same: ✓\n\nThe `--command` options can be used to specify a command which will start the runtime under test:\n\n    fbp-spec --command \"python2 protocol-examples/python/runtime.py\"\n\nIt sets the exit status to non-zero on failure, so is suitable for integrating into a `Makefile` or similar.\n\n## Running tests by integrating with Mocha\n\n[Mocha](http://mochajs.org/) iss a popular test runner framework for JavaScript/CoffeeScript on browser and node.js.\n\nSince fbp-spec communicates with your runtime over a network protocol,\nyou can use this also when your project is not JavaScript-based.\nThe Mocha runner is for instance [used in microflo-core](https://github.com/microflo/microflo-core/blob/master/spec/ComponentTests.coffee)\nto test C++ components for microcontrollers \u0026 embedded devices.\n\nYou can have your fbp-spec tests run in Mocha by calling the `fbpspec.mocha.run()` function, in a file which is\nexecuted with the standard Mocha runner. Eg. `mocha --reporter spec tests/fbpspecs.js`\n\n    // fbpspecs.js\n    fbpspec = require('fbp-spec');\n\n    rt = {\n      protocol: \"websocket\",\n      address: \"ws://localhost:3569\",\n      secret: \"py3k\", // Optional. If needed to connect/authenticate to runtime\n      command: 'python2 protocol-examples/python/runtime.py' // Optional. Can be used to start runtime automatically\n    };\n    fbpspec.mocha.run(rt, './examples/simple-passing.yaml', { starttimeout: 1000 });\n\nThe tests can be specified as a list of files, or directories.\nYou can use the standard `grep` option of Mocha to run only some tests.\n\nFor CoffeScript example, see [./spec/mocha.js](./spec/mocha.js).\n\n## Running tests interactively in Flowhub\n\n[Flowhub](http://app.flowhub.io) IDE (version 0.11 and later) has integrated support for fbp-spec. No installation is required.\n\n* Open existing project, or create a new one\n* Open a component, and write/copypaste in a test in the `Tests` panel\n* Ensure you have a runtime set up, and connected\n\nWhen you make changes to your project (components,graphs) or tests, Flowhub will now automatically (re-)run your tests.\nYou can see the status in the top-right corner. Clicking on it brings up more details.\n\n## Generating tests programatically\n\nThe test-format defined by fbp-spec is fairly generic and versatile. It is intended primarily as\na format one directly specifies tests in, but can also be generated from other sources.\n\nSometimes data-driven testing, one does a large amount of very similar tests,\nwith multiple test-cases per set of input data.\nBy capturing only the unique parts of testcases in a specialied data-structure (JSON, YAML, etc),\nand then transforming this into standard `fbp-spec` files with some code, adding/removing\ncases becomes even easier.\nFor instance in `imgflo-server`, [testcases](https://github.com/jonnor/imgflo-server/blob/master/spec/graphtests.yaml)\ncan be defined by providing a name, an URL and a reference result (a file with naming convention based on name).\n\nSimilarly, one can generate testcases using fuzzing, schema-based, model-based or similar tools.\n\n## Integrating test runner in an application\n\nThe test runner code is accessible as a JavaScript library,\nand can be integrated into other apps (like Flowhub does).\nSee examples of [commandline](./lib/cli.js) and webappp usage.\n\n## Add supporting for a new runtime\n\nYou need to implement the [FBP network protocol](https://github.com/flowbased/fbp-protocol).\nAt least the `protocol:runtime`, `protocol:graph`, and `protocol:network` capabilities are required.\n\nAll transports supported by [fbp-protocol-client]((https://github.com/flowbased/fbp-protocol))\nare supported by fbp-spec, including WebSocket, WebRTC, and iframe/postMessage.\n\nfbp-spec is intended to be used with flow-based and dataflow-programming,\nbut might be useful also outside these programming paradigms. Try it out!\n\n## Writing a test runner in another language\n\nAs long as you stay compatible with the [fbp-spec testformat](./schemata/)\nand [FBP protocol](http://noflojs.org/documentation/protocol/),\nyou can implement a compatible runner in any programming language.\n\nYou can consider the fbp-spec code (in CoffeeScript) as a reference implementation.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflowbased%2Ffbp-spec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflowbased%2Ffbp-spec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflowbased%2Ffbp-spec/lists"}