{"id":28089141,"url":"https://github.com/reactivesets/toubkal","last_synced_at":"2025-05-13T12:54:10.076Z","repository":{"id":5945191,"uuid":"7165892","full_name":"ReactiveSets/toubkal","owner":"ReactiveSets","description":"Fully reactive programming for nodejs and the browser","archived":false,"fork":false,"pushed_at":"2023-03-04T02:55:51.000Z","size":12671,"stargazers_count":72,"open_issues_count":16,"forks_count":13,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-15T19:23:14.328Z","etag":null,"topics":["dataflow","javascript","publish-subscribe","reactive","reactive-dataflows","transactional"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ReactiveSets.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-12-14T13:30:38.000Z","updated_at":"2025-02-07T09:42:32.000Z","dependencies_parsed_at":"2023-01-13T13:46:02.687Z","dependency_job_id":null,"html_url":"https://github.com/ReactiveSets/toubkal","commit_stats":null,"previous_names":["connectedsets/connectedsets"],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReactiveSets%2Ftoubkal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReactiveSets%2Ftoubkal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReactiveSets%2Ftoubkal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ReactiveSets%2Ftoubkal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ReactiveSets","download_url":"https://codeload.github.com/ReactiveSets/toubkal/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253948328,"owners_count":21988953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataflow","javascript","publish-subscribe","reactive","reactive-dataflows","transactional"],"created_at":"2025-05-13T12:54:09.223Z","updated_at":"2025-05-13T12:54:10.053Z","avatar_url":"https://github.com/ReactiveSets.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Toubkal\n\n**Fully reactive programming for nodejs and the browser**\n\n*Liberating your Creativity by improving your Productivity and runtime Performances*\n\n*1972 Continuous Integration Tests*\n\n[![Travis CI Build Status](https://travis-ci.org/ReactiveSets/toubkal.png?branch=master)](https://travis-ci.org/ReactiveSets/toubkal)\n[![npm version](https://badge.fury.io/js/toubkal.svg)](https://badge.fury.io/js/toubkal)\n\n[![Join the chat at https://gitter.im/ReactiveSets/toubkal](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/ReactiveSets/toubkal?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\nThe current version allows rapid development of non-trivial, complex,\nall-reactive applications. We are using it in production applications\ndevelopped for and with our clients.\n\n## Teaser\nDisplaying a reactive ```\u003ctable\u003e``` which DOM container is ```#sales_table```, ordered by date,\nfor the years 2013 \u0026 2014, from a source ```sales``` dataflow coming from a ```socket.io``` server,\npulling the minimum amount of data from the server and updating the table as soon as some\ndata is available from the server\n(complete working code including http server is available at\n[examples/teaser](https://github.com/ReactiveSets/toubkal/tree/master/examples/teaser)):\n\n#### client.js\n\n```javascript\nrs.socket_io_server()\n  .flow  ( 'sales' )\n  .filter( [ { year: 2013 }, { year: 2014 } ] )\n  .order ( [ { id: 'date' } ] )\n  .table ( $( '#sales_table' ), sales_columns )\n;\n```\n\n#### How does this work?\n\n```sales_table``` is updated reactively in realtime whenever sales are updated on the server.\n\n```rs.socket_io_server()``` connects the client to Toubkal socket.io server.\n\n```flow( 'sales' )``` declares that the ```sales``` dataflow is needed from the server.\n\n```[ { year: 2013 }, { year: 2014 } ]``` is a filter *query*, it controls how much sales data will be\npulled from the server therefore reducing both bandwidth usage and latency.\n\nLatency is further reduced by displaying the table as soon as the first sales come\nfrom the server, improving user experience.\n\nThis query can be a dataflow, updated by DOM controls and automatically pulling the\nminimum amount of data from the server.\n\n```[ { id: 'date' } ]``` is an *organizer*, it can also be a dataflow dynamically\nupdated by DOM controls, or any other application source.\n\nThe ```sales_columns``` dataflow controls table's columns. When updated, columns\nare reactively added or removed in realtime without any addtitional programing required.\n```sales_columns``` can be defined in the client or also come from the socket.io server\nusing the following declarative code:\n\n```javascript\nvar sales_columns = rs\n  .socket_io_server()\n  .flow( 'sales_columns' )\n;\n```\nThe above code automatically shares the same socket.io connection with the previous code,\nreducing resource usage on both clients and servers while only pulling from the server\nthe additional ```sales_columns``` dataflow.\n\nTable updates are optimized to add and remove the minimum set of rows and columns,\nimproving client responsiveness, battery life and user experience.\n\n#### What does this mean?\n\nThe Toubkal program above is expressed in one third the words required\nto express the problem in plain english replacing thousands of lines of\ncomplex and error-prone code.\n\nToubkal programs have no *loops*, and no *ifs*, dramatically reducing\nthe likelyhood of bugs and hence greatly improving productivity.\nUnder the hood, Toubkal provides all the optimized and comprehensively\ntested *loops* and *ifs* you need.\n\nThese same declarative techniques are applied on the server side\ndelivering a full stack scallable and secure framework with highest\nperformances featuring a reactive database and fine-grained authorization\ndesign patterns.\n\nThe bottom line is that Toubkal allows you to write with less code higher\nperformance fully reactive applications, liberating your creativity.\n\n## Installation\n\nFrom npm, latest release:\n```bash\n# npm install toubkal\n```\n\n## Documentation\n\nThis readme provides a short introduction to Toubkal.\n\nFull reference documentation including internals and the Toubkal\nprotocol is available at\n[https://toubkal.reactane.com/](https://toubkal.reactane.com/).\n\nThis documentation site is a Toubkal reactive application. It updates\nautomatically after each commit is pulled on our developpement server.\n\nDocumentation is extracted from code using the following Toubkal\nserver pipelets:\n- **[acorn()](https://toubkal.reactane.com/#pipelet_acorn)**: parse javascript using the acorn library\n- **[parse_documentation()](https://toubkal.reactane.com/#pipelet_parse_documentation)**: emit documentation items from parsed comments\n- **[documentation_markdown()](https://toubkal.reactane.com/#pipelet_documentation_markdown)**: format documentation items into markdown\n- **[markdown()](https://toubkal.reactane.com/#pipelet_markdown)**: markdown to html converter using \"markdown-it\" and \"highlight.js\"\n- **[documentation_manuals()](https://toubkal.reactane.com/#pipelet_documentation_manuals)**: Toubkal documentation manuals metadata (not content)\n\nTo work on the documentation on a local machine, run the documatation site:\n```bash\n  node site/server.js \u003e site.out\n```\n\nThen point a web browser at [localhost on port 8082](http://localhost:8082/).\n\n## Automated Tests, Continuous Integration\n\nWe have curently developped 1972 continuous integration tests for the Toubkal\ncore and library pipelets that run after every commit on Travis CI under\nnode version 8.12.0 LTS.\n\nIn the event that a test does not pass our top priority is to fix the test\nbefore anything else. We usualy fix broken tests within hours.\n\nThese tests also pass on Windows under [Cygwin](https://www.cygwin.com/)\nwhich although not officially supported by node, works with some caveats.\n\nIn any case, one should expect Toubkal applications to run equally well\non Windows and Linux.\n\nWe also do manual testing on the following web browsers:\n- Chrome (latest),\n- Firefox (latest),\n- Safai (latest),\n- IE 11.\n\n### Running tests locally\n```bash\n# npm install -g coffee-script\n# npm install -g mocha\n# git clone https://github.com/ReactiveSets/toubkal.git\n# cd toubkal\n# npm install\n# ./run_tests.sh\nFull test results are in test.out\n-\u003e passed 1972 of 1972 tests\n#\n# less -R test.out # for tests detailed traces\n```\n\n### Running core tests in a browser:\n```bash\n# node test/server/http.js\n```\n\nThen browse http://localhost:8080/test/manual/\n\n### Running browser examples:\n```bash\n# node examples/server.js\n```\n\nThen browse http://localhost:8081/\n\n## Development Stage\nToubkal is already quite reliable thanks to its comprehensive test suite\nand is currently used to deliver complex, all-reactive, enterprise\nprogressive web applications for clients.\n\nThe architecture of Toubkal is now stable, with main components (pipelets,\nplugs, queries, transactions, and more) well defined and implemented.\n\nAlthough the API may still change from time to time, we have entered a\nmore mature phase where the high level API is now quite stable, while\nlower-level API changes much less often.\n\n## Our team\nToubkal is developped by a dedicated small team of experienced and\npassionate full-stack developers for the web and other technologies.\n\nWe are now fully funded and profitable by delivering Toubkal\napplications for our clients at [Reactane](https://www.reactane.com/).\n\nWe have been developping Toubkal since December 2012 and cannot\nimagine ever going back to programming the old way.\n\nIf you are an experienced JavaScript programmer, understand the power\nof all-reactive programming and would like to join our team, please\ncontact us.\n\n## Introduction\nToubkal is a high-productivity, high-performances, scalable, all-reactive\nweb application library aiming at improving your productivity for the\ndevelopment of complex applications, reducing servers' environmental\nfootprint, and increasing mobile clients battery life by making an\noptimal use of server, network and client resources.\n\n### Why yet-another JavaScript application library?\nThe short answer is because we are not satisfied with the productivity,\nperformances, and authorization models, of existing frameworks and\nlibraries.\n\nInternet servers are consuming an increasily significant share of\nworldwide electricity production, contributing to climate change and\nthreatening Internet growth as the availability of cheap fosil fuels\ndecreases due to population growth and per capita consumption growth.\n\nThe power of Internet server is now mostly increasing through the\naddition of CPU cores, meaning that the key to efficient usage of server\nresources must shift from raw-single-thread performence to high\nconcurrency and parallelism performance. This in turn requires new\nprogramming patterns to keep, or increase programmers' productivity.\n\nAlso, one must realize that the bulk of the vast majorty of today's\napplications is about controling the motion of data throughout the\nnetwork. Such data is no-longer limited to strictly public or strictly\nprivate informtation, requiring complex authorization schemes. This calls\nfor new programing patterns that allows to greatly simplify the\nmanagement of user authorizations well beyond all-or-nothing\nauthentication.\n\n#### What do you mean by performances?\nOur first priority is high-performances, because we believe that\nperformance is the key to better user experience, lowest operational\ncosts, and lower environemental footprint.\n\nWe are fighting simultaneously against:\n- **CPU cycles** that consume energy to run and cool-down servers,\nslow-down mobile clients, and **drain mobile batteries** faster\nthan anyone desires;\n- **Latency** decreasing the responsiveness of applications and\nuser experiences;\n- **Bandwidth** usage that consume energy, and increase latency over\nlower-bandwidth networks.\n\nWe also want to keep **good performances at scale**. Most libraries\neither do-not-scale or scale with lower per-server performances further\nincreasing costs while increasing environemental footprints.\n\nToubkal addresses all of these issues thanks to its unique\n**Publish / Subscribe** all-reactive dataflow model that works accross\nweb browsers and nodejs servers, as well as just-in-time code generators\nand other unique optimizations.\n\nMost importantly Toubkal provides a programing model that can be\nfurther optimized while maintaining user-code compatibility.\n\n#### What's the big deal about authorizations?\n\nWriting a complex application is hard-enough, add to this any significantly-complex\nauthorization scheme and the whole thing breaks apart, slows-down to a crawl, clutters\nthe code with plenty of unspotted security holes throughout every part of the\napplication, and every now and then exposes end-users' data to unauthorized users.\n\nMost companies try to get away with it by sweeping each leak under the carpet and\npromissing end-users that this will never happen again, or better yet, that this never\nhappened. Internally, this usually ends-up with more meetings and paperwork, less\nthings done for a system that although marginally improved, will at best remain unproven.\n\nBecause it is so hard, most frameworks take a this-is-not-our-problem approach to\nauthorizations by stating that you should use third-party libraries or plugins to deal\nwith it, all of which have shortcomming and usually will not fit the complexity of any\nreal-world application let alone provide acceptable performances at scale.\n\nToubkal provides a simple yet efficient dataflow authorization model and system\narchitecture that delivers **Reactive UI updates on authorization changes** at scale.\n\nNow, you might consider that you don't need this, that end-users can refresh their page\non authorization changes. But the reality is that we can do this because we provide a\nmodel that works in all cases, without requiring you to write a single additional line of\ncode, so that you can sleep at night knowing that end-user data cannot be exposed by some\npiece of code that forgot to test a role in a corner-case.\n\n#### How does Toubkal improve your productivity?\n\nBy allowing you to describe **what** you need in a declarative style, instead of\n**how** this could ever be accomplished.\n\nFiguring-out **how** this should a) work securely, b) scale and c) have best performances\nas stated above, is hard, really hard. So hard that today the only way to achieve this is\nthrowing millions of dollars at the problem, and/or struggling with bugs, bottlenecks\nand hard-to-work-around architecture limitations.\n\nThe most important concept you need to know to understand Toubkal programs is about\n**Toubkal Pipelets** and how to combine them to create programs that react to data\nchange events:\n\n```javascript\nrs.upstream_pipelet  ( parameter, ... )\n  .a_pipelet         ( parameter, ... )\n  .downstream_pipelet( parameter, ... )\n;\n```\n\nA pipelet is a **factory function** which instances:\n- Maintain a **dataset** state, in memory, mass storage, the DOM, or virtually (stateless)\n- **Subscribe to** data change events from **upstream** pipelets\n- **React** to upstream events to update their dataset\n- **Emit** change events to **downstream** pipelets\n- Have **no side effects** on other pipelets upstream or downstream\n- Are **piped** to upstream and downstream pipelets using the **'.' operator**\n- May be connected to additional pipelets using parameters\n- Can be **composed** with other pipelets to provide new pipelets\n\nA Toubkal program is a JavaScript program where one can mix imperative-style programming\nwith Toubkal declarative-style programming.\n\nToubkal's **Publish / Subscribe** reactive model allows to solve the **how** so that you\ndon't have to deal with it.\n\nTo make it easier, the API describes **what** you want in **plain JavaScript** without\nrequiring a graphical UI to glue hard-coded and hard-to-comprehend xml or json \"nodes\"\nand \"links\" together as many other dataflow libraries require.\n\nToubkal reactive dataflow model provides higher level abstractions handling under the\nhood both subscribe dataflows and information push dataflows that allow to move the\nleast amount of information possible between clients and servers reactively.\n\n### Toubkal Publish / Subscribe Dataflow Model\n\nThe following describes implementation details implemented at Toubkal's low level.\nApplication Architects do not need do program anything for this to happen as it is\nentirely hidden by Toubkal pipelets. Understanding of the underlying model helps understand\nwhy Toubkal is so efficient and how it scales.\n\nMost dataflow libraries usually implement one of two models:\n- push: all data is pushed downstream as it happens, allowing realtime updates\n- pull: data is pulled upstream when needed, allowing lazy programming, pulling only what\nis required, when required\n\nFor web applications' communications between servers and clients these two models are\nusually not acceptable for these reasons:\n\n- The pull method is not reactive, introducing average latency of the polling period.\nWorse it and can consume large amounts of bandwidth if the polling period is too small.\nIt can nonetheless be used efficiently on the client side along with\nrequestAnimationFrame() to prevent over-updating the DOM between refreshes.\n- The push method pushes all data regardless of what the downstream many need. This can\nresult in the transmission of large amounts of unused data, usually introducing\nunacceptable latency and bandwidth charges.\n\nToubkal implements a **Publish / Subscribe** model where downstream pipelets subscribe to\nthe subset of data they are interested in and subsequently receive all updates in a push\nfashion only for that subset. This allows Toubkal to move less data between clients and\nservers while remaining realtime with lower latency.\n\nToubkal stateless pipelets use a lazy model where they will not subscribe to anything\nfrom upstream unless initial data is fetched by a downstream stateful pipelet. This again\nallows to transmit only what is really used by the application at any given time.\n\nA subscription is done using a query dataflow that represents a kind of filter on the\nupstream dataflow. Because the query is itself a dataflow, the subcription can change\nover time.\n\nWhen tens of thousands of downstream pipelets subscribe to a single pipelet using\ndifferent queries, Toubkal provides a query tree that routes data events very efficiently\nin O( 1 ) time (i.e. that does not depend on the number of connected clients) therefore\nproviding a more scalable solution within a single server. Sending actual data to n\nclients out of N connected clients is O( n ) so actual performances depends on the\napplication (i.e. whether n \u003c\u003c N or not).\n\nA network of Toubkal servers can be arranged in a tree-like fashion to provide\nunlimited size query trees, e.g. to dispatch data to millions of simultaneous clients.\nEach server subscribes to its upstream server the subset of data it dispatches to\ndownstream servers and clients. This allows efficient and low-latency routing thanks\nin part to the high performances of each individual server query tree.\n\n#### Data Events, Operations, Stateless Sets and Pipelets\n\nInternally, Toubkal dataflows represent the evolution of data sets over time where\neach event modifies a set. These dataflows are therefore reactive sets change flows.\n\nEach event carries an opperation name such as *add* or *remove* and an array of values\nto add to, or remove from, a set.\n\n**Stateless** pipelets process values which are not materialized either in memory or\nother storage, their state is vitual.\n\nStateless pipelets process data events independently of all other events and values in\nthe set allowing faster operations and lower memory footprints.\n\nStateless pipelets can therefore process events out of order, much like Internet Protocol\npackets can be routed through various paths within the Internet and may arrive at their\ndestinations in any order.\n\n#### Stateful Pipelets\n\nA **Stateful** pipelet maintains the state of a set either in memory, in mass storage,\nor any other API that provides a storage behavior.\n\nUser Interface pipelets are stateful as these present the state of a set through the DOM.\n\nMuch like the TCP protocol in the Internet which is responsible for end-to-end\ncommunications consistency, Stateful pipelets may receive data events in any order and\nare responsible for maintaining an application-consistent state.\n\nStateful pipelets are implemented thanks to the stateful set() pipelet that is typically\nused as a base pipelet for stateful pipelets.\n\nAlso, much like the TCP protocol, stateful pipelets are found at the edges of a\nToubkal network of stateless pipelets.\n\n#### Horizontal Distribution\n\nAllowing out-of-order data events is a key feature of Reactive Sets which greatly eases\nhorizontal distribution of workloads and charding, because no synchronization is needed\nbetween chards that may be processed in parallel either over a number of threads,\nprocesses, or servers in a true share-nothing architecture.\n\n#### Incremental Processing\n\nIncremental sets processing allows to split large sets into optimal chunks of data\nrendering data to end-users' interface with low-latency, improving end-user experience.\nData events update sets in real-time, on both clients and servers.\n\nIncremental aggregates allow to deliver realtime OLAP cubes suitable for realtime data\nanalysis and reporting over virtually unlimited size datasets.\n\n#### Loops, Just-In-Time Code Generation\n\nToubkal data events contain arrays of values which are typically processed in loops. In a\ntraditional programming environement, one typically writes code that processes values in\nloops. With Toubkal, architects do not write loops because these are absracted away as sets\nprocessed by pipelets.\n\nThis greatly simplifies programming while removing the likelihood for common programming\nerrors.\n\nHighest performances are provided thanks to Just-In-Time code generators delivering\nperformances only available to compiled languages such as C or C++. Unrolling nested\nloops provide maximum performance while in turn allowing JavaScript JIT compilers to\ngenerate code that may be executed optimally in microprocessors' pipelines.\n\n#### Toubkal Low-Level Pipelet Programming\n\nAt the lower level, Toubkal **Pipelets** use a JavaScript functional programming model\neliminating the typical callback hell of asynchronous request-response programming\nmodels.\n\nError and log dataflows originating on clients can easily be routed to servers to allow\nproactive debugging of errors while in production, and effective service quality\nmonitoring.\n\nTransactions allow to group related operations over time and allow synhronization\nof concurrent dataflows.\n\nDevelopping stateless pipelets is straightforward, requiring to write a simple and\nsimple transform function very much akin pure functional programming. Stateless Pipelets\nAPI takes care of everything else, managing add, remove, fetch functions as well as\ntransactions.\n\nDevelopping stateful pipelets requires to implement add and remove functions, a fetch\nfunction to return initial state, and properly handle transactions and out-of-order\noperations.\n\n### Service Architecture\n\nWith Toubkal, services are typically composed of three different services:\n\n- A stateful network of persistent database and external services pipelets\n- A stateless network of event dispatchers, acting as a marshalled multicasting network\nfor dataflows\n- A stateful network of client widgets delivering applications to end-users\n\nFor small applications with few simultaneous users the first two typically reside in a\nsingle server, while complex applications with large number of active users will be\nrunning on different servers. Because pipelets share no state they can easily be\ndistributed.\n\nA company could run multiple services through a single network of stateless event\ndispatchers, acting as web service aggregator.\n\nThe different nodes of a Toubkal network communicate using the Toubkal protocol that\nprovides the Publish / Subscribe service over a reliable transport (such as Sockets,\nWebSockets, ...) but not necessarily guarantying the order of packets. So Toubkal could\nalso work over a protocol that would only guaranty the delivery of packets.\n\nThe Toubkal protocol therefore provides a higher level alternative to existing web\nservices protocols such as SOAP and REST, allowing to build efficiently complex real-time\napplications with no additional code and less documentation since only application\ndataflows need to be documented.\n\n### Realtime Data Portability, Business Opportunities\n\nA network of services sharing the same event dispatcher network enables to effectively\nseparate **Toubkal Data Providers** from **Toubkal Application Providers** increasing\nbusiness opportunities arising from the portability of reactive dataflows updated in\nreal-time and as authorized by end-users and data-licenses.\n\nWithin a Toubkal network, end-users no longer need to duplicate their personal data\nendlessly and updates are propagated to all applications in realtime putting an end\nto today's world of out-of-date data between services.\n\nPeople will now expose their data, using a variety of services to view, edit, and publish\ntheir data to other people.\n\nUsing only stateless pipelets, this architecture will reach internet-scale very\nefficiently, delivering a Marshalled publish / subscribe multicasting data exchange\nfor services to share data among many service providers, while representing a business\nopportunity for **Toubkal Network Providers** much like today's CDNs but for marshalled\ndynamic real-time content solving caching issues thanks to the immutability of data\nevents.\n\nTo participate in this network, service providers only need to publish dataflows and/or\nsubscribe to third-party dataflows.\n\nEnd-users may use these services to backup their own data either on owned servers or\nusing third-party Toubkal Data Providers.\n\nEnd-Users control access to their own data through Toubkal Authorization dataflows\nproviding additional business opportunities for **Toubkal Authorization Management Providers**\nhelping end-users manage authorizations for all their data accross all their Toubkal\nApplications.\n\nMonetization of dataflows and applications can be controlled through Toubkal reactive\nauthorizations by **Toubkal Monetization Providers**.\n\nDisruptive new business opportunities arrising from **Toubkal Realtime Data Portability**\nwill prove stronger than the current closed, data-within-application model, resulting in\nmore data and more services available to more users and businesses.\n\n### Ecosystem\n\nToubkal backend runs on **Node.js** providing a scalable database, web\nserver, validation, and authorizations.\n\nOn the frontend, Toubkal provides reactive controlers and views driven\nby reactive dataflows.\n\nToubkal can optionally be coupled with any other framework but we\nrecommend using reactive libraries such as **AngularJS**, **Ember**,\n**Bacon.js**, **React**, which model is closer to Toubkal.\n\nFor responsive layouts, we recommand **Bootstrap** that we use it for our\nreactive Carousel and Photo Albums.\n\nFor DOM manipulation one can use any library, or none at all, as Toubkal\ncore has no dependencies.\n\nToubkal can either be used to gradually improve existing applications on\nthe back-end or front-end, or as a full-stack framework.\n\n### Integrated reactive database and model\n\nToubkal features a reactive database model with joins, aggregates, filters\nand transactions with eventual consistency allowing both normalized and\ndenormalized schemes.\n\nA reactive MySQL driver for Toubkal is available at\n[https://github.com/ReactiveSets/toubkal_mysql](https://github.com/ReactiveSets/toubkal_mysql).\n\n## Roadmap and Release Notes\n\nCurrent work in progress is for version 0.4. Expected release date is\nDecember 12th 2020.\n\n### Version 0.4.0 - Transactional Operations / Complex Expressions / DOM Handling / Documentation\n\nWork In Progress.\n\nThis release is a major stabilization and framework release using the\ntechnics we have learned using toubkal in production for serveral years\nnow. We are therefore developing pipelets representing best toubkal\npractices for building fully reactive applications.\n\n#### Main goals remaining\n\n- Authorizations, validationm error routing, eventual consistency\n  - Schema-based authorizations and validation\n  - Write authorizations and validation at the transactional level\n  - Adding sender to transaction to route errors back to sender\n  - Eventual consistency:\n    - Revert writes on transaction errors\n    - Acknowlege successful transactions completion to sender\n\n- Integration with third-party client libraries for both native apps and\n  view.js through a toubkal client API.\n\n- Complete Safe Complex Query expressions\n  - Complex query to SQL query\n  - Implement stateful features as operators, options, or special attributes\n    - limit value\n    - order_by expressions [,descending]\n    - group_by dimensions, measures\n    - pick\n\n- Rewrite and complete lib/server/file.js pipelets\n  - fs_watch()\n  - fs_lstat()\n  - fs_read_link()\n  - fs_read_directory()\n  - fs_append_file()\n  - fs_chmod()\n  - fs_chown()\n  - fs_mkdir()\n  - fs_rmdir()\n  - fs_mkdirp()\n  - fs_link()\n  - fs_symlink()\n  - fs_unlink()\n  - fs_rename()\n  - watch() and watch_directories() should be composed from redesigned\n  pipelets.\n\n#### Work in progress\n\n- 1972 Continuous Integration tests\n  - Exceeded goal which was at least 1750 continuous integration tests\n\n- Passport-express integration:\n  - CI tests for passport authrntication\n  - Refactor implementation, new pipelets WIP:\n    - express_use_passport(): Initialize express instance with passport instance\n    - express_application(): Instanciate an express application and get\n      express application instance\n    - passport_instance(): Instanciate a passport instance for an application\n\n- Error routing, eventual consistency\n  - Rollback transactions to recover from errors\n    - Using revert(), reverting add into remove, remove into add, and swapping update operations\n\n- Complex authorizations and validation\n  - Implement verrifications at the transaction level from schema\n  - new pipelets WIP:\n    - validate_schema_transactions()\n    - schema_to_models()\n    - fetched_transactions()\n    - validate_transactions()\n\n- Toubkal site, featuring documentation and examples\n  - Documentation available as a single page\n\n#### Completed Work (versions 0.3.x, x \u003e 0)\n\n- Other new pipelets:\n  - emit_transactions()\n\n- Pipelet api to ease integration with other libraries, e.g. front-end\n  libraries and native applications. It exposes a stable API providing\n  methods add(), remove(), update(), fetch(), update_subscriptions(),\n  and on().\n\n- new file system pipelets and improvements:\n  - fs_stat()\n  - fs_read_file()\n  - fs_write_file()\n  - directory_entries()\n  - all_directory_entries()\n  - configuration() write\n\n- Pipelets alter() amd map() now allow assynchronous cancelable transforms\n  using functions cancelable_map() and cancelable_limiter(). This form\n  also allows filter() and flat_map() behavior.\n  \n  If transform has more than 2 parameters it is consided stateful.\n\n- Safe Complex Query expressions\n  - Query and Query_Tree methods\n    - Query..generate(): Generate filter code from complex queries\n    - Query..and(): AND complex queries\n    - Query.least_restrictive_keys(): optionally try to determine which\n    AND-expression describes the largest set. Optimized for expressions\n    with one compararison operator in '\u003c', '\u003c=', '==', '\u003e=', '\u003e'. \n    - Query..differences(): Allow comparison of complex queries using\n    value_equals().\n    - Query_Tree..add(): Add complex query terms into query tree\n    - Query_Tree..remove(): Remove complex query terms from query tree\n    - Query_Tree..route(): Route operations with complex queries\n  \n  - Sanitized for safe execution on server even when crafted by untrusted clients\n  \n  - For execution by upstream servers to reduce bandwidth, cpu usage and latency\n  \n  - JSON Objects and Arrays for any JSON transport\n  \n  - Side-effect free\n  \n  - Any depth Abstract Syntax Tree, may be limited to prevent Denial of Service attacks\n  \n  - Consistent and rich semantic\n    - Nested Object and Array expressions\n    \n    - Regular expressions\n    \n    - All progressive operators, allowing to express the follozing:\n      ```18 \u003c= age \u003c= 25```\n      ```sales / ( 0 != count ) \u003e 1000```\n  \n  - Operators\n    - Control flow       : ```\u0026\u0026 || ! failed reset```\n    - Literals           : ```$ $$```\n    - Deep inspection    : ```. _ __```\n    - Grouping           : ```[]```\n    - Comparison         : ```== != \u003e \u003e= \u003c \u003c=```\n    - Ranges             : ```\u003c\u003c \u003c=\u003c \u003c=\u003c= \u003c\u003c=```\n    - Search in Array    : ```in```\n    - Search in set      : ```in_set```\n    - Arithmetic         : ```+ - * / %```\n    - Regular expressions: ```RegExp match match_index group split```\n    - Array / String     : ```length```\n    - Object valueOf     : ```value```\n    - Date               : ```Date year month day hours minutes seconds milliseconds time```\n    - Custom operators   : defined using JavaScript functions but used as JSON Strings and Arrays to prevent code-injection\n  \n  - Example: Expression to get active users whom last logged-in between 2005 included and 2013 not included:\n\n```javascript\n    {\n        flow   : 'user'\n      , active : [ '==', true, '||', '==', 1 ]\n      , profile: {\n            last_logged_in: [ '$', 2005, '\u003c=', [ 'year' ], '\u003c', 2013 ]\n        }\n    }\n```\n\n- Redesign of Plug.._fetch() \u0026\u0026 ..update_upstream_query():\n  - Synchronize update_upstream_query() and fetch() to guaranty data consistency\n  - Fetch receivers can now receive removes, updates and operations options, allows complementary sets\n  - Allow query transforms on all plugs\n  - Filters fetch_unfiltered() if defined\n  - Filters through _transform() if defined\n  - Cancel ongoing fetches when pipelet disconnects (still needs to revert previously fetched values)\n  - Abort cancelled fetches upstream for which fetched values will be ignored\n  - Union support for adding and removing sources while fetching\n  - Union fetch signals terminated sources downstream allowing to listen to these source while other sources terminate\n  - All plugs now have a source, enables plugs pipelines for fetch() \u0026\u0026 update_upstream_query()\n  - Implemented in Plug base class of all inputs and outputs\n  - Replaces all implementations of fetch on inputs and outputs\n  - Forward compatible with fetch() API, the API may be deprecated once migration to new API is complete\n\n- Allow automatic synchronization of all inputs of a pipelet, as long as one\n  uses method Pipelet.._add_input() to add additional inputs. This relies on\n  sharing input transactions between all inputs, and other modifications to\n  make this work properly on controllets. Implemented in filter() and\n  $to_dom().\n\n- Reinstate update as a first-class operation, making it easier to handle updates\n\n- Documentation extraction format from source code comments:\n  - Using pipelets:\n    - acorn()\n    - documentation_manuals()\n    - parse_documentation()\n    - documentation_markdown()\n    - markdown()\n  \n  - highly targeted towards Toubkal dataflow programming\n  \n  - augmented github-flavored markdown\n  \n  - output as dataflow, suitable for transformations to github-flavored markdown, plain html, and more\n  \n  - Input format:\n    - inside any comments\n    - line-oriented\n    - indentation-sensitive\n    - can contain github-flavored markdown\n    - The \"@\" character is interpreted as meta-data for this format, can be escaped using the \"\\\\\" character. \n\n  - @tag: indicates a documentation sub-section, first tag in a comment indicates start of documented item:\n    - if tag is followed by a column \":\", the sub-section is multiline, otherwise it is contained on a single line\n    - if an unknown tag is found, a warning is emitted\n    - plurals indicate a list of items described in paragraphs starting with a hyphen \"-\"\n\n    - list of top-level tags:\n    \n      Tag           | Description\n      --------------|---------------------------------\n      @term         | a term\n      @namespace    | namespace, e.g. \"rs\"\n      @flow         | a dataflow name\n      @pipelet      | pipelet signature\n      @function     | function signature\n      @class        | a class constructor signature\n      @method       | instance method signature\n      @class_method | class method signature\n\n    - documentation items attributes:\n    \n      Attribute     | Desciption\n      --------------|--------------------------------------------------------\n      @is_a         | parent class name or list of\n      @short        | a short description on one line\n      @description  | a long description\n      @parameters   | list of pipelet, function, method parameters\n      @returns      | function or method returned value\n      @throws       | list of conditions that throw errors\n      @examples     | list of usage examples\n      @source       | expected source values' attributes\n      @emits        | emitted output values' attributes\n      @todo         | suggestion for future version\n      @coverage     | indicates automatic tests coverage\n      @manual       | a documentation manual name this belongs-to\n      @section      | a section name within a manual\n      @api          | indicates API maturity: experimental, alpha, beta, stable, deprecated\n  \n  - @@keyword: indicates a link to another section of the documentation:\n    - e.g. This is @@stateless, @@synchronous, @@lazy pipelet.\n    - If keyword is a plural and no entry is found for it, attempt to locate it's singular form\n    - If it cannot be located, a warning is emitted\n\n  - many pipelets, functions and methods are already following this documentation format\n\n- Transactional design patterns and pipelets\n  - fetch(): Fetches the current state of a store dataflow based on source events\n  - fetch_as(): Set attribute with fetched values from store by fetch_query\n  - fetch_first(): Set attribute with first fectched value from store\n  - update_fetched(): Update fetched values in a transaction\n  - fetched_differences(): Emits differences (A - B) between two fetched sets A then B\n  - emit_operations(): Emits remove, update, and add operations in a transaction\n\n- Hot server-code reloading:\n  - on required module file update\n  - migrating state reliably\n  - simple to implement:\n    - programmer describes disconnections from dataflows\n  - in a transaction allowing optimization of updated data using optimize()\n  - implemented in examples/teaser/data.js\n\n- Live page reload to ease developpement, implemented in examples.\n\n- Namespaces:\n  - allow local scoping of singletons, multitons, outputs\n\n- Pipelet._add_input() greatly simplifies adding additional inputs to pipelets.\n\n- Single page applications routing:\n  - using url_parse() and url_pattern()\n\n- DOM handling pipelets:\n  - $query_selector()\n  - $to_dom()\n  - $on()\n  - $window()\n  - window_size()\n  - $add_class()\n  - $has_class()\n  - $has_not_class()\n\n- Improve programming patterns using pipelet methods:\n  - Compose()\n  - Singleton()\n  - Multiton()\n  - through()\n  - set_output()\n  - output()\n  - remove_source_with()\n  - remove_destination_with()\n\n- Pipelets manipulating operations:\n  - store(): Store operations history\n  - creates(): Discard remove and update operations.\n  - deletes(): Only forward remove operations\n  - updates(): Only forward update operations\n  - adds(): filter add operations, useful when these have a strict Create semantic\n  - removes(): filter remove operations, useful when these have a strict Delete semantic\n  - revert(): transforms adds into removes, removes into adds, swap remove and add into updates\n  - query_updates(): Emit query updates from pipelet\n\n- Additional functional stateless pipelets:\n  - map(): allows to emit zero or 1 value for each source value\n  - flat_map(): allows to emit zero to n values for each source value\n  - pick()\n  - filter_pick()\n\n- Additional functional stateful pipelets:\n  - group(): emit grouped values into content attribute\n\n- Caches (stateful lazy pipelets):\n  - cache(): Stateful and lazy pipelet\n  - database_cache(): Cache for an entire database from schema\n\n- Server pipelets:\n  - require_pipeline(): Load, and auto-unload, a module exporting a Toubkal pipeline\n  - path_relative(): Alter path using node path.relative()\n  - path_join(): Prepend base directories to relative ```\"path\"```, resolve ```\"~\"``` to home directory\n  - Based on the piexif library, handling JPEG EXIF:\n    - piexif_parse(): Parses content EXIF using the piexifjs library\n    - piexif_insert(): Inserts EXIF into content, using the piexif library\n  - json_hide(): Hide attribute values in downstream pipelets using JSON.stringify()\n\nNew Pipelets or method              | Short Description\n------------------------------------|--------------------------------------------------------------\nssh_exec()                          | Executes ssh commands using pipelet child_process_exec()\nbuild_bundles()                     | Provides minified bundles from base/bundles.json configuration\nwww_files()                         | Provide all public static assets for web clients\ndirectory_entries()                 | @@multiton directories watcher from base directory\nall_directory_entries()             | @@multiton all sub-directories watcher from base directory\nfs_read_file()                      | Reads file into content property\nfs_write_file()                     | Writes file content property into file\nhandle_errors()                     | Handles errors for asynchronous map transform\nfs_stat()                           | Gets file or directory stat information\nchild_process_exec()                | Executes a command in a child process\nrename_properties()                 | Renames properties\nstore()                             | Store operations history\njson_hide()                         | Hide attribute values in downstream pipelets using JSON.stringify()\ndebug()                             | Conditional trace()\nsource_map_support_min()            | Provides minified asset for browser source map support\nremove_destination_with()           | Disconnect pipelet's first destination when source disconnects\nremove_source_with()                | Disconnect pipelet's input when source disconnects\npath_relative()                     | Alter path using node path.relative()\npath_join()                         | Prepend base directories to relative ```\"path\"```, resolve ```\"~\"``` to home directory\nrequire_pipeline()                  | Load, and auto-unload, a module exporting a Toubkal pipeline\nsocket_io_synchronizing()           | Emits start of synchronization events, collects synchronized events\nsocket_io_state_changes()           | Pipelet socket_io_server() connections' state changes\nfetched_differences()               | Emits differences (A - B) between two fetched sets A then B\nemit_operations()                   | Emits remove, update, and add operations in a transaction\nmodules_files()                     | Singleton dataflow of toubkal modules' files from toubkal/lib/modules.json\npiexif_insert()                     | Inserts EXIF into content, using the piexif library\npiexif_parse()                      | Parses content EXIF using the piexif library\nmarkdown()                          | Markdown to html converter using \"markdown-it\" and \"highlight.js\"\ndocumentation_manuals()             | Toubkal documentation manuals metadata (not content)\ndocumentation_markdown()            | Format documentation items into markdown\nparse_documentation()               | Emit documentation items from parsed \"comments\" attribute\nacorn()                             | Parse javascript \"content\" attribute using the acorn library\nprocess_variables()                 | Gets command line arguments, environment variables, and more\ndatabase_cache()                    | Cache for an entire database from schema\ncache()                             | Stateful and lazy pipelet\nquery_updates()                     | Emit query updates from pipelet\nfilter_pick()                       | Select source dataflow from matching parent dataflow values\ndelivers()                          | Specifies which upstream dataflows can be subscribed-to\n$has_not_class()                    | Emits source values which have a $node attriute without css_class set\n$has_class()                        | Emits source values which have a $node attriute with    css_class set\n$add_class()                        | Add css_class to $node attribute\nwindow_size()                       | Provide a dataflow of window size change events\n$window()                           | Singleton for the global Window\n$on()                               | Listen and emits DOM events registered using addEventListener()\nthrottle_last()                     | Emit last received source value before throttle events, without throttle events\nthrottle()                          | Emit last received source value before throttle events\nupdate_fetched()                    | Update fetched values in a transaction\nfetch()                             | Fetches the current state of a store dataflow based on source events\nfetch_as()                          | Set attribute with fetched values from store\nfetch_first()                       | Set attribute with first fectched value from store\npick()                              | Forwards only specified attributes\nhas_none()                          | Has one value if source has none, has no value if source has one\nlog_namespace()                     | Helps debug namespace issues\nnamespace()                         | Get namespace of current pipelet\nset_namespace()                     | Set namespace of current pipelet\ncreate_namespace()                  | Create child namespace of current pipelet or namespace\n$query_selector()                   | Emits a node if query selector found, used as a parameter to $to_dom()\noutput()                            | Retrieves a global reference for an output pipelet set by set_output()\nset_output()                        | Sets a global reference for an ouput pipelet\nrevert()                            | Revert operations, adds to removes, removes to adds, updates are swapped\nanimation_frames_polyfill()         | animation_frames() polyfilled\nurl_pattern()                       | Parses url for patterns such as /users/:id\nMultiton()                          | Creates multiton pipelets out of composition\nSingleton()                         | Creates singleton pipelets out of composition\nCompose()                           | Add boolean options single and singleton\n$to_dom()                           | Updates DOM from dataflow for a DOM node, using render function, creating container child\nthrough()                           | Getting dataflow through another pipelet (implemented a Pipelet method)\nnot_exists()                        | Existence validation for dataflow adds (no-exists), removes and updates (exists)\nfetch_flow_key()                    | Fetch dataflow key from flows metadata\ncreates()                           | Discard remove and update operations.\ndeletes()                           | Only forward remove operations\nupdates()                           | Only forward update operations\nadds()                              | Selects \"add\" operations only, operations that create objects\nremoves()                           | Selects \"remove\" operations only, operations that remove objects\ngroup()                             | Group input values by function into content attribute\nmap()                               | Maps input values to function returning a single value or falsy\nflat_map()                          | Maps input values to function returning an Array of output values\n\n### Version 0.3.0 - Authentication \u0026\u0026 Authorizations, Persistance - November 12th 2015\n\nAllows to build complex streaming applications with Social Authentication and MySQL persistance.\n\nPipelet API has been significantly refactored and getting close to version 1.0 API.\n\n- 1486 continuous integration tests\n\n- Reactive Authentication with Passport:\n  - passport_profiles(): authenticated user profiles dataflow\n  - Provider credentials from configuration file\n  - Strategies initialization\n  - Strategies routes from initialized strategies\n  - Integration with socket_io_clients()\n  - Express session support:\n    - Providing a session_store() dataflow\n    - Providing a Express_Session_Store(), an Express session store\n    - Getting users form session_store() with passport_user_sessions()\n    - In serve() to allow to set session when serving static assets\n    - In socket_io_clients() to retrieve session id\n  - Multi-provider Sign-in Widget example from initialized strategies\n  - Reactive Authorizations example from session id and session_store()\n\n- Web Storage API:\n  - local_storage( flow ) pipelet\n  - multiton (one singleton per dataflow )\n  - supports both localStorage and sessionStorage via storage option\n  - listens to other pages \"storage\" events to update in realtime dataflow\n\n- MySQL read/write dataflows pipelets:\n  - mysql(): provides read/write dataflows for MySQL table, uses lower-level pipelets:\n    - mysql_connections(): manages MySQL connections\n    \n    - mysql_read(): statelessly read MySQL table:\n      - Builds SELECT .. WHERE from downstream query\n    \n    - mysql_write(): statelessly writes (DELETE and INSERT) to MySQL table\n    \n    - configuration(): to retrieve MySQL user credentials, removing credentials from code\n\n- react(): Facebook React client-side pipelet\n\n- beat(): a pipelet to emit events at time intervals\n- once(): a pipelet to emit a single event after a timeout\n- next(): a pipelet to maintain auto-incremented attributes on trigger\n\n- Operations Optimizer:\n  - waits for transactions to complete to emit operations\n  - on transaction completion, emits optimized operations, i.e. the minimum set of adds, removes and updates\n\n- join() pipelet:\n  - Adding exhaustive tests suites (517 tests total)\n  - Finalize outer joins (left, right, full) with transactions handling\n  - Implement dynamic filters on inner and right join to reduce data pull\n  - Optmizations: stateless output\n  - Refactoring and documentation\n\n- Examples:\n  - Examples server to serve assets and data for client examples, using configuration files for datasets\n  - Teaser using socket_io_server(), flow(), filter(), order(), trace(), and table()\n  - Reactive user table with react()\n  - Chat using socket_io_server() and form()\n\n- Refactor / stabilize pipelet API\n\n- Error handling using 'error' dataflow\n  - Error values have the following attributes:\n    - flow (String): the string 'error'\n    - code (String): the error code\n    - message (String): an optional error message\n    - operation (String): 'add' or 'remove', the operation that caused the error\n    - error_flow (String): the flow for which the error occurred\n    - sender (String): an identification of the sender of the operation to allow\n      routing back to sender. Sender's valued comes from operations option 'sender'\n    - values (Array of Objects): the values for which the error occurred\n  \n  - Allows downstream pipelets to handle errors by reverting failed operations:\n    - a failed add is reverted by remove\n    - a failed remove is reverted by add\n    - errors can be routed back to sender using filters for the flow and sender attributes\n\n  - Error dataflow is handled specifically by some core pipelets:\n    - set_flow() does not alter the flow attribute when set to 'error'\n    - unique() forwards incomming errors to underlying set\n  \n- Refactor modules, packaging:\n  - use undefine() to load modules from any situation, node, browser, AMD loaders\n  - split into meaningful subdirectories to eventually split the project into subprojects\n\n- Concurrent Transactions Synchronization with branch tags\n\n- Refactor pipelet class model:\n  - Ease the definition of multiple, semantically distinct, inputs and outputs without definining pipelets\n  - Define Plug base class for:\n    - Inputs (Pipelet.Input)\n    - Outputs (Pipelet.Output)\n  - Pipelet.Input class:\n    - Receives data events from upstream to be processed by pipelet\n    - Provides upstream transactions management\n    - Provides methods for connectivity to source outputs\n    - Fetches upstream data as requested by pipelet\n    - Determines lazyness\n    - Emits upstream query updates\n  - Pipelet.Output class:\n    - Provides methods for connectivity to destination inputs\n    - Fetches \n    - Fetches and filters pipelet's state\n    - Emits data events to downstream inputs using query trees\n    - Emits transaction events\n    - Manages output transactions\n    - Updates query tree from downstream query updates\n    - Propagates query updates to pipelet\n    \n  - Pipelet modified class:\n    - Manages state\n    - Defaults remains stateless (i.e. uses altered upstream state)\n  \n  - RS.Options object defined methods for manipulating operations' options:\n    - forward(): returns an options Objects with options that must be forwarded\n    - has_more(): returns truly if there is an incomplete transaction\n\nNew Pipelets                        | Short Description\n------------------------------------|--------------------------------------------------------------------------------------\npassport_profiles()                 | Manages Passport authenticated user profiles\npassport_strategies_configuration() | Watch configuration file for passport strategies\npassport_strategies()               | Initialize Passport strategies\npassport_strategies_routes()        | Updates strategies routes from initialized strategies\nexpress_route()                     | Reactive express routes middleware\nsession_store()                     | A session store implemented as a bidirectional dataflow\npassport_user_sessions()            | Get Passport authenticated users from session_store() and session id\ncontent_order()                     | Orders the array of values contained in a content attribute\ncontent_sort()                      | Sorts the array of values contained in a content attribute\ncontent_transform()                 | Modifies content attribute using a transform()\ncontent_route()                     | Add a route and express middleware handler for a content\nvalues_to_attribute()               | Embed input values under a content attribute of a single output value.\nbeat()                              | Emit events at time intervals\nonce()                              | Emit an event on timeout\nlocal_storage()                     | Implements Web Storage API\nnext()                              | A pipelet to maintain auto-incremented attributes on trigger\nmysql()                             | In toubkal_mysql repository, provides read/write dataflows to/from MySQL tables\noptimize()                          | On complete, remove unnecessary adds, removes, updates, emit updates when possible\nhtml_serialize()                    | Serialize DOM tree generated by html_parse()\nhtml_parse()                        | Parse html content to htmlparser2 DOM tree\nreact_table()                       | Reactive table rows and columns implemented using react()\nreact()                             | Transform a full set to a DOM widget using a Facebook React render function\nhttp_listen()                       | Listen to http servers, allows to get the 'listening' event (used by socket.io 0.9 for its garbage collector)\nvirtual_http_servers()              | Allows to run many frameworks and socket.io servers virtual hosts\nserve_http_servers()                | Bind http event handlers to HTTP_Router()\ngreedy()                            | A non-lazy stateless pipelet\nmake_base_directories()             | Create base directories for a dataset of file paths\n\nOther Classes \u0026\u0026 methods  | Short Description\n--------------------------|---------------------------------------------------------------------------------------------------------\nExpress_Session_Store()   | An Express session store using a bidirectional dataflow as the underlying store\nvalue_equals()            | Allows comparison of any types of values, including deeply nested objects and arrays\nundefine()                | Universal module loader for node and the browser, exported as own npm module\nLap_Timer                 | Helps calculate time difference between events, used by loggers\nConsole_Logger            | Logger to console.log() with timestamps and lap lines \nClient assets             | Sets to ease assembly of minified files for clients\nHTTP_Router               | Efficiently route HTTP requests using base URLs\nLazy_Logger               | Logger controlled by queries using '\u003c=' operator\nQuery_Error               | Custom Error class for Queries\nQuery.Evaluation_Context  | Evaluation context for complex query expressions\nQuery.evaluate()          | Query class method to evaluate complex query expressions\nQuery.Operator()          | Adds a Query expression operator\nQuery.fail                | Failure value for Query expressions\nPlug                      | Base class for Input and Output plugs\nPipelet.Input             | Pipelet default Input plug\nPipelet.Output            | Base Output plug\nControllet.Input          | Input plug for controllets\nControllet.Output         | Output plug for controllets\nUnion.Input               | Input plug for Union (allows many sources)\nUnion.Output              | Output plug for Union\nSet.Output                | Output plug for Set\nIO_Transactions           | Base class for Input_Transactions and Output_Transactions\nInput_Transactions        | Concurrent Transactions Synchronization at Inputs\nOutput_Transactions       | Concurrent Transactions Synchronization at Outputs\nIO_Transaction            | Base class for Input_Transaction and Output_Transaction\nInput_Transaction         | Manage an input transaction\nOutput_Transaction        | Manage an output transaction\n\n### Version 0.2.0 - Publish / Subscribe Dataflow Model - March 31 2014:\n\n- Finalize Publish / Subscribe reactive dataflow model using optimized Query Tree Router and lazy connection of stateless pipelets\n- Filter support for static and dynamic queries\n- Transactions\n- Automate UI tests on Travis\n- 309 continuous integration tests\n- Controllets which control upstream query trees using downstream queries\n- Improve Pipelet API and naming conventions\n- Virtual Hosts w/ optimized routing\n- Touch Events on bootstrap pipelets\n\nNew Pipelets              | Short Description\n--------------------------|------------------------------------------------\nwatch_directories()       | Updated when entries in directories are updated\nurl_events()              | Browser url changes\nanimation_frames()        | Request Animation Frame events\nencapsulate()             | Hide a graph of pipelets behind one pipelet\nrequire_resolve()         | Resolve node module files absolute path\ntimestamp()               | Add timestamp attribute\nevents_metadata()         | Add events metadata attributes\nauto_increment()          | Add auto-increment attribute\nset_flow()                | Add flow attribute\nto_uri()                  | Transforms a relative file name into a DOM uri\nthumbnails()              | Image thumbnails using ImageMagick\nload_images()             | Load images in the DOM one at a time\nbootstrap_carousel()      | Bootstrap responsive images carousel \nbootstrap_photos_matrix() | Bootstrap responsive photo matrix\nbootstrap_photo_album()   | Bootstrap responsive photo album\njson_stringify()          | JSON Stringifies content attribute\njson_parse()              | JSON parse content attribute\nattribute_to_value()      | Replace value with the value of an attribute\nvalue_to_attribute()      | Sets value as an attribute and add other default attributes\n\n### Version 0.1.0 - April 8th 2013:\n\n#### Features:\n\n- Push reactive dataflow model with lazy evaluation of stateless pipelets\n- Core Database engine with order / aggregates / join / union, and more\n- Automated tests\n- Dataflows between clients and server using socket.io\n- DOM Tables w/ realtime updates\n- DOM Controls as dataflows: Drop-Down / Radio / Checkboxes\n- DOM Forms with client-side and server-side validation\n- Realtime Minification using Uglify w/ source maps\n- HTTP(s) servers\n- File watch w/ realtime updates\n- JSON Configuration Files w/ realtime updates\n\nCore Pipelets             | Short Description\n--------------------------|------------------------------------------------\nset()                     | Base stateful pipelet\nunique()                  | Set of unique values, discarding duplicates\nfilter()                  | Filters a dataflow\norder()                   | Order a set\nordered()                 | Follow an ordered set (typically derived)\naggregate()               | Aggregates measures along dimensions (GROUP BY)\njoin()                    | Joins two dataflows\nwatch()                   | Dataflow updated on file content changes\ndispatch()                | Dispatches dataflows to a dataflow of branches\nparse_JSON()              | JSON dataflow to parsed JSON dataflow\n\nServer Pipelets           | Short Description\n--------------------------|------------------------------------------------\nuglify()                  | Minifies a dataflow of files into a bundle, using [Uglify JS 2](https://github.com/mishoo/UglifyJS2)\nhttp_servers()            | A dataflow of http servers\nserve()                   | Serve a dataflow of resources contents to http (or other) servers\nsocket_io_clients()       | A dataflow server for socket.io clients\nsocket_io_server()        | A dataflow client for socket.io server\nsend_mail()               | Send emails from email dataflow\nconfiguration()           | Dataflow of application configuration parameters\n\nDOM Pipelets              | Short Description\n--------------------------|------------------------------------------------\ntable()                   | DOM table bound to incoming dataflows\nform()                    | DOM Form using fields dataflow, emiting submited forms\nform_validate()           | Client and server form validation\ncheckbox()                | DOM input checkbox\ncheckbox_group()          | DOM input chexbox group\nradio()                   | DOM radio button\ndrop_down()               | DOM drop-down menu\n\nEC2 Pipelets              | Short Description\n--------------------------|------------------------------------------------\nec2_regions()             | Set of AWS EC2 regions, starts ec2 clients\n\n## Licence\n\n    Copyright (c) 2013-2017, Reactive Sets\n\n    This program is free software: you can redistribute it and/or modify\n    it under the terms of the GNU Affero General Public License as\n    published by the Free Software Foundation, either version 3 of the\n    License, or (at your option) any later version.\n\n    This program is distributed in the hope that it will be useful,\n    but WITHOUT ANY WARRANTY; without even the implied warranty of\n    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n    GNU Affero General Public License for more details.\n\n    You should have received a copy of the GNU Affero General Public License\n    along with this program.  If not, see \u003chttp://www.gnu.org/licenses/\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freactivesets%2Ftoubkal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freactivesets%2Ftoubkal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freactivesets%2Ftoubkal/lists"}