Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/metwork-framework/mfdata
metwork/mfdata module
https://github.com/metwork-framework/mfdata
autoreadme2 github-actions integration-level-5
Last synced: 9 days ago
JSON representation
metwork/mfdata module
- Host: GitHub
- URL: https://github.com/metwork-framework/mfdata
- Owner: metwork-framework
- License: bsd-3-clause
- Created: 2018-09-20T12:03:44.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2024-10-27T21:34:03.000Z (23 days ago)
- Last Synced: 2024-10-28T01:25:47.134Z (23 days ago)
- Topics: autoreadme2, github-actions, integration-level-5
- Language: Python
- Homepage: http://metwork-framework.org/
- Size: 3.11 MB
- Stars: 6
- Watchers: 3
- Forks: 4
- Open Issues: 18
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG-0.4.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
[](http://www.metwork-framework.org)
# mfdata
[//]: # (automatically generated from https://github.com/metwork-framework/github_organization_management/blob/master/common_files/README.md)
**Status (master branch)**
[](https://github.com/metwork-framework/mfdata/actions?query=workflow%3ACI+branch%3Amaster)
[](https://github.com/metwork-framework/resources/blob/master/badges/maintained.svg)
[]()
[](https://gitter.im/metwork-framework/community-en?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
[](https://gitter.im/metwork-framework/community-fr?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
## What is MetWork FrameWork?
[MetWork Framework](https://metwork-framework.org) is an opensource system
for building and managing production grade applications or micro-services
(from development to operations).
## What is it?
This is the **M**etwork **F**ramework **DATA** processing **module**. This module is a "production ready" framework
for processing "incoming files".
*Incoming files* can arrive by:
- system FTP
- system SCP
- Restful HTTP
- MQTT
- AMQP
- [...] (you can provide a "listener plugin" to support other things)
These "incoming files" are transient. The goal in to do something with them
**in the stream**.
To do that, you can easily code and provide **plugins** to build a
workflow. This workflow can be really easy and stupid (for example: do something
for every incoming files) but also really complex with hundreds of plugins, routing rules,
plugin sequences, file duplications...
To help with complex workflows, you can rely on :
- "loose coupling" and "dynamic business routing rules" between plugins thanks to the provided **switch plugin**
- "context transmission" thanks to the **xattrfile** library
- "full inspection support" in case of errors thanks to **failure policy**, **xattrfile** libraries and **archive plugins**
- "retry mechanisms" thanks to **failure policy** and **reinject plugins**
- "full monitoring" thanks to embedded metrics and logs management (and **mfadmin module**)
It's designed to process terrabytes of data every day in a very robust way:
- you can choose the parallelism level of each plugin with a configuration file
- plugins are automatically restarted if necessary
- "waiting queues" and "resources limits" protect the system for overloading
Last but not least, thanks to the "no polling at all" principle, you can process
several thousands of file by second.
## What is not in?
Of course, you can code the behavior you want but **mfdata module** is
mainly designed to be "stateless". If you want to store durably something,
have a look at **mfbase module**.
## Cheatsheet
A cheatsheet for this module is available [here](https://metwork-framework.org/pub/metwork/continuous_integration/docs/master/mfdata/800-cheatsheet/)
## Reference documentation
- (for **master (development)** version), see [this dedicated site](http://metwork-framework.org/pub/metwork/continuous_integration/docs/master/mfdata/) for reference documentation.
- (for **latest released stable** version), see [this dedicated site](http://metwork-framework.org/pub/metwork/releases/docs/stable/mfdata/) for reference documentation.
For very specific use cases, you might be interested in
[reference documentation for integration branch](http://metwork-framework.org/pub/metwork/continuous_integration/docs/integration/mfdata/).
And if you are looking for an old released version, you can search [here](http://metwork-framework.org/pub/metwork/releases/docs/).
## Installation guide
See [this document](https://metwork-framework.org/pub/metwork/continuous_integration/docs/master/mfdata/100-installation_guide/).
## Configuration guide
See [this document](https://metwork-framework.org/pub/metwork/continuous_integration/docs/master/mfdata/300-configuration_guide/).
## Contributing guide
See [CONTRIBUTING.md](CONTRIBUTING.md) file.
## Code of Conduct
See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) file.
## Sponsors
*(If you are officially paid to work on MetWork Framework, please contact us to add your company logo here!)*
[](http://www.meteofrance.com)