{"id":13506019,"url":"https://github.com/telefonicaid/fiware-cygnus","last_synced_at":"2026-02-04T12:11:01.160Z","repository":{"id":13909051,"uuid":"16607947","full_name":"telefonicaid/fiware-cygnus","owner":"telefonicaid","description":"      A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context","archived":false,"fork":false,"pushed_at":"2024-10-29T14:20:34.000Z","size":17146,"stargazers_count":64,"open_issues_count":199,"forks_count":105,"subscribers_count":24,"default_branch":"master","last_synced_at":"2024-10-29T14:58:40.058Z","etag":null,"topics":["fiware","fiware-cygnus","historical-data"],"latest_commit_sha":null,"homepage":"https://fiware-cygnus.rtfd.io/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/telefonicaid.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES_NEXT_RELEASE","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-02-07T07:57:41.000Z","updated_at":"2024-10-29T12:32:36.000Z","dependencies_parsed_at":"2023-10-20T23:21:45.528Z","dependency_job_id":"93c11c9a-96c9-43a3-87f6-f0ec1866ee6c","html_url":"https://github.com/telefonicaid/fiware-cygnus","commit_stats":null,"previous_names":[],"tags_count":105,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telefonicaid%2Ffiware-cygnus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telefonicaid%2Ffiware-cygnus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telefonicaid%2Ffiware-cygnus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telefonicaid%2Ffiware-cygnus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/telefonicaid","download_url":"https://codeload.github.com/telefonicaid/fiware-cygnus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247142065,"owners_count":20890652,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fiware","fiware-cygnus","historical-data"],"created_at":"2024-08-01T01:00:32.948Z","updated_at":"2026-02-04T12:11:01.154Z","avatar_url":"https://github.com/telefonicaid.png","language":"Java","funding_links":[],"categories":["FIWARE Catalogue"],"sub_categories":["Core Context Management"],"readme":"# Cygnus\n\n[![](https://fiware.github.io/catalogue/badges/chapters/core.svg)](https://www.fiware.org/developers/catalogue/)\n[![License](https://img.shields.io/github/license/telefonicaid/fiware-cygnus.svg)](https://opensource.org/licenses/AGPL-3.0)\n[![Quay badge](https://img.shields.io/badge/quay.io-fiware%2Fcygnus--ngsi-grey?logo=red%20hat\u0026labelColor=EE0000)](https://quay.io/repository/fiware/cygnus-ngsi)\n[![Docker badge](https://img.shields.io/badge/docker-telefonicaiot%2Ffiware--cygnus-blue?logo=docker)](https://hub.docker.com/r/telefonicaiot/fiware-cygnus)\n[![](https://img.shields.io/badge/tag-fiware--cygnus-orange.svg?logo=stackoverflow)](http://stackoverflow.com/questions/tagged/fiware-cygnus)\n[![Support badge]( https://img.shields.io/badge/support-askbot-yellowgreen.svg)](https://ask.fiware.org/questions/scope%3Aall/tags%3Acygnus/)\n\u003cbr/\u003e\n[![Documentation badge](https://readthedocs.org/projects/fiware-cygnus/badge/?version=latest)](http://fiware-cygnus.rtfd.io)\n[![CI](https://github.com/telefonicaid/fiware-cygnus/workflows/CI/badge.svg)](https://github.com/telefonicaid/fiware-cygnus/actions?query=workflow%3ACI)\n![Status](https://fiware.github.io/catalogue/badges/statuses/status-deprecated.svg)\n[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4851/badge)](https://bestpractices.coreinfrastructure.org/projects/4851)\n\n**This repository is now archived. Cygnus uses an unsupported framework (Apache Flume), and further development is not envisaged. Users should upgrade to the newer Kafka-based [Kafnus](https://github.com/telefonicaid/kafnus) component instead**\n\nCygnus is a connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context. Internally, Cygnus is based on [Apache Flume](http://flume.apache.org/), Flume is a data flow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. It was built to automate the flow of data between systems. While the term 'dataflow' can be used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems.\n\nEach data persistence agent within Cygnus is composed of three parts - a listener or source in charge of receiving the data, a channel where the source puts the data once it has been transformed into a Flume event, and a sink, which takes Flume events from the channel in order to persist the data within its body into a third-party storage.\n\nWhilst active, this project was part of [FIWARE](https://www.fiware.org/). For more information\ncheck the FIWARE Catalogue entry for the\n[Core Context Management](https://github.com/Fiware/catalogue/tree/master/core).\n\n\n| :books: [Documentation](https://fiware-cygnus.rtfd.io)  | :mortar_board: [Academy](https://fiware-academy.readthedocs.io/en/latest/core/cygnus) | \u003cimg style=\"height:1em\" src=\"https://quay.io/static/img/quay_favicon.png\"/\u003e [quay.io](https://quay.io/repository/fiware/cygnus-ngsi)| :whale: [Docker Hub](https://hub.docker.com/r/telefonicaiot/fiware-cygnus) | :dart: [Roadmap](https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/roadmap.md) |\n|---|---|---|---|---|\n\n## Contents\n\n-   [Background](#background)\n-   [Install](#install)\n-   [Roadmap](#roadmap)\n-   [Further Documentation](#further-documentation)\n-   [License](#license)\n\n# Background\n\nInternally, Cygnus is based on [Apache Flume](http://flume.apache.org/), a technology addressing the design and execution of data collection and persistence \u003ci\u003eagents\u003c/i\u003e. An agent is basically composed of a listener or source in charge of receiving the data, a channel where the source puts the data once it has been transformed into a Flume event, and a sink, which takes Flume events from the channel in order to persist the data within its body into a third-party storage.\n\nCygnus is designed to run a specific Flume agent per source of data.\n\nCurrent stable release is able to persist the following sources of data in the following third-party storages:\n\n* NGSI-like context data in:\n    * [HDFS](http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html), the [Hadoop](http://hadoop.apache.org/) distributed file system.\n    * [MySQL](https://www.mysql.com/), the well-known relational database manager.\n    * [CKAN](http://ckan.org/), an Open Data platform.\n    * [MongoDB](https://www.mongodb.org/), the NoSQL document-oriented database.\n    * [STH Comet](https://github.com/telefonicaid/fiware-sth-comet), a Short-Term Historic database built on top of MongoDB.\n    * [Kafka](http://kafka.apache.org/), the publish-subscribe messaging broker.\n    * [DynamoDB](https://aws.amazon.com/dynamodb/), a cloud-based NoSQL database by [Amazon Web Services](https://aws.amazon.com/).\n    * [PostgreSQL](http://www.postgresql.org/), the well-known relational database manager.\n    * [Carto](https://carto.com/), the database specialized in geolocated data.\n    * [PostGIS](http://postgis.net/), a spatial database extender for PostgreSQL object-relational database.\n    * [Orion](https://github.com/telefonicaid/fiware-orion), the FIWARE Context Broker.\n    * [Elasticsearch](https://www.elastic.co/products/elasticsearch), the distributed full-text search engine with JSON documents.\n    * [Arcgis](https://www.arcgis.com/home/index.html), the Arcgis is a geographic information system (GIS).\n* Twitter data in:\n    * [HDFS](http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html), the [Hadoop](http://hadoop.apache.org/) distributed file system.\n\n**IMPORTANT NOTE**: for the time being, `cygnus-ngsi`, `cygnus-twitter` and `cygnus-ngsi-ld` agents cannot be installed in the same base path, because of an incompatibility with the required version of the `httpclient` library. Of course, if you are going to use just one of the agents, there is no problem at all.\n\ncygnus-ngsi: [![Docker badge](https://img.shields.io/docker/pulls/fiware/cygnus-ngsi.svg)](https://hub.docker.com/r/fiware/cygnus-ngsi/)\u003cbr/\u003e\ncygus-twitter : [![Docker badge](https://img.shields.io/docker/pulls/fiware/cygnus-twitter.svg)](https://hub.docker.com/r/fiware/cygnus-twitter/)\u003cbr/\u003e\ncygus-ngsi-ld : [![Docker badge](https://img.shields.io/docker/pulls/fiware/cygnus-ngsi-ld.svg)](https://hub.docker.com/r/fiware/cygnus-ngsi-ld/)\n\n## Cygnus place in FIWARE architecture\nCygnus (more specifically, cygnus-ngsi agent) plays the role of a connector between Orion Context Broker (which is a NGSI source of data) and many FIWARE storages such as CKAN, Cosmos Big Data (Hadoop) and STH Comet. Of course, as previously said, you may add MySQL, Kafka, Carto, etc as other non FIWARE storages to the FIWARE architecture.\n\n![FIWARE architecture](doc/images/fiware_architecture.png)\n\n## Install\nFiware/Cygnus has four sub-modules [cygnus-common](cygnus-common#welcome-to-cygnus-common), [cygnus-ngsi](cygnus-ngsi#welcome-to-cygnus-ngsi) and [cygnus-twitter](cygnus-twitter#welcome-to-cygnus-twitter), [cygnus-ngsi-ld](cygnus-ngsi-ld#welcome-to-cygnus-ngsi). Information about how to install these modules can be found at the corresponding section of Installation.\n\nFollowing are the links of install section of Fiware/Cygnus sub-modules:\n* cygnus-common:\n  - [Install from sources](doc/cygnus-common/installation_and_administration_guide/install_from_sources.md)\n  - [Install with docker](doc/cygnus-common/installation_and_administration_guide/install_with_docker.md)  \n* cygnus-ngsi:\n  - [Install from sources](doc/cygnus-ngsi/installation_and_administration_guide/install_from_sources.md)\n  - [Install with docker](doc/cygnus-ngsi/installation_and_administration_guide/install_with_docker.md)\n\n* cygnus-twitter:\n  - [Install from sources](doc/cygnus-twitter/installation_and_administration_guide/install_from_sources.md)\n  - [Install with docker](doc/cygnus-twitter/installation_and_administration_guide/install_with_docker.md)\n* cygnus-ngsi-ld:\n  - [Install from sources](doc/cygnus-ngsi-ld/installation_and_administration_guide/install_from_sources.md)\n  - [Install with docker](doc/cygnus-ngsi-ld/installation_and_administration_guide/install_with_docker.md)\n## Roadmap\n\nThe roadmap of this FIWARE GE is described [here](doc/roadmap.md).\n\n## Further documentation\nThe per agent **Quick Start Guide** found at readthedocs.org provides a good documentation summary ([cygnus-ngsi](http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/quick_start_guide/index.html), [cygnus-twitter](http://fiware-cygnus.readthedocs.io/en/latest/cygnus-twitter/quick_start_guide/index.html)).\n\nNevertheless, both the **Installation and Administration Guide** and the **User and Programmer Guide** for each agent also found at [readthedocs.org](http://fiware-cygnus.readthedocs.io/en/latest/) cover more advanced topics.\n\nThe per agent **Flume Extensions Catalogue** completes the available documentation for Cygnus ([cygnus-ngsi](http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/flume_extensions_catalogue/introduction/index.html), [cygnus-twitter](http://fiware-cygnus.readthedocs.io/en/latest/cygnus-twitter/flume_extensions_catalogue/introduction/index.html)).\n\nOther interesting links are:\n\n* Our [Apiary Documentation](http://telefonicaid.github.io/fiware-cygnus/api/latest) if you want to know how to use our API methods for Cygnus.\n* [cygnus-ngsi](doc/cygnus-ngsi/integration) **integration** examples .\n* [cygnus-ngsi](https://edu.fiware.org/mod/resource/view.php?id=1037) **introductory course** in FIWARE Academy.\n* The [Contributing Guidelines](doc/contributing/contributing_guidelines.md) if your aim is to extend Cygnus.\n\n### Reporting issues and contact information\nAny doubt you may have, please refer to the [Cygnus Core Team](./reporting_issues_and_contact.md).\n\n---\n\n## License\n\nCygnus is licensed under Affero General Public License (GPL)\nversion 3. You can find a [copy of this license in the repository](./LICENSE).\n\n© 2023 Telefonica Investigación y Desarrollo, S.A.U\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eFurther information on the use of the AGPL open source license\u003c/strong\u003e\u003c/summary\u003e\n\n### Are there any legal issues with AGPL 3.0? Is it safe for me to use?\n\nThere is absolutely no problem in using a product licensed under AGPL 3.0. Issues with GPL\n(or AGPL) licenses are mostly related with the fact that different people assign different\ninterpretations on the meaning of the term “derivate work” used in these licenses. Due to this,\nsome people believe that there is a risk in just _using_ software under GPL or AGPL licenses\n(even without _modifying_ it).\n\nFor the avoidance of doubt, the owners of this software licensed under an AGPL 3.0 license\nwish to make a clarifying public statement as follows:\n\n\u003e Please note that software derived as a result of modifying the source code of this\n\u003e software in order to fix a bug or incorporate enhancements is considered a derivative\n\u003e work of the product. Software that merely uses or aggregates (i.e. links to) an otherwise\n\u003e unmodified version of existing software is not considered a derivative work, and therefore\n\u003e it does not need to be released as under the same license, or even released as open source.\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftelefonicaid%2Ffiware-cygnus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftelefonicaid%2Ffiware-cygnus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftelefonicaid%2Ffiware-cygnus/lists"}