{"id":22868492,"url":"https://github.com/datavenueliveobjects/pushing-data-to-aws-simple-queue-service-sqs","last_synced_at":"2026-03-15T08:41:09.063Z","repository":{"id":39635128,"uuid":"212615351","full_name":"DatavenueLiveObjects/Pushing-data-to-AWS-Simple-Queue-Service-SQS","owner":"DatavenueLiveObjects","description":"This project is intended for Live Objects users wishing to explore integration patterns with AWS and for organizations already running business logic on AWS planning to work on events from IoT devices sourced via Live Objects.","archived":false,"fork":false,"pushed_at":"2024-07-04T13:37:14.000Z","size":917,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-06T15:53:31.054Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DatavenueLiveObjects.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-03T15:36:09.000Z","updated_at":"2024-07-04T13:37:16.000Z","dependencies_parsed_at":"2024-03-07T10:50:31.157Z","dependency_job_id":"d4a67db6-5c7f-48cd-84e1-71e31a26d123","html_url":"https://github.com/DatavenueLiveObjects/Pushing-data-to-AWS-Simple-Queue-Service-SQS","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DatavenueLiveObjects%2FPushing-data-to-AWS-Simple-Queue-Service-SQS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DatavenueLiveObjects%2FPushing-data-to-AWS-Simple-Queue-Service-SQS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DatavenueLiveObjects%2FPushing-data-to-AWS-Simple-Queue-Service-SQS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DatavenueLiveObjects%2FPushing-data-to-AWS-Simple-Queue-Service-SQS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DatavenueLiveObjects","download_url":"https://codeload.github.com/DatavenueLiveObjects/Pushing-data-to-AWS-Simple-Queue-Service-SQS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246458022,"owners_count":20780675,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-13T12:35:22.538Z","updated_at":"2025-10-15T12:22:34.154Z","avatar_url":"https://github.com/DatavenueLiveObjects.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mqtt2SQS\n\n## Table of contents\n* [Introduction](#introduction)\n* [Connector](#connector)\n  * [Performance and scalability](#performance-and-scalability)\n* [Technologies](#technologies)\n* [Requirements](#requirements)\n* [Getting the installation package](#getting-the-installation-package)\n* [Installation](#installation)\n* [Configuration](#configuration)\n  * [Live objects side](#live-objects-side)\n    * [Generate Live Objects API key](#generate-live-objects-api-key)\n  * [AWS Side](#aws-side)\n    * [Create AWS Access key](#create-aws-access-key)\n    * [Use AWS Access keys](#use-aws-access-keys)\n    * [Create AWS SQS queue](#create-aws-sqs-queue)\n  * [Connector side](#connector-side)\n    * [Application](#application)\n    * [Logging](#logging)\n* [Launching](#launching)\n  * [Launching on local machine](#launching-on-local-machine)\n  * [Installation on AWS machine](#installation-on-aws-machine)\n    * [App deployment](#app-deployment)\n  * [Start receiving messages](#start-receiving-messages)\n  * [Metrics](#metrics)\n  * [Health Check](#health-check)\n\n## Introduction\n\nThis project contains connector 'Live Objects MQTT to AWS Simple Queue Service (SQS)' and was designed to read events from Live Objects MQTT and pushing them to AWS Simple Queue Service (SQS).  \n\n\nThis project is intended for Live Objects users wishing to explore integration patterns with AWS and for organizations already running business logic on AWS planning to work on events from IoT devices sourced via Live Objects.\n\n\n## Connector\n\nThe connector (Mqtt2Sqs) subscribes to selected Live Objects MQTT queue, reads all events and publishes them to selected AWS SQS without any modification to events’ contents. It is intended to be run as a long-running process hosted on AWS. Connector code is written in Java, using the Spring Boot framework.\n\nMqtt2Sqs supports only the communication from Live Objects i.e. it reads messages send from IoT devices. Communication towards devices (sending commands to devices) is not supported.\n\nProvisioning of IoT devices is within the scope of Live Objects; Mqtt2Sqs has no knowledge on what devices are communicating; it is just transparently moving messages from Live Objects to AWS SQS. It is assumed that business logic acting on those messages is to be applied by applications consuming the messages from SQS.\n\n![architecture](./images/architecture.png)\n\n### Performance and scalability\n\nThe software acts as an integration demonstrator. Mqtt2Sqs comes without any guarantees related to percentage of messages successfully written to SQS, nor to the response time. Moreover, the ordering of messages is not guaranteed to be preserved; the application uses thread pools to run its MQTT and Event Hub adapters which may cause some messages to arrive in SQS out of order in which they were kept within Live Objects’ MQTT queue.\n\nTested on AWS, the connector was processing events at the rate of ~60,000 messages per minute. This should not be treated as a guarantee either, since the throughput depends i.a. on message size and number of applications shared within SQS.\n\nLive Objects platform supports load balancing between multiple MQTT subscribers. It is possible to run multiple instances of Mqtt2SQS, each of them will handle its own subset of messages.\n\n## Technologies\n* Java 8\n* Spring Boot 2.3.2\n\n\n## Requirements\nIn order to run the connector you need to have:\n* **Live Objects account**\n  * API key which can access the queue\n  * MQTT fifo queue (both MQTT fifo queue and API key generation is described in the [user guide](https://liveobjects.orange-business.com/#/cms/ressources-guide-utilisateur/))\n* **AWS account** \n  * Access keys (access key ID and secret access key) to AWS Management Console (see the [documentation](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_create.html))\n  * SQS set up (creation process is described in official [documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-setting-up.html))\n  \n* **Development tools (only when building the package)**\n   * Java and supporting IDE (e.g. IntelliJ, Eclipse)\n   * Apache Maven\n\n## Getting the installation package\n\nThe installation package can be acquired in one of two ways:\n\n### Release\n\nIt can be downloaded from https://github.com/DatavenueLiveObjects/Pushing-data-to-AWS-Simple-Queue-Service-SQS/releases\n\n### Building\n\nIt can be created by running the command:\n```\nmvn clean package -Prelease\n```\nAfter running this command, the file  `mqtt2sqs-[VERSION].zip` will be created in the target directory. \n\n## Installation\n\nThe file downloaded/created above should be placed where the connector will be started, and then unpacked. You can deploy this connector wherever you want (local server, cloud provider etc.).\n\nAfter unpacking the archive, you should get a structure similar to this:\n```\nbin/\nconf/\ndata/\nlib/\n```\n\n## Configuration\n\nTo start synchronization between Live Objects and AWS SQS, you need to configure Live Objects API key, Live Objects FIFO queue and AWS SQS queue URL in the `application.yaml` file.\n\nThe AWS Access keys will be configured in application host environment later.\n\n\n### Live objects side\n\n#### Generate Live Objects API key\nLogin to Live Objects Web Portal and go to **Administration** -\u003e **API keys**  \n![Api Keys 1](./images/api_key_1.png)  \n\nClick **Add** button and fill fields.  \n\n![Api Keys 2](./images/api_key_2_.png)\nTo  validate  the  creation  of  the  key,  click  on  the **Create** button.  Your  key  is  generated  in  the form of an alphanumeric sequence and a QR code.\n\n\n### AWS Side\n\n#### Create AWS Access key\nLogin to AWS Console, click on user name on top bar and select **My Security Credentials**.\n\n![My_Security_Credentials](./images/my_security_credentials.png)\n\nExpand **Access keys (access key ID and secret access key)** and click **Create New Access Key**\n\n![Create_New_Access_Key](./images/create_new_access_key.png)\n\n\n#### Use AWS Access keys\nCreated AWS Access key (access key id and secret access key) place in file **~/.aws/config** on machine where connector will be runned:\n\n```\n[default]\naws_access_key_id = \naws_secret_access_key = \n```  \n\n#### Create AWS SQS queue\n\nIn order to receive messages from Live Objects AWS Simple Queue Service (SQS) must be defined.\n\nIn AWS Console select **Services** -\u003e **Application Integration** -\u003e **Simple Queue Service**, and next click **Create queue** button.\n\nNext, please define **Type** and insert **Name** of Queue:\n\n![Create_Queue](./images/create_queue.png)\n\nAnd click button **Create queue**.\n\nAfter queue was created URL can be copied for further use in `application.yaml`.\n\n![Queue_Created](./images/queue_created.png)\n\n### Connector side\n#### Application\nConfiguration can be found in **application.yaml** file located in `conf/` directory.\n```\n1     lo:\n2       hostname: liveobjects.orange-business.com\n3       api-key:\n4       topic:\n5       synchronization-interval: 60000\n6       message-qos: 1\n7       mqtt-persistence-dir: ${basedir:.}/temp/\n8       keep-alive-interval-seconds: 30\n9       connection-timeout: 30000\n10      message-batch-size: 10\n11     \n12    aws:\n13      sqs:\n14        queue-url:\n15        thread-pool-size: 40\n16        connection-timeout: 5000\n17        task-queue-size: 150000\n18        throttling-delay: 5000\n19        max-send-attempts: 3\n20        message-group-id: messagegroup1\n21        region: eu-central-1\n22        keep-alive-time: 10\n23    \n24    cloud:\n25      aws:\n26        stack:\n27          auto: false\n```\nYou can change all values but the most important lines are:\n\n\n**2** - Live Objects hostname  \n**3** - Live Objects API key with at least BUS_R role  \n**4** - Name of the Live Objects MQTT queue  \n**14** - URL of your AWS SQS queue  \n**21** - Region where the SQS is placed  \n\n#### Logging\nLogging configuration can be found in **logback.xml** file located in `conf/` directory. You can find more information about how to configure your logs [here](http://logback.qos.ch/manual/configuration.html)\n\n\n## Launching\nIn order to run the connector, use the `app.sh` file for linux or `app.bat` if you are using windows. These files are located in the `bin/` directory.\n\n### Launching on local machine\nConnector is prepared for running on AWS machine. To start application on local environment, it's necessary to add extra configuration in **application.yml**, for example:\n\n```\ncloud:\n  aws:\n    region:\n      static: eu-central-1\n```\n\n### Installation on AWS machine\n\nIn order to be deployed to AWS, the project uses Elastic Beanstalk. The installation is described on the following tutorial: [documentation](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.html)\n\n#### App deployment\n\nDeployment to AWS is performed by the Elastic Beanstalk. Its deployment is described in [documentation](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Java.html).\n\n### Start receiving messages\nAfter starting the application it's necessary to go `http://localhost:8080/run` or similar address associated with virtual machine in order to start receiving messages. After this action connector will start fetching messages from Live Objects Queue and put them into AWS SQS queue.\n\n\n### Metrics\nApplication sends its metrics to Cloud Watch Metrics every 1 minute. There are number of available metrics e.g. amount of received, sent and failured messages. You can watch them all in Amazon Cloud Watch console under \"Metrics\" in namespace - mqtt2sqs.\n\n![metric1](./images/metr1.png)\n\n![metric2](./images/metr2.png)\n\n### Health Check\nYou can configure Alarm to notify when something goes wrong with your environment. Details can be found [here](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.alarms.html)   \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatavenueliveobjects%2Fpushing-data-to-aws-simple-queue-service-sqs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatavenueliveobjects%2Fpushing-data-to-aws-simple-queue-service-sqs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatavenueliveobjects%2Fpushing-data-to-aws-simple-queue-service-sqs/lists"}