{"id":13798476,"url":"https://github.com/ExpediaGroup/drone-fly","last_synced_at":"2025-05-13T05:32:16.480Z","repository":{"id":39803418,"uuid":"279279234","full_name":"ExpediaGroup/drone-fly","owner":"ExpediaGroup","description":"A service which allows Hive Metastore Listeners to be deployed outside of the Hive Metastore Service","archived":false,"fork":false,"pushed_at":"2024-03-05T08:21:42.000Z","size":476,"stargazers_count":10,"open_issues_count":5,"forks_count":5,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-03-05T09:33:53.500Z","etag":null,"topics":["hive","hive-metastore"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ExpediaGroup.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE-OF-CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-13T11:07:02.000Z","updated_at":"2024-08-04T00:02:51.764Z","dependencies_parsed_at":"2024-01-13T11:13:07.413Z","dependency_job_id":"989ab55a-5761-40a3-9783-00bd4580e342","html_url":"https://github.com/ExpediaGroup/drone-fly","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaGroup%2Fdrone-fly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaGroup%2Fdrone-fly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaGroup%2Fdrone-fly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaGroup%2Fdrone-fly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ExpediaGroup","download_url":"https://codeload.github.com/ExpediaGroup/drone-fly/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253883126,"owners_count":21978611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hive","hive-metastore"],"created_at":"2024-08-04T00:00:44.537Z","updated_at":"2025-05-13T05:32:15.916Z","avatar_url":"https://github.com/ExpediaGroup.png","language":"Java","readme":"![Logo](.README_images/DroneFly_logo.png)\n\nA service which allows Hive metastore (HMS) `MetaStoreEventListener` implementations to be deployed in a separate context to the metastore's own.\n\n## Overview\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.expediagroup/drone-fly-app/badge.svg?subject=com.expediagroup:drone-fly-app)](https://maven-badges.herokuapp.com/maven-central/com.expediagroup/drone-fly-app)\n[![Build Status](https://github.com/ExpediaGroup/drone-fly/workflows/Build/badge.svg)](https://github.com/ExpediaGroup/drone-fly/actions?query=workflow:\"Build\")\n[![Coverage Status](https://coveralls.io/repos/github/ExpediaGroup/drone-fly/badge.svg?branch=main)](https://coveralls.io/github/ExpediaGroup/drone-fly?branch=main)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Docker](https://img.shields.io/badge/docker-drone--fly-blue)](https://hub.docker.com/r/expediagroup/drone-fly-app)\n\nDrone Fly is a distributed Hive metastore events forwarder service that allows users to deploy metastore listeners outside the Hive metastore service.\n\nWith the advent of event-driven systems, the number of listeners that a user needs to install in the metastore is ever increasing. These listeners can be both internal or can be provided by third party tools for integration purposes. More and more processing is being added to these listeners to address various business use cases.\n\nAdding these listeners directly on the classpath of your Hive metastore couples them with it and can lead to performance degradation or in the worst case, it could take down the entire metastore (e.g. by running out memory, thread starvation etc.) Drone Fly decouples your HMS from the event listeners by providing a virtual Hive context. The event listeners can be provided on the Drone Fly's classpath and it then forwards the events received from [Kafka metastore Listener](https://github.com/ExpediaGroup/apiary-extensions/tree/main/apiary-metastore-events/kafka-metastore-events/kafka-metastore-listener) on to the respective listeners.\n\n## Start using\n\nA Terraform module for Kubernetes deployment is available [here](https://github.com/ExpediaGroup/apiary-drone-fly).\n\nDocker images can be found in Expedia Group's [dockerhub](https://hub.docker.com/search/?q=expediagroup%2Fdrone-fly\u0026type=image).\n\n## System architecture\n\nThe diagram below shows a typical Hive metastore setup without using Drone Fly. In this example, there are several HiveMetastoreListeners installed which send Hive events to other systems like Apache Atlas, AWS SNS, Apache Kafka and other custom implementations.\n\n![Hive Metastore setup without Drone Fly.](.README_images/drone-fly-before.png \"Multiple Hive metastore listeners are deployed in HMS context.\")\n\nWith Drone Fly, the setup gets modified as shown in the diagram below. The only listener installed in the Hive metastore context is the [Apiary Kafka Listener](https://github.com/ExpediaGroup/apiary-extensions/tree/main/apiary-metastore-events/kafka-metastore-events/kafka-metastore-listener). This forwards the Hive events on to Kafka from which Drone Fly can retrieve them. The other listeners are moved out into separate contexts and receive the messages from Drone Fly which forwards them on as if they were Hive events so the listener code doesn't need to change at all.\n\n![Hive Metastore setup with Drone Fly.](.README_images/drone-fly-after.png \"Only one Hive metastore listener is deployed in HMS context and others are deployed in Drone Fly context\")\n\nDrone Fly can be set up to run in dockerized containers where each instance is initiated with one listener to get even further decoupling.\n\n## Usage\n### Using with Docker\n\nTo install a new HMS listener within the Drone Fly context, it is recommended that you build your Docker image using the Drone Fly base [Docker image](https://hub.docker.com/r/expediagroup/drone-fly-app).\n\nA sample image to install the [Apiary-SNS-Listener](https://github.com/ExpediaGroup/apiary-extensions/tree/main/apiary-metastore-events/sns-metastore-events/apiary-metastore-listener) would be as follows:\n\n```\nfrom expediagroup/drone-fly-app:0.0.1\n\nENV APIARY_EXTENSIONS_VERSION 6.0.1\n\nENV AWS_REGION us-east-1\nRUN cd /app/libs \u0026\u0026 \\\nwget -q https://search.maven.org/remotecontent?filepath=com/expediagroup/apiary/apiary-metastore-listener/${APIARY_EXTENSIONS_VERSION}/apiary-metastore-listener-${APIARY_EXTENSIONS_VERSION}-all.jar -O apiary-metastore-listener-${APIARY_EXTENSIONS_VERSION}-all.jar\n```\n\n#### Running Drone Fly Docker image\n\n\tdocker run --env APIARY_BOOTSTRAP_SERVERS=\"localhost:9092\" \\\n\t\t   --env APIARY_LISTENER_LIST=\"com.expediagroup.sampleListener1,com.expediagroup.sampleListener2\" \\\n\t\t   --env APIARY_KAFKA_TOPIC_NAME=\"dronefly\" \\\n\t\t         expediagroup/drone-fly-app:\u003cimage-version\u003e\n\nThen [Drone Fly Terraform](https://github.com/ExpediaGroup/apiary-drone-fly) module can be used to install your Docker image in a Kubernetes container.\n\n\n### Using Uber Jar\n\nDrone Fly build also produces an [uber jar](https://mvnrepository.com/artifact/com.expediagroup/drone-fly-app) so it can be started as a stand-alone Java service.\n\n#### Running Drone Fly Jar\n\n\tjava -Dloader.path=lib/ -jar drone-fly-app-\u003cversion\u003e-exec.jar \\\n\t\t--apiary.bootstrap.servers=localhost:9092 \\\n\t\t--apiary.kafka.topic.name=apiary \\\n\t\t--apiary.listener.list=\"com.expediagroup.sampleListener1,com.expediagroup.sampleListener2\"\t\n\t\nThe properties `instance.name`, `apiary.bootstrap.servers`, `apiary.kafka.topic.name` and `apiary.listener.list` can also be provided in the spring properties file.\n\t\n\tjava -Dloader.path=lib/ -jar drone-fly-app-\u003cversion\u003e-exec.jar --spring.config.location=file:///dronefly.properties\n\t\nThe parameter `-Dloader-path` is the path where Drone Fly will search for configured HMS listeners.\n\n## Configuring Drone Fly\n\n### Drone Fly configuration reference\nThe table below describes all the available configuration values for Drone Fly.\n\n| Name | Description | Type | Default | Required |\n|------|-------------|------|---------|:--------:|\n| apiary.bootstrap.servers | Kafka bootstrap servers that receive Hive metastore events. | `string` | n/a | yes |\n| apiary.kafka.topic.name | Kafka topic name that receives Hive metastore events. | `string` | n/a | yes |\n| apiary.listener.list | Comma separated list of Hive metastore listeners to load from the classpath, e.g. `com.expedia.HMSListener1,com.expedia.HMSListener2` | `string` | `\"com.expediagroup.dataplatform.dronefly.app.service.listener.LoggingMetastoreListener\"` | no |\n| instance.name | Instance name for a Drone Fly instance. `instance.name` is also used to derive the Kafka consumer group. Therefore, in a multi-instance deployment, a unique `instance.name` for each Drone Fly instance needs to be provided to avoid all instances ending up in the same Kafka consumer group. | `string` | `drone-fly` | no |\n| endpoint.port | Port on which Drone Fly Spring Boot app will start. | `string` | `8008` | no |\n\n### Additional configuration parameters\nThe Kafka message reader supports properties that are passed to the Kafka consumer builder.\nThese are environment variables with the PREFIX apiary.messaging.consumer.\n\n #### Example for sending consumer parameters when using a Kafka cloud provider\n- apiary.messaging.consumer.security.protocol=SSL\n- apiary.messaging.consumer.sasl.mechanism=AWS_MSK_IAM\n- apiary.messaging.consumer.sasl_jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;\n- apiary.messaging.consumer.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler\n\nIn this case we are sending the properties to Kafka's consumer to be able to connect to AWS MSK which also requires the IAM library included as a dependency in the POM.xml file\n\n\tjava -Dloader.path=lib/ -jar drone-fly-app-\u003cversion\u003e-exec.jar \\\n\t\t--apiary.bootstrap.servers=localhost:9092 \\\n\t\t--apiary.kafka.topic.name=apiary \\\n\t\t--apiary.listener.list=\"com.expediagroup.sampleListener1,com.expediagroup.sampleListener2\" \\\n        --apiary.messaging.consumer.security.protocol=SSL \\\n        --apiary.messaging.consumer.sasl.mechanism=AWS_MSK_IAM \\\n        --apiary.messaging.consumer.sasl_jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; \\\n        --apiary.messaging.consumer.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler\n\n## Metrics\n\nDrone Fly exposes standard [JVM and Kafka metrics](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-metrics-meter) using [Prometheus on Spring Boot Actuator](https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-metrics-export-prometheus) endpoint `/actuator/prometheus`.\n\n### Some of the useful metrics to track are:\n\n```\nsystem_cpu_usage\nkafka_consumer_records_consumed_total_records_total\njvm_memory_committed_bytes\n```\n\n\n## Legal\nThis project is available under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0.html).\n\nCopyright 2020 Expedia, Inc.\n","funding_links":[],"categories":["Tools"],"sub_categories":["Metastore Tools"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FExpediaGroup%2Fdrone-fly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FExpediaGroup%2Fdrone-fly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FExpediaGroup%2Fdrone-fly/lists"}