{"id":19482096,"url":"https://github.com/harbby/sylph","last_synced_at":"2025-04-12T19:50:56.742Z","repository":{"id":33276705,"uuid":"137843879","full_name":"harbby/sylph","owner":"harbby","description":"Stream computing platform for bigdata","archived":false,"fork":false,"pushed_at":"2024-04-24T02:16:01.000Z","size":10941,"stargazers_count":401,"open_issues_count":67,"forks_count":173,"subscribers_count":45,"default_branch":"main","last_synced_at":"2025-04-03T23:11:24.466Z","etag":null,"topics":["big-data","flink","java","spark-streaming","sql","streamsql","sylph"],"latest_commit_sha":null,"homepage":"https://harbby.github.io/project/sylph/index.html","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harbby.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-19T05:26:24.000Z","updated_at":"2025-02-07T03:38:27.000Z","dependencies_parsed_at":"2024-11-18T01:20:06.397Z","dependency_job_id":null,"html_url":"https://github.com/harbby/sylph","commit_stats":{"total_commits":514,"total_committers":15,"mean_commits":"34.266666666666666","dds":0.4513618677042801,"last_synced_commit":"59874af86fc5d7147390a1d4bd3483596e3108d9"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harbby%2Fsylph","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harbby%2Fsylph/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harbby%2Fsylph/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harbby%2Fsylph/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harbby","download_url":"https://codeload.github.com/harbby/sylph/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248625501,"owners_count":21135513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","flink","java","spark-streaming","sql","streamsql","sylph"],"created_at":"2024-11-10T20:08:20.335Z","updated_at":"2025-04-12T19:50:56.712Z","avatar_url":"https://github.com/harbby.png","language":"Java","funding_links":[],"categories":["大数据"],"sub_categories":[],"readme":"# Sylph [![Build Status](https://travis-ci.com/harbby/sylph.svg?branch=master)](https://travis-ci.com/harbby/sylph)\n[![license](https://img.shields.io/badge/license-apache_v2-groon.svg)]()\n[![language](https://img.shields.io/badge/language-java_17-green.svg)]()\n[![os](https://img.shields.io/badge/os-Linux_macOS-blue.svg)]()\n\nWelcome to Sylph !\n\nSylph is Streaming Job Manager. \n\nSylph uses SQL Query to describe calculations and bind multiple source(input)/sink(output) to visually develop and deploy streaming applications.\nThrough Web IDE makes it easy to develop, deploy, monitor streaming applications and analyze streaming application behavior at any time.  \nSylph has rich source/sink support and flexible extensions to visually develop and deploy stream analysis applications and visualized streaming application lifecycle management.\n\nThe Sylph core is to build distributed applications through workflow descriptions.\nSupport for \n* Spark-Streaming (Spark1.x)\n* Structured-Streaming (Spark2.x)\n* Flink Streaming\n\n## License\n```\nCopyright (C) 2018 The Sylph Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n\n## StreamingSql\n```sql\ncreate function get_json_object as 'com.github.harbby.sylph.runner.flink.runtime.UDFJson';\n\ncreate source table topic1(\n    _topic varchar,\n    _key varchar,\n    _partition integer,\n    _offset bigint,\n    _message varchar,\n    ip varchar extend '$.conntent.ip',                 -- json path\n    event_time varchar extend '$.conntent.event_time'  -- json path\n) with (\n    type = 'kafka08',\n    kafka_topic = 'event_topic',\n    auto.offset.reset = latest,\n    kafka_broker = 'localhost:9092',\n    kafka_group_id = 'test1',\n    zookeeper.connect = 'localhost:2181'\n);\n\n-- 定义数据流输出位置\ncreate sink table event_log(\n    key varchar,\n    user_id varchar,\n    offset bigint\n) with (\n    type = 'kudu',\n    kudu.hosts = 'localhost:7051',\n    kudu.tableName = 'impala::test_kudu.log_events',\n    kudu.mode = 'INSERT',\n    batchSize = 5000\n);\n\ninsert into event_log\nselect _key,get_json_object(_message, 'user_id') as user_id,_offset \nfrom topic1\n```\n\n## UDF UDAF UDTF\nThe registration of the custom function is consistent with the hive\n```sql\ncreate function get_json_object as 'com.github.harbby.sylph.runner.flink.runtime.UDFJson';\n```\n\n## Building\nsylph builds use Gradle and requires Java 8.    \nAlso if you want read a chinese deploy docs,[中文部署文档](sylph-docs/src/main/docs/source/zh-cn/docs/intro/deploy.md) \nmay can help you.\n```\n# Build and install distributions\n./gradlew clean assemble dist\n```\n## Running Sylph in your IDE\nAfter building Sylph for the first time, you can load the project into your IDE and run the server. Me recommend using IntelliJ IDEA.\n\nAfter opening the project in IntelliJ, double check that the Java SDK is properly configured for the project:\n\n* Open the File menu and select Project Structure\n* In the SDKs section, ensure that a 1.8 JDK is selected (create one if none exist)\n* In the Project section, ensure the Project language level is set to 8.0 as Sylph makes use of several Java 8 language features\n* HADOOP_HOME(2.6.x+) SPARK_HOME(2.4.x+) FLINK_HOME(1.7.x+)\n\nSylph comes with sample configuration that should work out-of-the-box for development. Use the following options to create a run configuration:\n\n* Main Class: com.github.harbby.sylph.main.SylphMaster\n* VM Options: -Dconfig=etc/sylph/sylph.properties -Dlogging.config=etc/sylph/logback.xml\n* ENV Options: FLINK_HOME=\u003cyour flink home\u003e\n               HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop\n* Working directory: sylph-dist/build\n* Use classpath of module: sylph-main\n \n## Useful mailing lists\n1. yezhixinghai@gmail.com - For discussions about code, design and features\n2. lydata_jia@163.com -  For discussions about code, design and features\n3. jeific@outlook.com - For discussions about code, design and features\n\n## Getting Help\n* Send message to [Google Group](https://groups.google.com/forum/#!forum/sylph-streaming)\n* Add QQ Group: 438625067\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharbby%2Fsylph","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharbby%2Fsylph","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharbby%2Fsylph/lists"}