{"id":20173095,"url":"https://github.com/ohmycloud/sub_events","last_synced_at":"2026-05-04T12:40:30.335Z","repository":{"id":223098752,"uuid":"165838773","full_name":"ohmycloud/sub_events","owner":"ohmycloud","description":"多个事件的分段","archived":false,"fork":false,"pushed_at":"2019-01-18T01:38:34.000Z","size":27,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-24T16:14:28.486Z","etag":null,"topics":["raku","spark","spark-streaming","streaming"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ohmycloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-01-15T11:23:51.000Z","updated_at":"2024-02-18T07:44:52.000Z","dependencies_parsed_at":"2024-02-18T09:33:13.401Z","dependency_job_id":"9ad1f484-6af6-4457-963b-441d4fbe8061","html_url":"https://github.com/ohmycloud/sub_events","commit_stats":null,"previous_names":["ohmycloud/sub_events"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ohmycloud%2Fsub_events","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ohmycloud%2Fsub_events/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ohmycloud%2Fsub_events/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ohmycloud%2Fsub_events/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ohmycloud","download_url":"https://codeload.github.com/ohmycloud/sub_events/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241610977,"owners_count":19990505,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["raku","spark","spark-streaming","streaming"],"created_at":"2024-11-14T01:33:25.584Z","updated_at":"2026-05-04T12:40:25.313Z","avatar_url":"https://github.com/ohmycloud.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 模拟实时数据流\n\n运行 perl6 fake-streaming.pl6, 每隔 5 秒往 0.0.0.0 地址的 3333 Socket 端口发送一行 JSON 样本数据：\n\n```\n{'vin':'LSJA0000000000091','ts':1547727950000,'veh_odo':0,'alm_common_temp_diff':0,'alm_common_temp_high':1,'alm_common_esd_high':1}\n{'vin':'LSJA0000000000091','ts':1547727955000,'veh_odo':1,'alm_common_temp_diff':1,'alm_common_temp_high':1,'alm_common_esd_high':0}\n{'vin':'LSJA0000000000091','ts':1547727960000,'veh_odo':2,'alm_common_temp_diff':1,'alm_common_temp_high':1,'alm_common_esd_high':1}\n{'vin':'LSJA0000000000091','ts':1547727965000,'veh_odo':3,'alm_common_temp_diff':0,'alm_common_temp_high':1,'alm_common_esd_high':1}\n{'vin':'LSJA0000000000091','ts':1547727970000,'veh_odo':4,'alm_common_temp_diff':0,'alm_common_temp_high':1,'alm_common_esd_high':0}\n{'vin':'LSJA0000000000091','ts':1547727975000,'veh_odo':5,'alm_common_temp_diff':0,'alm_common_temp_high':0,'alm_common_esd_high':1}\n{'vin':'LSJA0000000000091','ts':1547727980000,'veh_odo':6,'alm_common_temp_diff':1,'alm_common_temp_high':0,'alm_common_esd_high':0}\n{'vin':'LSJA0000000000091','ts':1547727985000,'veh_odo':7,'alm_common_temp_diff':0,'alm_common_temp_high':1,'alm_common_esd_high':0}\n{'vin':'LSJA0000000000091','ts':1547727990000,'veh_odo':8,'alm_common_temp_diff':0,'alm_common_temp_high':0,'alm_common_esd_high':1}\n{'vin':'LSJA0000000000091','ts':1547727995000,'veh_odo':9,'alm_common_temp_diff':0,'alm_common_temp_high':0,'alm_common_esd_high':1}\n{'vin':'LSJA0000000000091','ts':1547728000000,'veh_odo':10,'alm_common_temp_diff':1,'alm_common_temp_high':0,'alm_common_esd_high':0}\n{'vin':'LSJA0000000000091','ts':1547728005000,'veh_odo':11,'alm_common_temp_diff':0,'alm_common_temp_high':1,'alm_common_esd_high':1}\n{'vin':'LSJA0000000000091','ts':1547728010000,'veh_odo':12,'alm_common_temp_diff':0,'alm_common_temp_high':1,'alm_common_esd_high':1}\n{'vin':'LSJA0000000000091','ts':1547728015000,'veh_odo':13,'alm_common_temp_diff':1,'alm_common_temp_high':1,'alm_common_esd_high':0}\n```\n\n该 Streaming 程序的任务是解析 JSON 流， JSON 中每个字段的意义如下：\n\n```scala\n/**\n  *\n  * @param vin 车架号\n  * @param ts 事件发生时间\n  * @param mileage 当前里程数\n  * @param alm_common_temp_diff 温度差异报警\n  * @param alm_common_temp_high 电池高温报警\n  * @param alm_common_esd_high 车载储能装置类型过压报警\n  */\ncase class SourceData (\n   vin:        String,\n   ts:         Long,\n   veh_odo:    Double,\n   alm_common_temp_diff: Integer,\n   alm_common_temp_high: Integer,\n   alm_common_esd_high: Integer\n)\n```\n\n程序的输出是一个 **EventUpdate** 类型的数组, EventUpdate 定义了每个事件的信息：\n\n```scala\ncase class EventUpdate(\n                        var vin: String           = \"\",   // 车架号\n                        var eventName: String     = \"\",   // 事件名称\n                        var eventStartTime: Long  = -999, // 事件开始时间\n                        var eventEndTime: Long    = -999, // 事件结束时间\n                        var eventStatus: Int      = -999, // 事件状态\n                        var eventType: Int        = -999, // 事件类型\n                        var startMileage: Double  = -999  // 事件发生时的里程数\n\n                      ) {\n  def eventDuration: Long = eventEndTime - eventStartTime  // 事件持续时长\n}\n\nobject EventUpdate {}\n```\n\n其中 **alm_common_temp_diff**, **alm_common_temp_high**, **alm_common_esd_high** 都是事件, 当其值为 1 时, 表示发生了该事件; 当其值为 0 时, 表示没有发生该事件。\n\n要求输出这三种类型的事件：\n\na)、当事件第一次发生时①  \nb)、俩个事件之前的时间之差超过 15 秒时, 则结束上一个事件②, 并开始一个新的事件③;  \n\n## 配置文件\n\nresources 目录下的文件：\n\n- application.conf\n\n```\nspark {\n  master = \"local[4]\"\n  streaming.batch.duration = 5000\n  eventLog.enabled         = true\n  ui.enabled               = true\n  ui.port                  = 4040\n  metrics.conf             = metrics.properties\n  spark.cleaner.ttl        = 3600\n  checkpoint.path          = \"/tmp/telematics\"\n  spark.cleaner.referenceTracking.cleanCheckpoints = true\n}\n\nsocket {\n  host = \"localhost\"\n  port = 3333\n}\n```\n\n- log4j.properties\n\n```\n# Set everything to be logged to the console\nlog4j.rootCategory=ERROR, console\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\nlog4j.appender.console.target=System.out\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\nlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %l %p %c{1}: %m%n\n\n# Settings to quiet third party logs that are too verbose\nlog4j.logger.org.spark-project.jetty=ERROR\nlog4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR\nlog4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO\nlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO\nlog4j.logger.org.apache.parquet=ERROR\nlog4j.logger.parquet=ERROR\n\n# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support\nlog4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL\nlog4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR\n```\n\n- metrics.properties\n\n```\n*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink\n*.sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet\n*.sink.slf4j.class=org.apache.spark.metrics.sink.Slf4jSink\n*.sink.slf4j.period=5\n*.sink.slf4j.unit=seconds\n```\n\n## 运行方式\n\n先启动上面的实时数据流模拟程序： perl6 fake-streaming.pl6，再开启另外一个窗口执行：./spark-submit.sh. 观察程序的输出。\n\n## 输出结果\n\n程序的输出显示如下：\n\n```\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_high,1547727950000,1547727950000,1,1,0.0), EventUpdate(LSJA0000000000091,alm_common_esd_high,1547727950000,1547727950000,1,1,0.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547727955000,1547727955000,1,1,1.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547727955000,1547727960000,0,1,1.0), EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547727980000,1547727980000,1,1,6.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547727980000,1547727980000,0,1,6.0), EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728000000,1547728000000,1,1,10.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_high,1547727950000,1547727985000,0,1,0.0), EventUpdate(LSJA0000000000091,alm_common_temp_high,1547728005000,1547728005000,1,1,11.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728000000,1547728020000,0,1,10.0), EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728040000,1547728040000,1,1,18.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728040000,1547728040000,0,1,18.0), EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728060000,1547728060000,1,1,22.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_high,1547728005000,1547728045000,0,1,11.0), EventUpdate(LSJA0000000000091,alm_common_temp_high,1547728065000,1547728065000,1,1,23.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728060000,1547728080000,0,1,22.0), EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728100000,1547728100000,1,1,30.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728100000,1547728100000,0,1,30.0), EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728120000,1547728120000,1,1,34.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_high,1547728065000,1547728105000,0,1,23.0), EventUpdate(LSJA0000000000091,alm_common_temp_high,1547728125000,1547728125000,1,1,35.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728120000,1547728140000,0,1,34.0), EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728160000,1547728160000,1,1,42.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728160000,1547728160000,0,1,42.0), EventUpdate(LSJA0000000000091,alm_common_temp_diff,1547728180000,1547728180000,1,1,46.0)))\n(LSJA0000000000091,ArrayBuffer(EventUpdate(LSJA0000000000091,alm_common_temp_high,1547728125000,1547728165000,0,1,35.0), EventUpdate(LSJA0000000000091,alm_common_temp_high,1547728185000,1547728185000,1,1,47.0)))\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fohmycloud%2Fsub_events","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fohmycloud%2Fsub_events","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fohmycloud%2Fsub_events/lists"}