{"id":18898493,"url":"https://github.com/luojilab/datatranshub","last_synced_at":"2025-08-21T06:30:54.652Z","repository":{"id":44873299,"uuid":"285166204","full_name":"luojilab/DataTransHub","owner":"luojilab","description":"跨平台Android/iOS海量数据上报组件，基于Xlog完善，解决Xlog痛点问题。","archived":false,"fork":false,"pushed_at":"2022-04-20T05:42:58.000Z","size":11068,"stargazers_count":159,"open_issues_count":9,"forks_count":39,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-12-10T10:03:20.793Z","etag":null,"topics":["android","data","data-report","ios","logger","xlog"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luojilab.png","metadata":{"files":{"readme":"ReadMe.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-05T03:17:21.000Z","updated_at":"2024-11-28T08:45:35.000Z","dependencies_parsed_at":"2022-09-13T17:51:10.605Z","dependency_job_id":null,"html_url":"https://github.com/luojilab/DataTransHub","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luojilab%2FDataTransHub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luojilab%2FDataTransHub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luojilab%2FDataTransHub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luojilab%2FDataTransHub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luojilab","download_url":"https://codeload.github.com/luojilab/DataTransHub/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230494921,"owners_count":18235046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","data","data-report","ios","logger","xlog"],"created_at":"2024-11-08T08:42:54.735Z","updated_at":"2024-12-19T20:08:12.320Z","avatar_url":"https://github.com/luojilab.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"## DataTransHub\n\n[![name](https://img.shields.io/badge/release-1.2.8-green?style=flat)]()\n[![license](https://img.shields.io/badge/license-MIT-brightgreen?style=flat)]()\n[![platform](https://img.shields.io/badge/Platform-Android%20%7C%20iOS-red?style=flat)]()\n\nDataTransHub 是基于mmap的客户端海量数据上报组件，其中包括两个组件DataTransHub和Flog\nDataTransHub主要用于数据的上报，Flog主要用于log的上报。\n实现客户端海量数据的高性能上报，上报过程不会block任何操作，实现高性能上报，基于腾讯Xlog的数据压缩和加密算法，实现数据的高压缩比压缩和加密存储。全新设计上报调度模块，策略调度完全由内部控制。\n\n## 特点：\n1. 跨平台，DataTransHub是跨平台实现，适用于Android和iOS，多端代码统一，逻辑统一，避免逻辑不一致性。\n2. 海量数据存储和上报，采用实时压缩和加密，实现海量数据高效上报服务端。\n3. 多实例实现，不同业务采用不同实例，不同上报接口，完全解决腾讯Xlog单实例的弊端。\n4. 不丢数据，采用mmap实现缓冲区。即使发生应用crash或者进程被杀，数据不会丢失，下次启动会再次上报。\n5. 低功耗，采用c++实现，整个上报共用一个线程，多实例多业务场景共用同一线程，当数据全部上报完成后，线程休眠。\n6. 上报策略合理封装，通过参数设置，可以实现完全自主策略设置，满足不同业务需求\n7. 合理设置缓冲区大小和上报文件大小，可以提高网络利用率，减少上报次数以及网络链接次数。\n\n## 原理\n数据通过调用接口进入DataTransHub，先缓存到数据缓冲区，缓冲区实现是mmap，即使发生crash数据不丢失。当缓冲区数据量达到阈值，异步线程进行数据落盘，或者设置定期数据落盘。当时间达到上报时间点后，内部调度线程会调用外部实现接口，进行数据上传，上层上传成功或失败后通知底层，底层进行相应策略调整，根据策略决定下次触发文件上传时间，到达时间点后再次触发上传。\n\ndemo 的使用请参考[这里](https://github.com/luojilab/DataTransHub/tree/master/sample)。\n\n## 开始接入\n\n接入 [Android](#android_cn) 或者 [iOS](#apple_cn)\n\n### \u003ca name=\"android_cn\"\u003e[Android]()\u003c/a\u003e\n## aar接入\n1. 在项目跟目录build.gradle中加入\n    repositories {\n        mavenCentral()\n    }\n2. 在项目build.gradle中加入\nimplementation 'io.github.luojilab:datatranshub:1.2.8'\n3. 按照demo的调用方式接入\n\n## 源码编译\n准备条件：\n1. AndroidStudio3.1及以上版本\n2. android-ndk-r14b及以上版本\n\n步骤：\n1. AndroidStuido打开源码路径project/android AndroidStuido项目文件\n2. 打开AndroidStudio中File菜单，Project Structure菜单\n3. 在Android NDK location中配置NDK位置\n4. 在AndroidStuido中的Gradle菜单中找到位于Android菜单下的datareporter，在Tasks中找到build，选择assembleRelease进行编译。\n5. 生成datatranshub-release.aar 导入到项目中进行使用\n\n## 调用方式\n\n    /**\n     * 创建DataTransHub实例\n     *\n     * @param cacheDir 缓存路径，必须全局唯一，不同业务不同路径\n     * @param dataDir 数据存储路径，必须全局唯一，不同业务不同路径\n     * @param encryptKey 缓存和数据加密的key，需要通过脚本生成\n     * @return\n     */\n    public synchronized static DataTransHub makeDataTransHub(String cacheDir, String dataDir, String encryptKey);\n\n\n​\t\n    /**\n     * 释放上报实例\n     *\n     * @param dataTransHub 需要释放的实例\n     * @param releaseDataTransHub \n     */\n    public synchronized static void releaseDataTransHub(DataTransHub dataTransHub);\n    \n    /**\n     * 设置数据文件上传实现方法\n     *\n     * @param uploadImp 文件上传的实现\n     */\n    public synchronized void setUploadImp(IUploadImp uploadImp);\n    \n    /**\n     * 设置上传文件最大的尺寸，单位字节\n     *\n     * @param fileMaxSize    最大尺寸\n     */\n    public synchronized void setFileMaxSize(long fileMaxSize);\n    \n     /**\n     * 设置缓存buffer的大小，该大小不应该大于单文件最大尺寸，单位字节\n     *\n     * @param bufferSize   buffer的尺寸\n     */\n    public synchronized void setBufferSize(long bufferSize);\n    \n    /**\n     * 设置文件上报的有效期，单位是秒，如果设置为0则所有数据都上报，不存在过期逻辑\n     *\n     * @param expiredTime 有效期\n     */\n    public synchronized void setExpiredTime(long expiredTime);\n    \n    /**\n     * 设置上报间隔，防止上报过度频繁，影响网络。可设置间隔上报\n     *\n     * @param reportingInterval 上报间隔 单位毫秒\n     */\n    public synchronized void setReportingInterval(long reportingInterval);\n\n\n    /**\n     * 设置重试间隔，一次上报失败后，重试阶梯通过该方法设置，防止上报失败后马上进行重试，通过该参数设置重试阶梯\n     *\n     * @param retryInterval 单位毫秒\n     */\n    public synchronized void setRetryInterval(long retryInterval);\n    \n    /**\n     * 设置数据文件前缀，用于方便过滤数据文件，可不传\n     *\n     * @param prefix         文件前缀\n     */\n    public synchronized void setDataFilePrefix(String prefix);\n    \n    /**\n     * 设置上报策略，如果设置为UploadTriggerWayWayManual，则不会自动上报，需要手动触发上报\n     *\n     * @param uploadTriggerWay UploadTriggerWayWayAuto 或者 UploadTriggerWayWayManual\n     */\n    public synchronized void setUploadTriggerWayWay(int uploadTriggerWay);\n    \n    /**\n     * 手动触发上传方法，如果策略设置为手动，则通过这个方法进行触发\n     *\n     * @param callback 上传成功后的回调\n     */\n    public synchronized void manualTriggerUpload(IUploadCallback callback)\n    \n    /**\n     * 设置写入周期，多久数据落盘一次，0表示不进行周期落盘，完全依赖缓冲区阈值控制\n     *\n     * @param period 单位毫秒\n     */\n    public synchronized void setWriteDiskPeriod(long period);\n    \n    /**\n     * 实例上报开始，设置完参数后调用该方法\n     *\n     */\n    public synchronized void start()\n    \n    /**\n     * 唤醒方法，当持续上报失败，会触发延长重试策略，当网络转好后，可以通过该方法马上触发再次上报\n     *\n     */\n    public synchronized void reaWaken()\n    \n    /**\n     * 真正的上报数据方法，通过该方法传入数据，然后大量数据存储磁盘成单文件\n     *\n     * @param data 字节数据\n     */\n    public synchronized void push(byte[] data)\n    \n    /**\n     * 通知上传成功，当上传成功后调用该方法\n     *\n     * @param filePath 需要上传的文件路径\n     */\n    public synchronized void notifyUploadSuccess(String filePath)\n    \n    /**\n     * 通知上传失败，当上传失败后调用该方法\n     *\n     * @param filePath 需要上传的文件路径\n     */\n    public synchronized void notifyUploadFailed(String filePath)\n    \n    /**\n     * 上传文件的真正实现接口，需要上层实现该接口\n     *\n     * @param filePath 需要上传的文件路径\n     */\n    public interface IUploadImp {\n        void upload(String filePath);\n    }\n\n## 混淆配置\n-keep class com.iget.datatranshub.\\*\\*{\\*;}\n-keep class com.iget.flog.\\*\\*{\\*;}\n\n\n### \u003ca name=\"apple_cn\"\u003e[iOS]()\u003c/a\u003e\n\n## 库接入\n1. 拷贝release\\iOS下对应架构的静态库文件导入到应用项目\n2. 参照下面的调用方式接入\n\n## 源码编译\n准备条件：\n1. xcode最新版本\n2. cmake最新版本\n\n步骤：\n1. 使用终端进入项目中build/iOS目录\n2. 编译真机版本库执行 build_all.sh 执行后生成的Framwork在libs/all目录下\n3. 编译模拟器版本库执行 build_x86_64.sh 执行后生成的Framwork在libs/x86_64目录下\n4. 生成调试工程执行 generate_project.sh 执行后会生成调试的xcode工程，工程文件在build文件夹下\n\n## 调用方式\n\n    /**\n     * 创建DataTransHub实例\n     *\n     * @param cacheDir 缓存路径，必须全局唯一，不同业务不同路径\n     * @param dataDir 数据存储路径，必须全局唯一，不同业务不同路径\n     * @param encryptKey 缓存和数据加密的key，需要通过脚本生成\n     * @return\n     */\n    (instancetype)log_MakeLogCacheDir:(NSString *) cacheDir dataDir:(NSString *) dataDir encryptKey:(NSString *) encryptKey\n\n​\t\n    /**\n     * 释放上报实例\n     *\n     */\n    (void)log_Destroy\n    \n    /**\n     * 设置数据文件上传实现方法\n     *\n     * @param callBack 文件上传的实现\n     */\n    (void)log_SetUploadBlock:(void (^)(NSString *filePath)) callBack;\n    \n    /**\n     * 设置上传文件最大的尺寸，单位字节\n     *\n     * @param fileMaxSize    最大尺寸\n     */\n    (void)log_SetFileMaxSize:(NSUInteger) fileSize\n    \n     /**\n     * 设置缓存buffer的大小，该大小不应该大于单文件最大尺寸，单位字节\n     *\n     * @param bufferSize   buffer的尺寸\n     */\n    (void)log_SetBufferSize:(NSUInteger) bufferSize\n    \n    /**\n     * 设置文件上报的有效期，单位是秒，如果设置为0则所有数据都上报，不存在过期逻辑\n     *\n     * @param expiredTime 有效期\n     */\n    (void)log_SetExpiredTime:(NSTimeInterval) expiredTime\n    \n    /**\n     * 设置上报间隔，防止上报过度频繁，影响网络。可设置间隔上报\n     *\n     * @param reportingInterval 上报间隔 单位毫秒\n     */\n    (void)log_SetReportingInterval:(NSTimeInterval) reportingInterval\n\n\n    /**\n     * 设置重试间隔，一次上报失败后，重试阶梯通过该方法设置，防止上报失败后马上进行重试，通过该参数设置重试阶梯\n     *\n     * @param retryInterval 单位毫秒\n     */\n    (void)log_SetRetryInterval:(NSTimeInterval) retryInterval\n    \n    /**\n     * 设置数据文件前缀，用于方便过滤数据文件，可不传\n     *\n     * @param prefix         文件前缀\n     */\n    (void)log_SetDataFilePrefix:(NSString *) filePrefix\n    \n    /**\n     * 设置上报策略，如果设置为UploadTriggerWayWayManual，则不会自动上报，需要手动触发上报\n     *\n     * @param uploadTriggerWay UploadTriggerWayWayAuto 或者 UploadTriggerWayWayManual\n     */\n    (void)log_SetUploadTriggerWayWay:(HubUploadTriggerWay) uploadTriggerWay\n    \n    /**\n     * 手动触发上传方法，如果策略设置为手动，则通过这个方法进行触发\n     *\n     * @param completionHandler 上传成功后的回调\n     */\n    (void)log_ManualTriggerUploadCompletionHandler:(dispatch_block_t) completionHandler\n    \n    /**\n     * 设置写入周期，多久数据落盘一次，0表示不进行周期落盘，完全依赖缓冲区阈值控制\n     *\n     * @param period 单位毫秒\n     */\n    (void)data_SetWriteDiskPeriod:(NSUInteger) period\n    \n    /**\n     * 实例上报开始，设置完参数后调用该方法\n     *\n     */\n    (void)log_Start\n    \n    /**\n     * 唤醒方法，当持续上报失败，会触发延长重试策略，当网络转好后，可以通过该方法马上触发再次上报\n     *\n     */\n    (void)log_ReaWaken\n    \n    /**\n     * 真正的上报数据方法，通过该方法传入数据，然后大量数据存储磁盘成单文件\n     *\n     * @param data 字节数据\n     */\n    (void)data_PushData:(NSData *) data\n    \n    /**\n     * 通知上传成功，当上传成功后调用该方法\n     *\n     * @param filePath 需要上传的文件路径\n     */\n    (void)log_NotifyUploadSuccess:(NSString *) filePath\n    \n    /**\n     * 通知上传失败，当上传失败后调用该方法\n     *\n     * @param filePath 需要上传的文件路径\n     */\n    (void)log_NotifyUploadFailed:(NSString *) filePath\n\n## Support\n\n还有其他问题？\n\n1. 参看 [DataTranshHub/sample](https://github.com/luojilab/DataTransHub/tree/master/sample)；\n2. 阅读 [源码](https://github.com/luojilab/DataTransHub/tree/master/src)；\n3. 阅读 [wiki](https://github.com/luojilab/DataTransHub/wiki)；\n4. 联系我们。得到电子书技术组 wx:hlj-lixiaoyu\n5. 开发成员：\n        跨平台、android：李晓宇\n\t\t      iOS对接层：由振北\n\t     go语言解密算法：刘思远\n\n## 致谢：\n感谢腾讯团队开源的MMKV项目。本项目借鉴了部分源码。\n感谢开源。\n\n## License\nDataTransHub 使用的 MIT 协议，详细请参考 [LICENSE](https://github.com/luojilab/DataTransHub/blob/master/LICENSE.md)。","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluojilab%2Fdatatranshub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluojilab%2Fdatatranshub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluojilab%2Fdatatranshub/lists"}