{"id":17964884,"url":"https://github.com/feng19/tdata","last_synced_at":"2025-03-25T06:31:05.977Z","repository":{"id":57555826,"uuid":"82014911","full_name":"feng19/tdata","owner":"feng19","description":"transform data","archived":false,"fork":false,"pushed_at":"2019-07-22T02:51:11.000Z","size":232,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-19T09:40:45.142Z","etag":null,"topics":["configparser","erlang"],"latest_commit_sha":null,"homepage":null,"language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/feng19.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-15T03:10:25.000Z","updated_at":"2024-01-30T15:50:54.000Z","dependencies_parsed_at":"2022-09-02T11:01:12.989Z","dependency_job_id":null,"html_url":"https://github.com/feng19/tdata","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feng19%2Ftdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feng19%2Ftdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feng19%2Ftdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feng19%2Ftdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/feng19","download_url":"https://codeload.github.com/feng19/tdata/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245413663,"owners_count":20611351,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["configparser","erlang"],"created_at":"2024-10-29T12:09:24.510Z","updated_at":"2025-03-25T06:31:05.642Z","avatar_url":"https://github.com/feng19.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"tdata\n=====\n\n专为`erlang`开发的配置生成工具\n\n![tdata](doc/tdata.png)\n\n## Getting Started\n\n* erlang\n* python\n* xlrd\n\n```shell\n$ pip install xlrd  \n```\n\n* rebar3\n* build\n``` shell\n$ make es \u0026\u0026 ./tdata -h\n```\n\n## 使用说明\n\n### 帮助文档\n\n```shell\n$ ./tdata help\nUsage: tdata [-i [\u003cinput_dir\u003e]] [-o [\u003coutput_dir\u003e]] [-t [\u003ctemplate_dir\u003e]]\n             [-r [\u003crecursive\u003e]] [-f [\u003cforce\u003e]] [-c \u003cchild_dir\u003e]\n             [-F [\u003cconfig_file\u003e]] [-a \u003capp\u003e] [-h] [-v]\n\n  -i, --input_dir     Input directory eg: data or \"data/*\". [default: data]\n  -o, --output_dir    Output Dir eg: output. [default: output]\n  -t, --template_dir  Template directory eg: template_dir. [default: \n                      templates]\n  -r, --recursive     Recursive search ChildDir of InputDir: true or \n                      false. [default: false]\n  -f, --force         Force gen all output: true or false. [default: false]\n  -c, --child_dir     Specify ChildDir of InputDir: '-c child_dir1 -c \n                      child_dir2 ...'\n  -F, --config_file   ConfigFile eg: tdata.config. [default: tdata.config]\n  -a, --app           Which application want to start.\n  -h, --help          Print help.\n  -v, --version       Print version.\n```\n\n### 全局配置\n\n* `input_dir`: 指定输入目录,存放配置文件的目录\n* `output_dir`: 指定输出目录,存放生成文件的目录\n* `template_dir`: 指定模板目录\n* `force`: 忽略时间因素强制执行\n* `child_dir`: 输入目录子目录\n* `config_file`: 配置文件,默认加载当前目录下的`tdata.config`配置文件\n* `app`: 执行脚本需要启动的依赖应用\n* `help`: 打印帮助内容\n* `version`:  打印`tdata`的版本号\n\n### 启动程序\n\n`tdata`会自动增加`ebin`和`_build/default/lib/*/ebin`目录到`vm`加载目录;\n\n自动启动`src`目录下的应用或者通过`--app`制定的应用;\n\n自动遍历所有定义了`-tdata(ds).`的`数据处理模块`,然后从接口`transform_defines/0`或者`transform_defines/1`获取具体要转换的输入文件列表等.\n\n### 编写数据处理模块\n\n数据处理模块示例: [`getting_started.erl`](example/GettingStarted/src/getting_started.erl)\n\n```erlang\n%% 配置\ntransform_defines() -\u003e\n    [#{\n        % 输入文件列表 \u0026 已经加载配置\n        input_file_defines =\u003e [#{file =\u003e\"config.xlsx\", opts =\u003e ExcelLoaderOpts}],\n        % 生成目标文件\n        output_file =\u003e \"src/config.erl\",\n        % 转换数据函数\n        % transform_fun =\u003e fun only_print/1,\n        transform_fun =\u003e fun transform_fun/1,\n        % 模板文件\n        tpl_file =\u003e \"config.erl.tpl\"\n    }].\n```\n\n上图,我们定义了一个输入文件(`config.xlsx`),输出文件为`src/config.erl`,\n\n转换数据方法为:`fun transform_fun/1`,模板文件为: `config.erl.tpl`\n\n原始`excel`表格:\n\n![excel_table](doc/table_view.png)\n\n经过`excel_loader`转换后的数据:\n\n```erlang\n{\"config.xlsx\",\n #{\u003c\u003c\"groups\"\u003e\u003e =\u003e\n       #{1 =\u003e\n             #{11 =\u003e\n                   [#{level_1 =\u003e 1,level_2 =\u003e 11,level_3 =\u003e 111},\n                    #{level_1 =\u003e 1,level_2 =\u003e 11,level_3 =\u003e 112}],\n               12 =\u003e\n                   [#{level_1 =\u003e 1,level_2 =\u003e 12,level_3 =\u003e 113},\n                    #{level_1 =\u003e 1,level_2 =\u003e 12,level_3 =\u003e 114}]},\n         2 =\u003e\n             #{21 =\u003e\n                   [#{level_1 =\u003e 2,level_2 =\u003e 21,level_3 =\u003e 221},\n                    #{level_1 =\u003e 2,level_2 =\u003e 21,level_3 =\u003e 222}],\n               22 =\u003e\n                   [#{level_1 =\u003e 2,level_2 =\u003e 22,level_3 =\u003e 223},\n                    #{level_1 =\u003e 2,level_2 =\u003e 22,level_3 =\u003e 224}]}}\n  }\n}\n```\n\n处理方法示例 `fun transform_fun/1`:\n\n```erlang\n%% 转换数据\ntransform_fun([{_InputFile, Sheets}]) -\u003e\n    % 经过转换后的数据\n    #{\n        \u003c\u003c\"类型\"/utf8\u003e\u003e := RecordsRows,\n        \u003c\u003c\"groups\"\u003e\u003e := GroupsRows,\n        \u003c\u003c\"same_level_groups\"\u003e\u003e := SameLevelGroupsRows\n    } = Sheets,\n    ...\n\t% 转印到模板的数据\n    {ok, #{records =\u003e Records, ...}}.\n```\n\n模板支持:\n\n* [`bbmustache`](\u003chttps://github.com/soranoba/bbmustache\u003e): [`mustache`](http://mustache.github.io/)\n* [`erlydtl`](\u003chttps://github.com/erlydtl/erlydtl/wiki\u003e): [`django`](\u003chttps://django.readthedocs.org/en/1.6.x/ref/templates/builtins.html\u003e)\n\n## Excel Loader 配置\n\n配置分为两个层级\n\n第一层层级,定义每个文件的表单(`sheet`)的列表配置:\n\n```erlang\n#{\n    \u003c\u003c\"类型\"/utf8\u003e\u003e =\u003e SheetOpts,\n    \u003c\u003c\"groups\"\u003e\u003e =\u003e SheetOpts#{\n        groups =\u003e [level_1, level_2]\n    },\n    \u003c\u003c\"same_level_groups\"\u003e\u003e =\u003e SheetOpts#{\n        groups =\u003e [[level_1, level_2]]\n    }\n}.\n```\n\n第二层级,定义每个表单(`sheet`)配置:\n\n```erlang\n#{\n    skip_comments =\u003e true,\n    type_comment =\u003e true,\n    only_rows =\u003e true,\n    ...\n}\n```\n\n- `skip_comments`: 跳过N行注释, `true` 表示跳过1行,`false`表示不跳过;\n- `type_comment`: 是否指定类型, 默认注释之后的第一行为类型注释,支持的类型有: `integer(int)/float/boolean(bool)/string(str)/binary(bin)/atom`,括号内的为别名,在类型前加`@`符号表示检查当前列的数据;\n- `only_rows`: 返回数据时,只返回行数据,不返回额外的数据;\n- `groups`: 指定合并单元格的列: 示例,如果第一列为`level_1`为合并单元格,则配置为`[level_1]`;\n- `checks`: 指定每一列的检查方法或者转换函数.\n\n具体请看[tdata_excel_loader_SUITE](test/ct/tdata_excel_loader_SUITE/tdata_excel_loader_SUITE.erl)文件\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeng19%2Ftdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffeng19%2Ftdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeng19%2Ftdata/lists"}