{"id":13582156,"url":"https://github.com/sirkon/ldetool","last_synced_at":"2025-04-07T08:14:21.496Z","repository":{"id":52212069,"uuid":"97313814","full_name":"sirkon/ldetool","owner":"sirkon","description":"Code generator for fast log file parsers","archived":false,"fork":false,"pushed_at":"2024-11-12T09:29:11.000Z","size":843,"stargazers_count":319,"open_issues_count":0,"forks_count":22,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-31T07:04:14.679Z","etag":null,"topics":["bigdata","datamining","log-parsing","logs-analysis","logs-parsing","parsing","parsing-csv"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sirkon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-15T12:14:18.000Z","updated_at":"2025-01-23T13:59:01.000Z","dependencies_parsed_at":"2022-09-02T00:02:26.105Z","dependency_job_id":"93396116-769c-47b6-8876-1007adf0a900","html_url":"https://github.com/sirkon/ldetool","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sirkon%2Fldetool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sirkon%2Fldetool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sirkon%2Fldetool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sirkon%2Fldetool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sirkon","download_url":"https://codeload.github.com/sirkon/ldetool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247615377,"owners_count":20967184,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","datamining","log-parsing","logs-analysis","logs-parsing","parsing","parsing-csv"],"created_at":"2024-08-01T15:02:27.695Z","updated_at":"2025-04-07T08:14:21.477Z","avatar_url":"https://github.com/sirkon.png","language":"Go","readme":"# ldetool means line data extraction tool\n\n`ldetool` is a command line utility to generate Go code for fast log files parsing.\n\n```bash\ngo install github.com/sirkon/ldetool@latest\n```\n\n1. [Contributors](CONTRIBUTORS.md)\n1. [Rationale](RATIONALE.md)\n2. [Typical operations and formal set of rules](TOOL_RULES.md)\n3. [Performance comparison against regexp and Ragel](PERFORMANCE.md)\n5. [Usage examples](EXAMPLES.md)\n\n\n### How it works.\n1. First write extraction script, we usually name it `\u003csomething\u003e.lde`\n2. Generate go code with `ldetool \u003csomething.lde\u003e --package main`. Of course\n   you can use your own package name, not only `main`\n3. Use it via the generated extraction method `Parse(line []byte)`.\n\n\u003e It turned out we like using it even for non-performant tasks, where we are dealing with strings, not slices of bytes \n\u003e and it would be handy to use it for strings as well without manual type casting. There's an option to generate code\n\u003e that use string, just put an option `--go-string`\n\n##### CLI utility options\n1. `--go-string` generates code that uses `string` everywhere instead of `[]byte`. You better not to use it for log processing as it may lead to excessive memory allocations.\n2. `--yaml-dict` or `--json-dict` sets translation rules for names. For instance, if we have YAML file with\n    ```yaml\n    http: HTTP\n    ```\n    and feed this file to the `ldetool` then every name (of field or rule itself) like `GetHttpHandle` or `get_http_handle` will be translated into `GetHTTPHandle`\n3. `--package \u003cpkg name\u003e` name of the package to use in generated code. If a directory of `*.lde` file has other Go files package name will automatically setup with these files' package name.\n4. `--big-endian` or `--little-endian` sets the target architecture to be either big or little endian. This\n    enables prefix check optimization \n\n#### Example\n\nTake a look at these two lines\n\n```\n[2017-09-02T22:48:13] FETCH first[1] format[JSON] hidden[0] userAgent[Android App v1.0] rnd[21341975] country[MA]\n[2017-09-02T22:48:14] FETCH first[0] format[JSON] userAgent[Android App v1.0] rnd[10000000] country[LC]\n```\n\nWe likely need a time, value of parameter `first`, `format`, `hidden`, `userAgent` and `country`. We obviously don't need `rnd`\n\n##### Extraction script syntax\nSee [more details](https://github.com/sirkon/ldetool/blob/master/TOOL_RULES.md) on extraction rules\n\n```perl\n# filename: line.lde\nLine =                                   # Name of the extraction object' type\n  ^'[' Time(string) ']'                  # The line must start with [, then take everything as a struct field Time string right to ']' character\n  ^\" FETCH \"                             # Current rest must starts with \" FETCH \" string\n  ^\"first[\" First(uint8) ']'[1]          # The rest must starts with \"first[\" characters, then take the rest until ']' as uint8. It is\n                                         # known First is the single character, thus the [1] index.\n                                         # under the name of First\n  ^\" format[\" Format(string) ~']'        # Take format id. Format is a short word: XML, JSON, BIN. ~ before lookup oobject suggests\n                                         # generator to use for loop scan rather than IndexByte, which is although fast\n                                         # has call overhead as it cannot be inlined by Go compiler.\n  ?Hidden (^\" hidden[\" Value(uint8) ']') # Optionally look for \" hidden[\\d+]\"\n  ^\" user_agent[\" UserAgent(string) ']'  # User agent data\n  _ \"country[\" Country(string)  ']'      # Look for the piece starting with country[\n;\n```\n\n##### Code generation\nThe recommended way is to put something like `//go:generate ldetool --package main Line.lde` in `generate.go` of a package and then generate a code with\n```bash\ngo generate \u003cproject path\u003e\n```\nIt will be written into `line_lde.go` file in the same directory. It will look like [this](SAMPLE.md)\n\nNow, we have\n1. Data extractor type\n    ```go\n    // Line autogenerated parser\n    type Line struct {\n        Rest   []byte\n        Time   []byte\n        First  uint8\n        Format []byte\n        Hidden struct {\n            Valid bool\n            Value uint8\n        }\n        UserAgent []byte\n        Country   []byte\n    }\n    ```\n2. Parse method\n    ```go\n    // Extract autogenerated method of Line\n    func (p *Line) Extract(line []byte) (bool, error) {\n       …\n    }\n    ```\n    Take a look at return data. First bool signals if the data was successfully matched and error that is not nil signals if there were\n    any error. String to numeric failures are always treated as errors, you can put `!` into extraction script and all\n    mismatches after the sign will be treated as errors\n3. Helper to access optional `Hidden` area returning default Go value if the the area was not matched\n    ```go\n    // GetHiddenValue retrieves optional value for HiddenValue.Name\n    func (p *Line) GetHiddenValue() (res uint8) {\n        if !p.Hidden.Valid {\n            return\n        }\n        return p.Hidden.Value\n    }\n    ```\n\n##### Generated code usage\nIt is easy: put\n```go\nl := \u0026Line{}\n```\nbefore and then feed `Extract` method with lines:\n```go\nscanner := bufio.NewScanner(reader)\nfor scanner.Scan() {\n    ok, err := l.Extract(scanner.Bytes())\n    if !ok {\n        if err != nil {\n            return err\n        }\n        continue\n    }\n    …\n    l.Format\n    l.Time\n    l.GetHiddenValue()\n    …\n}\n```\n\n#### custom types\n\n\u003e Special thanks to Matt Hook (github.com/hookenz) who proposed this feature\n\nIt is possible to use custom types in generated structure. You should declare them first via\n\n```perl\ntype pkg.Type from \"pkgpath\";\n```\n\nfor external types and \n\n```perl\ntype typeName;\n```\n\nfor local types before all rules definitions and you can use them as field types. The parsing to be done via\n\n```go\np.unmarshal\u003cFieldName\u003e([]byte) (Type, error)\n``` \n\nfunction.\n\nExample:\n\n```perl\ntype time.Time from \"time\";\ntype net.IP from \"net\";\n\nCustom = Time(time.Time) ' ' ?Addr(^\"addr: \" IP(ip.IP) ' ');\n```\n\nNow, two parsing functions will be needed to parse this (they are to be written manually):\n\n```go\nfunc (p *Custom) unmarshalTime(s string) (time.Time, error) { … }\n\nfunc (p *Custom) unmarshalAddrIP(s string) (net.IP, error) { … }\n```\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsirkon%2Fldetool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsirkon%2Fldetool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsirkon%2Fldetool/lists"}