{"id":13804622,"url":"https://github.com/libmir/asdf","last_synced_at":"2025-07-16T20:31:48.141Z","repository":{"id":33123861,"uuid":"252686975","full_name":"libmir/asdf","owner":"libmir","description":"JSON library","archived":false,"fork":false,"pushed_at":"2024-08-20T07:09:05.000Z","size":2241,"stargazers_count":21,"open_issues_count":5,"forks_count":10,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-09T23:31:09.006Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://asdf.libmir.org","language":"D","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/libmir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-03T09:24:07.000Z","updated_at":"2025-03-20T04:27:07.000Z","dependencies_parsed_at":"2024-03-28T05:38:07.019Z","dependency_job_id":"aa3a233e-a3a5-499c-a7c2-dfc631549ff6","html_url":"https://github.com/libmir/asdf","commit_stats":null,"previous_names":[],"tags_count":83,"template":false,"template_full_name":null,"purl":"pkg:github/libmir/asdf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libmir%2Fasdf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libmir%2Fasdf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libmir%2Fasdf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libmir%2Fasdf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/libmir","download_url":"https://codeload.github.com/libmir/asdf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libmir%2Fasdf/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265538202,"owners_count":23784554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T01:00:51.344Z","updated_at":"2025-07-16T20:31:48.073Z","avatar_url":"https://github.com/libmir.png","language":"D","funding_links":[],"categories":["Data serialization"],"sub_categories":["JSON"],"readme":"[![Dub version](https://img.shields.io/dub/v/asdf.svg)](http://code.dlang.org/packages/asdf)\n[![Dub downloads](https://img.shields.io/dub/dt/asdf.svg)](http://code.dlang.org/packages/asdf)\n[![License](https://img.shields.io/dub/l/asdf.svg)](http://code.dlang.org/packages/asdf)\n[![codecov.io](https://codecov.io/github/libmir/asdf/coverage.svg?branch=master)](https://codecov.io/github/tamediadigital/asdf?branch=master)\n[![Build Status](https://travis-ci.org/libmir/asdf.svg?branch=master)](https://travis-ci.org/libmir/asdf)\n[![Circle CI Docs](https://circleci.com/gh/libmir/asdf.svg?style=shield\u0026circle-token=:circle-ci-badge-token)](https://circleci.com/gh/libmir/asdf)\n[![Build status](https://ci.appveyor.com/api/projects/status/libmir/asdf?svg=true)](https://ci.appveyor.com/project/9il/asdf-sesrc)\n\n# A Simple Document Format\n\nASDF is a cache oriented string based JSON representation.\nBesides, it is a convenient Json Library for D that gets out of your way.\nASDF is specially geared towards transforming high volumes of JSON dataframes, either to new \nJSON Objects or to custom data types.\n\n#### Why ASDF?\n\nasdf was originally developed at [Tamedia](https://www.tamedia.ch/) to extract and transform real-time click streams.\n\n- ASDF is fast. It can be really helpful if you have gigabytes of JSON line separated values.\n- ASDF is simple. It uses D's modelling power to make you write less boilerplate code.\n- ASDF is tested and used in production for real World JSON generated by millions of web clients (we call it _the great fuzzer_).\n\nsee also [github.com/tamediadigital/je](https://github.com/tamediadigital/je) a tool for fast extraction of json properties into a csv/tsv.\n\n#### Simple Example\n\n1. define your struct\n2. call `serializeToJson` ( or `serializeToJsonPretty` for pretty printing! )\n3. profit! \n\n```D\n/+dub.sdl:\ndependency \"asdf\" version=\"~\u003e0.2.5\"\n\n#turns on SSE4.2 optimizations when compiled with LDC\ndflags \"-mattr=+sse4.2\" platform=\"ldc\"\n+/\nimport asdf;\n\nstruct Simple\n{\n    string name;\n    ulong level;\n}\n\nvoid main()\n{\n    auto o = Simple(\"asdf\", 42);\n    string data = `{\"name\":\"asdf\",\"level\":42}`;\n    assert(o.serializeToJson() == data);\n    assert(data.deserialize!Simple == o);\n}\n```\n#### Documentation\n\nSee ASDF [API](http://asdf.libmir.org) and [Specification](https://github.com/tamediadigital/asdf/blob/master/SPECIFICATION.md).\n\n#### I/O Speed\n\n - Reading JSON line separated values and parsing them to ASDF - 300+ MB per second (SSD).\n - Writing ASDF range to JSON line separated values - 300+ MB per second (SSD).\n\n#### Fast setup with the dub package manager\n\n[![Dub version](https://img.shields.io/dub/v/asdf.svg)](http://code.dlang.org/packages/asdf)\n\n[Dub](https://code.dlang.org/getting_started) is D's package manager.\nYou can create a new project with:\n\n```\ndub init \u003cproject-name\u003e\n```\n\nNow you need to edit the `dub.json` add `asdf` as dependency and set its targetType to `executable`.\n\n(dub.json)\n```json\n{\n    ...\n    \"dependencies\": {\n        \"asdf\": \"~\u003e\u003ccurrent-version\u003e\"\n    },\n    \"targetType\": \"executable\",\n    \"dflags-ldc\": [\"-mcpu=native\"]\n}\n```\n\n(dub.sdl)\n```sdl\ndependency \"asdf\" version=\"~\u003e\u003ccurrent-version\u003e\"\ntargetType \"executable\"\ndflags \"-mcpu=native\" platform=\"ldc\"\n```\n\nNow you can create a main file in the `source` and run your code with \n```\ndub\n```\nFlags `--build=release` and `--compiler=ldmd2` can be added for a performance boost:\n```\ndub --build=release --compiler=ldmd2\n```\n\n`ldmd2` is a shell on top of [LDC (LLVM D Compiler)](https://github.com/ldc-developers/ldc).\n`\"dflags-ldc\": [\"-mcpu=native\"]` allows LDC to optimize ASDF for your CPU.\n\nInstead of using `-mcpu=native`, you may specify an additional instruction set for a target with `-mattr`.\nFor example, `-mattr=+sse4.2`. ASDF has specialized code for\n[SSE4.2](https://en.wikipedia.org/wiki/SSE4#SSE4.2 instruction set).\n\n#### Main transformation functions\n\n| uda | function |\n| ------------- |:-------------:|\n| `@serdeKeys(\"bar_common\", \"bar\")` | tries to read the data from either property. saves it to the first one |\n| `@serdeKeysIn(\"a\", \"b\")` | tries to read the data from `a`, then `b`. last one occuring in the json wins |\n| `@serdeKeyOut(\"a\")` | writes it to `a` |\n| `@serdeIgnore` | ignore this property completely |\n| `@serdeIgnoreIn` | don't read this property |\n| `@serdeIgnoreOut` | don't write this property |\n| `@serdeIgnoreOutIf!condition` | run function `condition` on serialization and don't write this property if the result is true |\n| `@serdeScoped` | Dangerous! non allocating strings. this means data can vanish if the underlying buffer is removed.  |\n| `@serdeProxy!string` | call to!string |\n| `@serdeTransformIn!fin` | call function `fin` to transform the data |\n| `@serdeTransformOut!fout`  | run function `fout` on serialization, different notation |\n| `@serdeAllowMultiple`  | Allows deserialiser to serialize multiple keys for the same object member input. |\n| `@serdeOptional`  | Allows deserialiser to to skip member desrization of no keys corresponding keys input. |\n\n\nPlease also look into the Docs or Unittest for concrete examples!\n\n#### ASDF Example (incomplete)\n\n```D\nimport std.algorithm;\nimport std.stdio;\nimport asdf;\n\nvoid main()\n{\n    auto target = Asdf(\"red\");\n    File(\"input.jsonl\")\n        // Use at least 4096 bytes for real world apps\n        .byChunk(4096)\n        // 32 is minimum size for internal buffer. Buffer can be reallocated to get more memory.\n        .parseJsonByLine(4096)\n        .filter!(object =\u003e object\n            // opIndex accepts array of keys: {\"key0\": {\"key1\": { ... {\"keyN-1\": \u003cvalue\u003e}... }}}\n            [\"colors\"]\n            // iterates over an array\n            .byElement\n            // Comparison with ASDF is little bit faster\n            //   than comparison with a string.\n            .canFind(target))\n            //.canFind(\"red\"))\n        // Formatting uses internal buffer to reduce system delegate and system function calls\n        .each!writeln;\n}\n```\n\n##### Input\n\nSingle object per line: 4th and 5th lines are broken.\n\n```json\nnull\n{\"colors\": [\"red\"]}\n{\"a\":\"b\", \"colors\": [4, \"red\", \"string\"]}\n{\"colors\":[\"red\"],\n    \"comment\" : \"this is broken (multiline) object\"}\n{\"colors\": \"green\"}\n{\"colors\": \"red\"]}}\n[]\n```\n\n##### Output\n\n```json\n{\"colors\":[\"red\"]}\n{\"a\":\"b\",\"colors\":[4,\"red\",\"string\"]}\n```\n\n\n#### JSON and ASDF Serialization Examples\n\n##### Simple struct or object\n```d\nstruct S\n{\n    string a;\n    long b;\n    private int c; // private fields are ignored\n    package int d; // package fields are ignored\n    // all other fields in JSON are ignored\n}\n```\n\n##### Selection\n```d\nstruct S\n{\n    // ignored\n    @serdeIgnore int temp;\n    \n    // can be formatted to json\n    @serdeIgnoreIn int a;\n    \n    //can be parsed from json\n    @serdeIgnoreOut int b;\n    \n    // ignored if negative\n    @serdeIgnoreOutIf!`a \u003c 0` int c;\n}\n```\n\n##### Key overriding\n```d\nstruct S\n{\n    // key is overrided to \"aaa\"\n    @serdeKeys(\"aaa\") int a;\n\n    // overloads multiple keys for parsing\n    @serdeKeysIn(\"b\", \"_b\")\n    // overloads key for generation\n    @serdeKeyOut(\"_b_\")\n    int b;\n}\n```\n\n##### User-Defined Serialization\n```d\nstruct DateTimeProxy\n{\n    DateTime datetime;\n    alias datetime this;\n\n    SerdeException deserializeFromAsdf(Asdf data)\n    {\n        string val;\n        if (auto exc = deserializeScopedString(data, val))\n            return exc;\n        this = DateTimeProxy(DateTime.fromISOString(val));\n        return null;\n    }\n\n    void serialize(S)(ref S serializer)\n    {\n        serializer.putValue(datetime.toISOString);\n    }\n}\n```\n\n```d\n//serialize a Doubly Linked list into an Array\nstruct SomeDoublyLinkedList\n{\n    @serdeIgnore DList!(SomeArr[]) myDll;\n    alias myDll this;\n\n    //no template but a function this time!\n    void serialize(ref AsdfSerializer serializer)\n    {\n        auto state = serializer.listBegin();\n        foreach (ref elem; myDll)\n        {\n            serializer.elemBegin;\n            serializer.serializeValue(elem);\n        }\n        serializer.listEnd(state);\n    }   \n}\n```\n\n##### Serialization Proxy\n```d\nstruct S\n{\n    @serdeProxy!DateTimeProxy DateTime time;\n}\n```\n\n```d\n@serdeProxy!ProxyE\nenum E\n{\n    none,\n    bar,\n}\n\n// const(char)[] doesn't reallocate ASDF data.\n@serdeProxy!(const(char)[])\nstruct ProxyE\n{\n    E e;\n\n    this(E e)\n    {\n        this.e = e;\n    }\n\n    this(in char[] str)\n    {\n        switch(str)\n        {\n            case \"NONE\":\n            case \"NA\":\n            case \"N/A\":\n                e = E.none;\n                break;\n            case \"BAR\":\n            case \"BR\":\n                e = E.bar;\n                break;\n            default:\n                throw new Exception(\"Unknown: \" ~ cast(string)str);\n        }\n    }\n\n    string toString()\n    {\n        if (e == E.none)\n            return \"NONE\";\n        else\n            return \"BAR\";\n    }\n\n    E opCast(T : E)()\n    {\n        return e;\n    }\n}\n\nunittest\n{\n    assert(serializeToJson(E.bar) == `\"BAR\"`);\n    assert(`\"N/A\"`.deserialize!E == E.none);\n    assert(`\"NA\"`.deserialize!E == E.none);\n}\n```\n\n\n##### Finalizer\nIf you need to do additional calculations or etl transformations that happen to depend on the deserialized data use the `finalizeDeserialization` method.\n\n```d\nstruct S\n{\n    string a;\n    int b;\n\n    @serdeIgnoreIn double sum;\n\n    void finalizeDeserialization(Asdf data)\n    {\n        auto r = data[\"c\", \"d\"];\n        auto a = r[\"e\"].get(0.0);\n        auto b = r[\"g\"].get(0.0);\n        sum = a + b;\n    }\n}\nassert(`{\"a\":\"bar\",\"b\":3,\"c\":{\"d\":{\"e\":6,\"g\":7}}}`.deserialize!S == S(\"bar\", 3, 13));\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibmir%2Fasdf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flibmir%2Fasdf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibmir%2Fasdf/lists"}