{"id":13801538,"url":"https://github.com/nrktkt/ninny-json","last_synced_at":"2026-01-11T16:57:43.058Z","repository":{"id":45015925,"uuid":"278688727","full_name":"nrktkt/ninny-json","owner":"nrktkt","description":"JSON typeclasses that know the difference between null and absent fields","archived":false,"fork":false,"pushed_at":"2025-03-31T20:03:32.000Z","size":342,"stargazers_count":21,"open_issues_count":9,"forks_count":8,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-31T21:22:20.349Z","etag":null,"topics":["absent-fields","ast","json","scala","typeclasses"],"latest_commit_sha":null,"homepage":"https://www.youtube.com/watch?v=3v-5Q4NLdNE","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nrktkt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["kag0"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2020-07-10T17:01:21.000Z","updated_at":"2025-03-31T20:03:37.000Z","dependencies_parsed_at":"2023-11-21T20:29:34.041Z","dependency_job_id":"3e36601f-cacc-42d4-9e86-33a21a443512","html_url":"https://github.com/nrktkt/ninny-json","commit_stats":{"total_commits":92,"total_committers":4,"mean_commits":23.0,"dds":0.5217391304347826,"last_synced_commit":"dc0956f612fda7b828f9a531125f4247e04db475"},"previous_names":[],"tags_count":37,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nrktkt%2Fninny-json","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nrktkt%2Fninny-json/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nrktkt%2Fninny-json/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nrktkt%2Fninny-json/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nrktkt","download_url":"https://codeload.github.com/nrktkt/ninny-json/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253932882,"owners_count":21986469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["absent-fields","ast","json","scala","typeclasses"],"created_at":"2024-08-04T00:01:24.049Z","updated_at":"2026-01-11T16:57:43.016Z","avatar_url":"https://github.com/nrktkt.png","language":"HTML","funding_links":["https://github.com/sponsors/kag0"],"categories":["Table of Contents"],"sub_categories":["JSON"],"readme":"# None Is Not Null\n\nninny-json is an experiment to look at what JSON type classes would look like\nif they made a distinction between absent JSON fields, and fields with `null` \nvalues.  \nThis project does include its own AST, but the point here is really not to \nintroduce a new AST or look at the ergonomics of manipulating the AST directly.\nThus, the AST included is kept simple.  \nWhy not use json4s, the project created to provide one unifying AST? Read on.\n\n## Why does this matter?\n\nIn principle, we want our libraries to be as expressive as possible.  \nIn practice, the limitations of libraries today make it hard or impossible to \nimplement things like [JSON merge patch](https://tools.ietf.org/html/rfc7396) or \n[JSON-RPC](https://www.jsonrpc.org/specification#response_object).\nWhether a field will be included in the final JSON is also left up to the \nconfiguration of the JSON serializer (whether to include nulls or not) rather \nthan the AST. When the AST doesn't match the JSON output, testability issues \ncan open up.\n\n## [Jump to proposal](#what-are-we-proposing)\n## [User Guide / Getting Started](https://nrktkt.github.io/ninny-json/USERGUIDE)\n\n## What do libraries do today? \n\nLet's look at three popular libraries and see how they deal with converting\n`Option[A]` to and from JSON.\n\n### json4s\n\njson4s uses the following type classes\n\n```scala\ntrait Reader[T] {\n  def read(value: JValue): T\n}\ntrait Writer[-T] {\n  def write(obj: T): JValue\n}\n```\n\nThese are fairly standard and pretty similar to Play JSON, with the difference \nthat they throw exceptions.\n\nInterestingly json4s includes a `JNothing` in its AST. Technically there is no \nsuch thing as \"nothing\" in JSON, but I can see how it would allow for maximum \nflexibility with other JSON libraries given that's the goal of json4s.\n\n`JNothing` *would* let us distinguish between `None` and a missing field. \nHowever, the default `Writer[Option[A]]` doesn't leverage `JNothing`, \nrather it just writes `None` as `JNull`. \nThe default reader for `Option` on the other hand just aggregates a failure to \nparse for any reason into `None`.\n\n#### Pros\n\n* It is technically possible to distinguish null from absent both when reading \nand writing JSON.\n\n#### Cons\n\n* Default readers/writers don't distinguish null from absent.\n* `JNothing` makes for a strange AST. We can imagine bugs where \n    ```scala\n    myObj.obj.map(_._1).contains(\"myField\") // true\n    // and yet\n    myObj \\ \"myField\" // JNothing\n    ```\n  Some might suggest \n  \"well you should have done `myObj \\ \"myField\" != JNothing` instead\", \n  but ideally that's a mistake that wouldn't compile.\n  \n### Play JSON\n\nPlay JSON uses the type classes\n\n```scala\ntrait Writes[A] { \n  def writes(o: A): JsValue\n}\ntrait Reads[A] {\n  def reads(json: JsValue): JsResult[A]\n}\n```\n\nwith a more standard AST.\n\nIt does provide a `Writes[Option[A]]`, which writes `None` as `null`. \nHowever, there is no `Reads[Option[A]]` since the type class has no way to know \nif the field was missing. \n\nThe nice thing about Play JSON is the macro based type class derivation, so you \ncan just write `implicit val format = Json.format[MyModel]`. \nNow you might think \n\"well, that's not very useful if there is no `Reads[Option]`\" \nand `MyModel` can't have any optional fields. \nHowever, that's not the case, and the macro generated code *will* read an \n`Option` using some internal logic. \nThis works for the common use case, but if we want to distinguish between an \nabsent field and some null, then we can't use the automatic format because we \nneed access to the fields on the outer object.\n\n```scala\nReads(root =\u003e JsSuccess(MyClass((root \\ \"myField\") match {\n  case JsDefined(JsNull) =\u003e Null\n  case JsDefined(value) =\u003e Defined(value)\n  case JsUndefined() =\u003e Absent\n})))\n```\n\n#### Pros\n\n* Automatic format derivation (although circe will call it semi-automatic)\n\n#### Cons\n\n* Inconsistent handling of `Option` between `Reads` and `Writes`.\n* If we want to take direct control, we lose the composability of type classes.\n\n### circe\n\ncirce uses the type classes\n\n```scala\ntrait Encoder[A] { \n  def apply(a: A): Json\n}\ntrait Decoder[A] {\n  def apply(c: HCursor): Decoder.Result[A]\n  def tryDecode(c: ACursor): Decoder.Result[A] = c match {\n    case hc: HCursor =\u003e apply(hc)\n    case _ =\u003e\n      Left(\n        DecodingFailure(\"Attempt to decode value on failed cursor\", c.history)\n      )\n  }\n}\n```\n\nThe `Encoder` here is the same as we've seen in the others (and it also encodes \n`None` as null), but the `Decoder` is interesting. Since circe uses cursors to \nmove around the JSON, there is an `ACursor` which has the ability to tell us \nthat the cursor was unable to focus on the field we're trying to decode (the \nfield wasn't there). circe can and does use this to decode missing fields into \n`None`, and we can use it to distinguish null from absent fields.\n\n```scala\nnew Decoder[FieldPresence[A]] {\n  def tryDecode(c: ACursor) = c match {\n    case c: HCursor =\u003e\n      if (c.value.isNull) Right(Null)\n      else\n        d(c) match {\n          case Right(a) =\u003e Right(Defined(a))\n          case Left(df) =\u003e Left(df)\n        }\n    case c: FailedCursor =\u003e\n      if (!c.incorrectFocus) Right(Absent) \n      else Left(DecodingFailure(\"[A]Option[A]\", c.history))\n  }\n}\n``` \nBecause this is a `Decoder` for the value rather than the object containing the \nvalue, we can still use circe's awesome fully automatic type class generation \nwhich doesn't even require us to invoke a macro method like we do in Play.\n\nSadly there is nothing we can do with the `Encoder` to indicate that we don't \nwant our field included in the output.\n\n#### Pros\n\n* `Decoder` can distinguish between null and absent fields.\n\n#### Cons\n\n* `Encoder` can't output an indication that the field should be absent.\n* Cursors might be intimidating to the uninitiated.\n\n## What are we proposing?\n\nNow that we have the lay of the land, what are we proposing to shore up the \ncons without losing the pros?\n\nTwo simple type classes (the signatures are what matter, not the names) \n\n```scala\ntrait ToJson[A] {\n  // return None if the field should not be included in the JSON\n  def to(a: A): Option[JsonValue]\n}\ntrait FromJson[A] {\n  // None if the field was not present in the JSON\n  def from(maybeJson: Option[JsonValue]): Try[A]\n}\n```\n\u003e note: `Try` and `Option` aren't strictly required, anything that conceptually \n\u003e conveys the possibility of failure and absence will work.\n \n`ToJson[Option[A]]` is implemented predictably\n```scala\nnew ToJson[Option[A]] {\n  def to(a: Option[A]) = a.flatMap(ToJson[A].to(_))\n}\n```\n\n`FromJson[Option[A]]` is pretty straightforward as well\n```scala\nnew FromJson[Option[A]] {\n  def from(maybeJson: Option[JsonValue]) = maybeJson match {\n    case Some(JsonNull) =\u003e Success(None)\n    case Some(json)     =\u003e FromJson[A].from(json).map(Some(_))\n    case None           =\u003e Success(None)\n  }\n}\n```\n\nIf we want to distinguish between a null and absent field\n\n```scala\nnew FromJson[FieldPresence[A]] {\n  def from(maybeJson: Option[JsonValue]) = Success(maybeJson match {\n    case Some(JsonNull) =\u003e Null\n    case Some(json)     =\u003e Defined(json)\n    case None           =\u003e Absent\n  })\n}\n```\n\nHow are we doing with our pros and cons?\n\n- [x] Able to distinguish null from absent fields when reading and writing JSON \nfrom inside the type class.\n- [x] AST is predictable and closely models JSON.\n- [x] We can automatically (or semi-automatically) derive type classes using shapeless.\n- [x] `Option` is handled in the same way when reading and writing.\n\n### Ergonomic improvements\n\nAlways dealing with `Option` could get annoying, \nso some simple additions can alleviate that\n\n#### `FromJson`\nAddition of a method that takes the AST directly saves us from having to \nconstantly invoke `Some()`.\n```scala\ndef from(json: JsonValue): Try[A]\n```\n\n#### `ToJson`\nSome types (like `String`) will always result in a JSON output. Instances for \nthose types can be implemented with `ToSomeJson` to remove the `Option` from \ncreated AST.\n\n```scala\ntrait ToSomeJson[A] extends ToJson[A] {\n  def toSome(a: A): JsonValue\n  override def to(a: A) = Some(toSome(a))\n}\n```\n### [Example](ninny/test/src-2/nrktkt/ninny/example/Example.scala)\n\nAn example of updating a user profile which clears one field, \nsets the value of another, and leaves a third unchanged without overwriting it \nwith the existing value.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnrktkt%2Fninny-json","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnrktkt%2Fninny-json","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnrktkt%2Fninny-json/lists"}